A data profile for every crawl run

Context

A crawl run produces a dataset, and the first question about it is “does this look right?” Row count, fill rate per column, duplicate rows, size. Before this service, nobody computed those numbers per run, and a run with half its fields empty looked the same as a good one until a customer noticed.

I created a data-profiler service that profiles every run’s export and stores the result. A teammate co-led it with me and later built anomaly detection on top: comparing each run’s profile with earlier runs and raising action items.

Constraints

• Profiling is heavy. It reads the whole export into memory, and exports range from tiny to very large.
• Shared cluster. Profiling jobs share Kubernetes capacity with crawlers and other services.
• Long, unreliable waits. A job can finish, fail or vanish without telling anyone.
• Multi-page datasets. One run can extract several record types (“pages”), each with its own columns.

Design

A request to profile a run starts a Temporal workflow. It reads the dataset’s page metadata, finds the latest export file, and starts a Kubernetes job that profiles the file and writes the reports to S3. It then waits for the job, records the outcome, and returns a report link for each page of the dataset.

Key decisions

A job per profile, not work in the service. Profiling runs in a disposable Kubernetes job with its own memory limit, not inside the gRPC service. A huge export can only hurt its own job. Finished jobs clean themselves up after two hours.

Gate on size before spending. Before starting a job, the workflow multiplies rows by columns. A dataset above 10 million cells is skipped with a clear status instead of starting a job that would be slow, expensive or killed for memory.

Retries belong to the workflow. The job is not restarted by Kubernetes when it fails. The workflow sees the failure, records it, and decides what happens next, so there is one place that knows the run’s state.

Signal, with a fallback. The workflow waits for a completion signal with a timeout. On each timeout it asks Kubernetes for the job’s real status. A job that finishes without anyone being told is still noticed, within a bounded number of checks.

One report per page. A multi-page dataset gets a separate report for each page, because each page has its own columns. Single-file exports get a single default report.

Searchable workflows. Each workflow is tagged with its report, run and requester as search attributes, so operations can find any profile’s execution by the business ids they already know.

Failure handling

• Every dead end is a recorded status. Too many cells, no supported export format, or a job that never started — each one marks the profile as failed with a specific message instead of leaving it pending.
• Wrong report names. Per-page report names were built with an extra prefix, so links pointed at files that did not exist. Fixed in the name builder.
• Unindexable search attribute. The generation type was stored as a raw enum number, which was useless for searching. It is now stored as the enum’s name.

Result

Every crawl run can carry a data profile — rows, fill rate, duplicates and size per page — computed in isolation from the services around it. Those profiles became the input for anomaly detection, which a teammate built on top.