Archiving crawl data, and bringing it back

Context

Every crawl run writes its records into MongoDB, so they can be queried and exported quickly. Most runs are never read again after delivery, and MongoDB is an expensive place to keep data nobody reads.

A Flink pipeline, built earlier by a teammate, already streamed every record to S3 as Avro. What was missing was the lifecycle around it: when a run’s MongoDB data may go, and how to bring a run back when a customer asks for an old export.

I built both: the archival schedule in the export service, and the restore path across the export service (Python, Temporal) and the crawl-data service (Go).

Constraints

• Hot stays fast. Recent runs must stay queryable in MongoDB, with their indexes.
• Not all runs are equal. A good run, a failed run and an empty run have different reasons to be kept.
• Restores are big and slow. One run can hold millions of rows, and report sizes vary widely.
• Restored data must be usable at once. A run restored without its indexes is technically there and practically useless.

Design

Archival. When a run is delivered, the export workflow computes its archive date and writes it on the run. A separate RPC lets other services schedule archival for a run explicitly. A sweep later removes runs whose date has passed from MongoDB. Their records are already in S3.

Restore. A restore workflow takes a report and a list of runs. For each run, in parallel, the crawl-data service ensures the collection’s indexes, then reads the run’s Avro schema and data files from S3 and bulk-inserts the records back into MongoDB.

Key decisions

Retention follows the outcome. A successful run keeps a short grace window after delivery. I cut the window for failed runs from 30 days to 7, because failures are investigated in days, not weeks. A run that produced no data is archived after 30 minutes — there is nothing to keep it for.

Indexes before data. The restore ensures the collection has its expected indexes before inserting, and fails the step if it does not. Building indexes on an empty collection is cheap. Building them under a freshly loaded one is not, and a restored run without them is unusable until they finish.

Batch size per report. Inserts go in batches with an explicit size, with a default and a per-report override. A narrow report and a very wide one need different batch sizes, and one global constant cannot suit both.

Heartbeat the long call. A single run’s restore can take a long time. The activity heartbeats every 30 seconds while the restore runs. Temporal can tell a slow restore from a dead one without a huge timeout.

Do not re-process a fresh restore. Just after a run is restored, the export workflow skips its usual sync step for that run, and gives it a new grace window before it can be archived again.

One run’s failure is one run’s failure. Runs in a restore batch proceed independently. The workflow reports how many failed, instead of failing the whole batch on the first bad run.

Later: moving storage

When the archive storage moved to a new location, I added a filter to the Flink pipeline that routes each record by its run id: older runs to the old bucket, newer runs to the new one. The read side uses the same boundary, so old runs remain restorable and the stream never had to be drained.

Result

Finished runs leave MongoDB on a schedule that reflects why they ended, and any archived run can be restored on demand — indexed and ready to query — with one workflow call.