Crawler scheduling: from a polling loop to Temporal

Context

The scheduler starts every crawl on Grepsr’s platform: one-off runs, recurring schedules and the child tasks a crawl fans out into. Each crawl runs as a Kubernetes job.

It used to work like a cron job. A poller locked a MySQL table, picked the next five due runs, marked them as processing and called the service that starts the job. Retries were counters in the same table. If the poller died mid-run, nothing remembered what it was doing.

A teammate had built a first Temporal workflow for a crawl run, behind an allow-list of reports. I took it from that pilot to every schedule on the platform, and then owned the service: its releases, its pods, its timezone bugs and its move to async.

Constraints

• No big-bang switch. Crawls run around the clock. Old and new paths had to run side by side until the new one was proven.
• Workflows already in flight. A deploy cannot break a crawl that started on the previous version of the code.
• Long-running workflows. A crawl workflow loops until every child task finishes. Temporal caps a workflow’s event history, so a loop cannot grow forever.
• Kubernetes is half the system. A stuck or slow-to-die pod costs money and holds a concurrency slot, whatever Temporal thinks its state is.

Design

A due schedule starts a run workflow instead of calling the job service directly. The workflow starts the crawler job, records its own id on the run so other services can signal it, and then loops: it compares running tasks with their pod phases, syncs the database, and waits for a status signal or a timeout. A stop signal cancels queued tasks and stops the pod gracefully.

Key decisions

Roll out in steps, behind the old path. One-off runs moved to Temporal first. Scheduled runs followed about ten weeks later, once one-off runs had proven the path. Until then, an allow-list decided which reports used the workflow, and the old poller handled everything else.

Versioned task queues as the deploy switch. At each phase I moved new work to a fresh task queue — enhanced, then -ii, then -iii — and later made the queue name configuration. Workflows already in flight drained on the old queue and its workers, while new runs started on the new ones. Changing workflow code never had to replay against old history.

Continue-as-new for the long loop. A large crawl can loop for a long time. I added continue-as-new to the run workflow, carrying its state forward, so its history stays bounded however long the crawl runs.

Make Kubernetes enforce what Temporal cannot.

• I added a liveness probe to every crawler job, so Kubernetes restarts a wedged pod by itself.
• I cut the termination grace period from one hour to five minutes, so a stopped crawl releases its node quickly.
• I made child-task stops graceful by default, so a child flushes its work before the pod goes away.
• I added node selectors and tolerations, so crawlers land on the right capacity.

Async, then all-in. I converted the RPCs and signal handling in the run path to async calls, behind a list of test projects. When the async path held up, I removed the fallback and turned it on for every run.

Failure handling: the timezone saga

Monthly schedules (“the second Tuesday of every month, 9:00 in the customer’s timezone”) broke in ways that took two months and 18 PRs to fully fix. The root causes, in the order I found them:

1. Naive and aware datetimes mixed. Recurrence rules were rebuilt from strings that sometimes had a UTC offset and sometimes did not. I made every rule timezone-aware, and dates without a timezone are now treated as UTC explicitly instead of naive.
2. Old rules stayed naive. Schedules created before the fix still stored naive rules. They are now rewritten to the aware form the next time they are touched.
3. End dates too. The schedule’s end date was compared with “now” without a timezone, so schedules could end a few hours early or late.
4. Next-run previews. The calculator that shows upcoming runs used naive UTC time. It now uses aware time throughout.

The lesson: fix the representation, not the instance. Each bug was the same mistake in a new place, and the fixes only stuck once every datetime in the path had a timezone.

Result

Every scheduled and one-off crawl runs as a durable workflow that survives a worker restart. A lost signal costs a timeout, not a stuck run. Deploys switch task queues instead of breaking in-flight workflows.

I later started a second-generation scheduler service on async SQLAlchemy, with tests and CI from the first commit. It reimplements the task-queue RPCs as the first step of modernising the original.