Adding a second AWS region to a running platform

Context

Grepsr’s platform ran in one AWS region: crawlers, their Kubernetes cluster, the data stores, the pipelines. In 2026 it needed a second region, with its own Kubernetes cluster and its own storage.

A crawl touches many services. The scheduler starts it, task queues fan it out, the data service stores its records, and the pipelines archive them. All of them had to learn that “where” is now a question, without stopping the platform to switch.

Constraints

• No big-bang cut-over. The platform runs crawls around the clock. Moving everything at once was not an option.
• Old data stays where it is. Years of archived runs live in the first region. They must remain readable.
• Separate clusters. The second region has its own Kubernetes API, certificate and credentials.
• Two schedulers. Each region runs its own scheduler deployment, so queued work can be picked up by either.

Design

Key decisions

Move one report at a time. Routing starts from a per-report allow-list in configuration. An allowed report’s crawl jobs are tagged with the platform region and created through that region’s Kubernetes client. Every other report runs exactly as before. The first report moved alone.

Parameterise, do not fork. The existing Kubernetes wrapper learned to take a region: it resolves that region’s API host, certificate and token itself. Callers ask for a region, they do not wire up clusters. There is one code path, not two.

Region-aware task queues. Child tasks are dispatched by a task-processor workflow. It now holds a client for each scheduler deployment and sends each task to the deployment of the region it belongs to. The routing key and value are configuration, not code.

Find data by run, not by migration. The data service picks the storage location by comparing the run id with a cut-over boundary: older runs read from the original region, newer runs from the new one. The archival pipeline routes its writes by the same boundary. Years of archived runs stayed where they were, and both sides agree on where any run lives.

Failure handling

Stuck tasks. With two scheduler deployments, a queued task could be left waiting when its dispatch fell between them. I added a ReconcileQueuedTasks call that finds a run’s tasks still pending or queued after 30 minutes and puts them back in the queue.

Along the way: role-based AWS credentials

Some services authenticated to AWS with long-lived access keys from configuration. Static keys leak, and rotating them is manual. Between 2025 and 2026 I moved the scheduler, the export service and the Flink and archival pipelines to the default AWS credential chain: the workload’s IAM role in the cluster, and nothing in configuration. In the Java pipelines this meant every AWS client — S3, CloudWatch, the event and processing stages — took the provider chain instead of key parameters.

Where a change broke a service in one environment, I rolled it back and reintroduced it per environment, rather than forcing it everywhere at once.

Result

The platform runs in two regions behind one control plane. Reports moved over one at a time with a configuration change, old data stayed readable in place, and fewer services hold AWS keys at all.