Two small services: a dedup cache and an event router

Context

Not every service is a platform. Some are small, single-purpose pieces that many others lean on, and they need to be boringly correct. These are two I created at Grepsr.

• A key-value cache for crawlers. Crawlers need to know, per record, “have I already seen this in this run?” There was no shared place to ask.
• An event router. Platform services publish events — a run completed, an export started, logs downloaded. Each event should start the matching Temporal workflow, once.

The dedup cache

One question, one call. The core RPC is CheckIfNew: “is this key new for this run — and if it is, remember it”. A crawler asks once per record and skips the record if it has been seen. Get, Set and Exists cover the rest.

Scoped by caller and run. Every key is namespaced by a service code and by the run or batch it belongs to. Two crawlers, or two runs of the same crawler, can never collide on a key.

Hash the key, not the payload. Keys are hashed to a short fixed length before they are stored, which keeps memory per entry small. Hashing is memoised in process, so hot keys cost one hash. When one caller needed lower collision risk, I added a longer hash as an opt-in prefix on the service code, so no wire contract changed.

Synchronous first, async when it mattered. I started it as a plain threaded gRPC server. When a teammate moved the server to async, I moved the storage side with it the same day: a contextual async Redis client on a shared pool, and parallel reads across cache targets.

A cache is not a source of truth. Read errors degrade to “not found” instead of failing the crawler. The worst case is a record processed twice, never a crawl that stops.

The service began on DynamoDB with selectable backends. A teammate later moved it to primary and secondary Redis behind the same RPC contract.

The event router

I built it as a small AWS SAM application: an EventBridge rule per stage, and one Lambda function.

Routing by convention. The router maps an event to a workflow by name: ReportRunCompletedEvent starts ReportRunCompletedEventWorkflow. A new event type needs a new workflow, not a router change. An event with no matching workflow goes to a catch-all workflow, so it is recorded, not dropped.

Exactly-once starts on at-least-once delivery. EventBridge can deliver an event more than once. The workflow id is the workflow name plus the event id, with a reuse policy that allows a new start only after a failure. A redelivered event finds its workflow already started and is skipped cleanly.

Retry what can succeed. Failed starts are retried a bounded number of times — except for status codes that will never succeed, such as an invalid argument or an already-existing workflow. Those fail at once instead of using up the retries.

Keep heavy work apart. Export-related events go to their own task queue, so a burst of exports cannot delay every other event.

Skip without a deploy. Some events for specific reports should not start workflows at all. Those reports come from configuration. Earlier I had built a settings service, with bulk add and remove calls, for this kind of report skip list.

Failure handling in the event store

Events also land in an activity store on DynamoDB, which feeds the product’s activity history. I am one of that service’s main contributors. Two fixes there:

• Filtered queries came back short. DynamoDB applies a filter after it reads a page, so one query can return few or no matches while more exist further on. The code trusted the first page. It now follows the pagination key until the end.
• Payloads could expose personal data. Activity responses returned raw event payloads. They now pass through a redaction step at the read boundary, because the services that write events do not control what ends up in them.

Result

Crawlers deduplicate with one call per record, scoped to their run. Platform events become Temporal workflows exactly once, and adding an event type needs no router change.