Putting API Gateway in front of a scraping engine

Context

pline.ai’s scraping engine does expensive work for every request: proxies, headless browsers, LLM tokens. It runs inside a Kubernetes cluster, and it should never be the thing that decides whether a caller may use it.

I put AWS API Gateway in front of it as the only public edge. The gateway answers four questions before the engine sees a byte: is this a real key, which team owns it, does that team have credits for this kind of request, and how fast may it go? Only a request that passes all four reaches the engine.

Constraints

• The engine stays private. No public load balancer, no public ingress.
• The check runs on every request. Whatever decides access sits in the hot path, so it has to be one fast lookup.
• Balances change constantly. A team that runs out of credits must stop at once, not after a cache expires.
• Keys are secrets. They must never be stored or logged in plain text.
• Some requests are slow. A browser scrape or a site map can legitimately take longer than API Gateway’s default integration timeout.
• Callers deserve a reason. “Forbidden” is not enough when the fix is “add credits”.

Design

One way in. The public API is a regional REST API on a custom domain. Every route integrates over a VPC link with an internal network load balancer in front of the cluster’s private ingress. The engine has no public address; the gateway is the only path to it.

A request authorizer on every call. A Lambda authorizer reads the x-api-key header, hashes it, and resolves it in one query to its team, the team’s active plan and its credit balance. It allows the call only if the key and the team are active, the key has not expired, and the balance covers the minimum for that route — a search request needs more than a plain page fetch.

Identity travels as headers. On allow, the authorizer returns the team, the key, the plan and the plan’s browser-concurrency limit in its context. Each route’s integration maps those values to request headers, so the engine knows who is calling and how many browser sessions that team may hold — without ever seeing a key.

The gateway still meters keys. The API’s key source is the authorizer, not the client header: the authorizer hands the key back to API Gateway, so the key’s usage plan keeps throttling it at the edge.

Key decisions

Keep the engine private. A public engine with its own auth would be a second front door to secure. With a VPC link to an internal load balancer, there is nothing to reach except through the gateway.

No caching on authorizer results. API Gateway can cache an authorizer’s answer per key. That would let a team keep spending for the cache’s lifetime after its balance hit zero. Caching stays off; the lookup is kept to a single query so running it on every request is cheap.

Leave out what is unset. API Gateway turns every context value into a string. A plan with no concurrency limit would have reached the engine as the literal string "None", a value the engine was never meant to interpret. The authorizer omits the value when it is unset, so the header is simply absent and “no limit configured” stays unambiguous.

Refusals that say why. I made the gateway’s refusals JSON that carries the authorizer’s error code and message — “insufficient credits, add credits and try again” — and gave a missing key its own response with a hint about the header.

Version in the path, not the base path. The API is served under /v1 as a resource prefix. Mapping v1 as a base path would have stripped it before routing, and the engine needs to see the version to build versioned job URLs.

Timeouts per route. Most routes keep API Gateway’s default integration timeout. The slow ones — browser scrapes, site maps, interactive sessions — get longer, per route, instead of one generous timeout for everything.

Observe without leaking. Stage metrics and execution logs are on; data tracing is off, so request and response bodies never land in the logs.

Keys and plans

Keys are real API Gateway keys, created by the application. Only a hash, a prefix and the provider’s key id are stored.

• Plans map to usage plans. Each product plan points at a gateway usage plan, and several plans can share one. When a team changes plan, its keys are attached to the new usage plan and detached from the old one.
• Revoking is disabling. A revoked key is marked revoked in the database and disabled in API Gateway, so it fails at the edge in two independent places.
• No waiting at sign-up. Creating a key in AWS is slow. A long-running workflow keeps a warm pool of pre-created keys topped up, and a new team claims one inside its sign-up transaction. Concurrent sign-ups take different keys (SKIP LOCKED), and a rolled-back sign-up returns its key to the pool.

Result

The engine is no longer an edge. Every request is authenticated, priced against the team’s balance and throttled by its plan before it reaches the cluster, and every refusal tells the caller what to fix. New teams get a working key the moment they sign up, and plan changes move their keys between usage plans without anyone touching the console.