pline.ai platform architecture

Context

pline.ai is Grepsr’s web data API: scrape, crawl, batch, search and structured extraction behind one key. My teammates built the scraping engine. I designed and built everything around it — the part that decides who is calling, what they may do, what it costs, and who pays.

That layer has to be right in a way the engine does not. A failed scrape can be retried. A request charged twice, or a key that outlives its plan, is a support ticket and a loss of trust.

Constraints

• Usage-based pricing. Every request has a price that depends on what actually ran: proxy tier, browser rendering, LLM tokens for structured output.
• Long jobs. A crawl can run for minutes and fetch hundreds of pages, so credits must be held before the work starts and settled after.
• The database lives outside the cluster. Every query is a network round trip, so the number of queries matters more than CPU.
• A small team. The design had to be operable by the people who build it, with failures that explain themselves.

Design

The request path stays short: the gateway’s authorizer resolves the key to a team, its plan and its concurrency limit, then the engine does the work. Everything about money happens after the response, off the request path.

The engine publishes one metadata-only usage event per request. A Temporal worker in Go prices it and records it through an internal-only ingestion service, which writes the usage record and the credit debit into PostgreSQL atomically. The dashboard API — auth, teams, keys, request history and Stripe billing — reads and writes the same ledger.

Key decisions

One identity: the API key. Producers send only the key’s id. The team and the gateway key behind it are derived server-side, and an unknown key is rejected. Nobody can bill another team by sending the wrong team id, because there is no team id to send.

A warm pool of gateway keys. Creating a key in the gateway is slow, so keys are pre-created and claimed instantly at sign-up. How the gateway, its authorizer, usage plans and the key pool fit together is its own case study: Putting API Gateway in front of a scraping engine.

Reserve, then settle. Crawl and batch jobs hold their full page cost before they start and settle against what actually ran. A cleanup job releases reservations that stay unsettled for an hour, so a crashed job can never lock credits forever.

Stripe decides what is sold; the ledger decides what is owed. Checkout, webhooks, refunds, failed payments and plan changes reconcile into the credit ledger. When a webhook breaks a billing invariant, the log names the invariant — a refusal you can act on, not a silent mismatch.

Least privilege in the database. Row-level security is enabled on every table, the ingestion service is reachable only inside the cluster, and the dashboard ships an enforcing Content-Security-Policy.

Result

pline.ai launched on this architecture and serves ~200k requests a day. Every request is authorised at the edge and metered after it runs.

The split also made pricing cheap to change. When pricing moved from a flat rate to per-action prices with itemised receipts, the engine only had to report what ran — proxy tier, render mode, tokens. Deciding what that costs stayed in one place: the metering worker.