Proxy cost per crawl run, end to end

Context

Residential and mobile proxies are one of the largest variable costs of a crawl, and the provider bills by bandwidth. At Grepsr that cost was invisible per run: nobody could say what one run of one dataset spent on proxies, or whether this run cost more than the last one.

A teammate had started a small proxy forwarder in Node, and the team had an older Flink job for crawler events. I turned them into a metering path: every byte a crawler sends through a proxy becomes a priced line item on the run that used it.

Constraints

• Measure what the provider bills. Only bytes between the proxy and the target site count, not the traffic between the crawler and the proxy.
• HTTPS hides everything. A CONNECT tunnel is opaque, and it can stay open for minutes. The usual “request finished” event only fires for plain HTTP.
• Volume. Every proxied request produces stats. One RPC per request would overwhelm the services behind it.
• Prices change. Provider prices move over time, and old runs must keep the price they ran at.
• Cost is internal data. It is shown to admins, not to every user of a dataset.

Design

At the proxy. Crawlers pass their run, batch and provider in the proxy credentials. The server reads each connection’s byte counters and logs only the delta since the last read, as one structured line per tick. The log format matches what the Flink job parses, so no adapter sits in between.

In the stream. The proxy’s logs reach a Kinesis stream. A Flink job parses them, keys them by run, batch and proxy type, and sums upload, download and request counts in 10-second event-time windows. Each window becomes one gRPC call, not one call per request.

Priced once. A dedicated add_history_proxy_stats RPC receives each window. The stats service adds the bytes to the run’s daily total, prices that total against an effective-dated price table, and upserts one row per run, day and proxy type.

Read per run. The projects API sums those rows into a run’s proxy cost and compares it with the previous run of the same dataset, so a sudden jump is visible next to the run that caused it.

Key decisions

Count target bytes, not client bytes. The pipeline first counted bytes between the crawler and the proxy. The provider bills the other side, so I switched the pipeline to the proxy-to-target counters.

Poll open HTTPS tunnels. A crawler can hold an HTTPS tunnel open for minutes, and nothing reports on it until it closes. I added a tracker that reads each open tunnel’s counters every 3 seconds and logs the delta, plus one final flush on close. Long tunnels now report usage while they run, not in one lump at the end.

Tumbling windows instead of session windows. My first version used processing-time session windows, which close only after a quiet gap. A busy run never goes quiet, so its usage could wait indefinitely. A week later I replaced them with fixed event-time windows, which emit on a steady cadence for every key, whatever the traffic.

A narrow RPC for a narrow job. Instead of reusing a generic stats call with fields the stream never sets, the pipeline calls an RPC built for exactly what it produces: run, batch, proxy type, bytes up, bytes down, timestamp.

Prices are data, not code. Prices live in a table keyed by proxy type and effective date. A price change is a new row, not a deploy, and historical runs keep the price that applied when they ran.

Failure handling

• A mixed-up sum. The window’s reducer added one record’s upload to the next record’s download. A one-word bug, but it inflated every upload total. Fixed to sum like with like.
• Negative costs. When a proxy type had no price yet, the pricing step returned a -1 sentinel, and that sentinel flowed into the run’s total. The read path now clamps each row at zero, and the summary RPC excludes unpriced rows.
• A slow leak. The proxy kept the last stats of each connection to compute deltas, and never removed them. It grew by one entry per connection for the life of the process. The entry is now deleted in the delayed close handler, after the final stats line is written.

Result

Every crawl run now carries its proxy cost, broken down by proxy type, with the change from its previous run. The same path measures a long HTTPS tunnel and a single HTTP request, and a price change no longer needs a deploy.