LLM structured output: from customer feeds to an API feature

Context

Crawled data rarely arrives in the shape a customer wants. Every customer has its own field names, formats and rules, and hand-written transformation code for each one does not scale.

At Grepsr I directed the build of an LLM transformation engine for exports: each row goes to a model with only the columns it needs and a fixed output schema, instead of a bespoke transform per customer. I built its schema locking, ordered output and cost controls — the full story is in AI data transformations.

pline.ai’s structured JSON output builds on that work. When it needed a “give me this page as JSON in my schema” feature, the same idea applied — but now for any page, any schema, and any caller on the public internet.

Constraints

• Pages are bigger than context windows. A product listing can be far larger than what a model reads well in one pass.
• Splitting can corrupt meaning. Cut a table or a list item in half and the model invents or drops data.
• Output must be trusted. Callers parse the result in code, so it has to match their schema, not merely look like JSON.
• Tokens cost money. LLM usage must be measured per request and billed like every other unit.

Design

Markdown first. In pline.ai the extraction runs in Rust, inside the engine’s request path. The page is converted to Markdown before extraction. It keeps the structure a model needs — headings, lists, tables — and drops markup it does not.

A chunker that respects structure. For list-shaped schemas on large pages, the chunker splits only on block boundaries. Tables stay whole, indented tables stay inside their list item, and an oversized list is split between items, never inside one.

Extract, merge, validate. Each chunk is extracted against the caller’s schema, list results are merged, and the output is validated before it is returned — the caller gets data that matches the schema or a clear failure.

Bounded and configurable. Extraction has its own timeout and its own API credential, and the model temperature is configurable per deployment.

Metered like everything else. The extraction reports its input and output tokens on the usage event, and the metering worker prices JSON output by tokens alongside proxy and rendering costs.

Result

The approach now runs in two places: customer-specific transformations at Grepsr, and structured JSON output for any page in pline.ai — including batch jobs, where every page in the batch can return JSON.

The lesson carried over too: the model is the easy part. The work is in what surrounds it — what you feed it, how you split it, how you check what comes back, and how you charge for it.