oshan.shrestha
← All work

2026 · Designed and built the service; directed the LLM engine

AI data transformations: generated code, sandboxed

Turning a plain-English instruction into a validated, sandboxed Python transform — and an LLM path for the rest, with a cost ceiling and real spend.

Published · Updated

2
safety layers before code touches data
2
paths: generated code or per-row LLM
≥ 90%
test coverage gate in CI

Context

Customers want their crawled data in their own shape: split a full name, normalise prices, map a category, tag a review’s sentiment. At Grepsr these transforms were Python transform(row) functions, and someone had to write each one by hand.

I built a new service that lets a user describe the transform in plain English and get working, previewed code back. For transforms that code cannot express, I directed a teammate in building an LLM execution path in the engine that applies rules across a dataset. I then built its schema locking, ordered output and cost controls myself.

Constraints

  • Generated code is untrusted. It comes from a model and runs against customer data. It must not read files, import arbitrary modules, loop forever or exhaust memory.
  • Some transforms are not code. “Is this review positive?” has no deterministic function. Those rows need a model per row.
  • Per-row LLM calls cost real money. A large dataset can burn through a budget before anyone notices.
  • Order matters. The output file must keep the dataset’s row order, even when LLM calls finish out of order.

Design

AI data-transformation flow A user's instruction and target columns go to the transformation service, which sends sample rows and the instruction to an LLM. The returned Python is checked by a static validator, then previewed in an isolated subprocess on real sample rows. Saved rules run over the full dataset in a Temporal workflow: code rules run as sandboxed Python, LLM rules run per row in bounded batches through an ordered Redis buffer. A cost ceiling blocks runs that are estimated to cost too much, and real spend is read back from the LLM gateway after every run. Instruction + target columns Generate LLM + sample rows Validate AST allow-list Preview isolated process Save full run · Temporal workflow Cost ceiling estimate, then block Code or per-row LLM ordered Redis buffer Real spend read back per file
Generated code passes two checks before it runs. Full runs are capped before they start and reconciled after they finish.

Authoring. The user writes an instruction and picks the output columns. The service loads a few real sample rows and asks the model, with structured output, for one Python file that defines transform(row: dict) -> dict using only the standard library — or an explicit error if the instruction cannot be done deterministically. The service can also profile a dataset and suggest transforms, so the user does not start from a blank box.

Checking. Before anything runs, a static validator walks the code’s syntax tree: an import allow-list, the exact function signature, no exec, eval, open or dunder access, and a size limit.

Previewing. The code then runs on real sample rows in a separate OS process, with restricted built-ins, a memory limit and a hard timeout. The user sees the output before saving the rule.

Running. A saved rule runs across every page of the dataset in a Temporal workflow. Code rules run as Python over each row, with the same import allow-list. LLM rules call the model once per row.

Key decisions

Two layers, not one. A syntax-tree check cannot stop an infinite loop or a memory blow-up, and starting a process for every static question is wasteful. So the validator rejects what it can see, and the sandbox contains what it cannot.

spawn, not fork. The sandbox starts its worker with spawn. Forking a process that holds C-extension state can deadlock, so the worker always starts clean. Getting the start method right on Linux took a follow-up fix.

Send the model only what it needs. An LLM rule names its dependent columns and its output schema. Each call sends only those columns, and the schema constrains the reply. Inputs are smaller, outputs are checkable, and the model never sees columns it does not need.

Concurrent calls, ordered output. Rows go to the model in bounded concurrent batches and finish in any order. Each result is written to an ordered Redis buffer under its row id, then drained in the original order into the output file.

A ceiling before, the truth after. The engine estimates a run’s cost before it starts. I made that estimate a hard ceiling: a run estimated above the limit is refused before the first token is spent. Every run — successful or failed — ends by reading the actual spend for that file from the LLM gateway’s tagged usage and storing it. The estimate prevents surprises; the read-back is what gets reported.

Progress as a stream. Progress used to travel back as a Temporal signal, which meant passing a workflow handle into another service. I replaced it with a streaming gRPC response. The activity reads the stream and heartbeats on real progress, not on a timer.

Failure handling

  • Invalid JSON from the model is repaired if possible, otherwise retried up to three times, with the error fed back to the model.
  • Connection errors back off and retry instead of failing the row.
  • A bad rule is not a crashed run. A rule that fails on missing input is reported back to the user as something to fix, separately from an infrastructure failure.
  • Cost is recorded on every exit path, including failures, so partial runs are never free in the books.

Result

Users get from an English sentence to a previewed, saved transform in one step. Generated code never runs before it passes the validator. Per-row LLM transforms are capped before they start, and their real cost is on the file when they finish.

The same idea — an LLM shaping data to a caller’s schema, validated and metered — later became pline.ai’s structured JSON output: LLM structured output.