Scaffold Wisely

How do you make a customer-support chatbot reliable when its models have small context windows and struggle to reason? Before leading applied AI at Ramp, Rahul Sengottuvelu worked on that problem with GPT-2 and BERT. The practical response was to write code around the models, compensating for behavior they could not reliably produce themselves.

As models improved, some of that surrounding code became unnecessary. Deleting it revealed a recurring design question: which parts of an agent help it use greater intelligence, and which parts merely compensate for limitations that may disappear? Sengottuvelu’s Jsonformer was another instance of this scaffolding, addressing models’ difficulty producing structured JSON. That experience leads into three progressively more flexible architectures for a Ramp agent, then a mail-client experiment that pushes the same idea further.

The Bitter Lesson, by Rich Sutton, supplies the architectural starting point: general methods that can exploit increasing computation have a long-term advantage. Build systems whose capabilities can improve when they receive more compute. Sengottuvelu connects this to the opportunity presented by exponential progress: a system can benefit from improvements that its own developers did not have to produce.

The competing approach is to encode human expertise into clever features, heuristics, and carefully structured software. Sengottuvelu invokes chess, Go, computer vision, and Atari games to illustrate the tradeoff. His qualification matters: with compute held fixed, specialized engineering can win. As compute grows, general search and learning can overtake it. The distinction concerns the ability to exploit scale, not a blanket opposition between AI and deterministic computation.

Ramp’s finance platform covers expenses, payments, procurement, travel, and bookkeeping. Automating those workflows often means interacting with other systems, including legacy systems whose data does not arrive in a convenient format. The switching report is one such integration problem.

A customer joining Ramp uploads transactions exported from another card provider. Ramp wants to turn those transactions into a checklist that helps the customer move spending onto its platform. The difficulty is that the incoming CSV can have an arbitrary schema: different providers arrange and name their fields differently. The agent’s job is to transform that input into a format Ramp understands. The checklist is simple; reliably interpreting its source data is the hard part.

The first approach is conventional integration work. Sengottuvelu proposes supporting the fifty most common third-party card vendors: collect their CSV exports, inspect each schema, and write a dedicated mapping into Ramp’s format. For known inputs, this can work well. But the supported formats are bounded by the integrations engineers have implemented. When a provider changes its export, a parser may break and someone must repair it. The ongoing cost is maintaining knowledge of external schemas in application code.

The second approach introduces inference inside the existing pipeline. Instead of specifying every provider’s mapping, conventional code can call a model or use embedding similarity to classify incoming columns. Is this a date, a transaction amount, a merchant name, or a user’s name? Once those roles are identified, the pipeline maps the columns into the target schema.

This is more general than maintaining a parser for each vendor, but conventional code still controls the workflow. The model supplies an answer to a predefined subproblem, and the program resumes execution with that answer. Most computation remains in the fixed pipeline; model inference occupies selected steps.

The third approach gives the model control over how to solve the transformation. Its working environment includes:

• The input CSV: the model can inspect the head, tail, or whichever rows it needs.
• A code interpreter: it can write Python using pandas or other available packages to perform the transformation.
• A target format: the required output is a CSV with a specified structure.
• A verifier: a unit test or other check lets the agent assess whether its output meets the requirement.

The application specifies the desired result and a way to check it, while the model chooses the inspection and transformation steps.

Sengottuvelu reports that a single attempt was unreliable, while fifty parallel attempts worked well across many CSV formats. Parallel attempts provide more opportunities to find a working transformation, with the verifier supplying a check on the result. The talk gives no success percentage, evaluation dataset, or procedure for selecting among candidates, so this is an implementation report rather than a reproducible benchmark.

Sengottuvelu estimates roughly 10,000 times the compute of vendor-specific parsing, with a probable cost below one dollar for this scenario. Those are tentative figures from the talk, not a current pricing promise. His economic argument is that engineer time is scarce, and a failed CSV import can cost Ramp more in lost business value than the additional computation. A cheaper execution path is not necessarily a cheaper system to build and operate.

The three architectures differ in who controls execution. In the talk’s diagrams, black arrows represent conventional computation and blue arrows represent model computation.

In the second architecture, code calls the model. In the third, the model calls code: it decides when Python or another deterministic operation is useful. The parallel version branches into multiple attempts rather than following a single model-driven path.

This distinction extends beyond CSV parsing. A backend accepts a request—such as a read, update, or other CRUD operation—performs the necessary work, and returns a response. Conventionally, programmers specify the execution path. A model-controlled backend preserves that request–response boundary while delegating more of the decisions inside it to the model.

Sengottuvelu reports that parts of Ramp’s codebase are moving toward the third approach. His expectation is that delegating more work to models exposes more of the application to future capability improvements funded by model developers. The blue arrows may get better without the application team rewriting each individual workflow. That is the proposed architectural advantage, rather than a guarantee that every model upgrade automatically improves the whole application.

Push that architecture further and consider a webmail application. In the conventional version, a server sends JavaScript, HTML, and CSS to the browser. The browser renders an inbox. Clicking an email causes the frontend to request its contents from the backend, which queries storage and returns the result. An LLM might have helped an engineer write that application, but once deployed, its execution is still conventional computation.

The alternative places the LLM in the execution path of each request. The model is doing backend work at runtime, using a code interpreter, network requests, and potentially database access. To explore that idea, Sengottuvelu introduces an experimental mail client and invites the audience to send messages to a test address. Regular JavaScript still connects the browser to the LLM; the experiment changes who decides what to execute, rather than eliminating all application code.

After login, the client takes time to respond. Sengottuvelu explains that the Gmail token is being sent into an LLM chat session with code-interpreter access. The model is instructed to simulate a Gmail client and render a reasonable homepage using that access. The result is a Markdown inbox containing messages sent by audience members; he clarifies that Markdown rendering is probably specified in the instructions. This describes the experiment’s credential handling, not a validated security design.

Opening a message follows a different path from a conventional application-specific route:

1. The user clicks the inbox entry labeled Hello from California.
2. The client sends the clicked text and its identifier to the model.
3. The model receives the interaction and gets another opportunity to render the page.
4. The resulting page displays the email body.

Sengottuvelu infers that the model probably makes a GET request to retrieve the message. The visible result is the rendered email; the underlying request is not independently established by the demonstration. The model is effectively choosing the next interface in response to an event, much as a web framework updates a view.

The opened message also has actions for marking it unread or deleting it. These controls illustrate how the model can supply interface elements beyond the initial inbox. Sengottuvelu tries the delete action, but displaying a control and clicking it do not establish that the corresponding mutation completed.

The final interaction remains slow, and the model continues trying to respond to a click without a clearly established successful outcome. Sengottuvelu describes this kind of software as barely working today. Unlike the production switching-report case, the mail client demonstrates an experimental direction, with latency and reliability still conspicuous.

Its future depends on whether improving models make runtime delegation practical for more of an application. That would extend the same progression seen in CSV parsing: from manually specified behavior, through narrowly delegated inference, toward models deciding which operations to perform. Sengottuvelu leaves the destination unresolved. Whether more software will look like this remains an open question.