# Scaffold Wisely Rahul Sengottuvelu, Ramp AI Engineer Summit 2025 [Talk page](https://ai.engineer/talks/-rsTkYgnNzM-scaffold-wisely) · [Video](https://www.youtube.com/watch?v=-rsTkYgnNzM) ## Scaffold Wisely An arbitrary CSV import and an experimental mail client expose an architectural choice: should conventional code call the model, or should the model decide when to run code? Familiarity with CSV files, basic web request–response flows, and LLM tool use will help you follow the architecture comparisons. ### When better models make code disposable [0:18](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=18s) How do you make a customer-support chatbot reliable when its models have small context windows and struggle to reason? Before leading applied AI at Ramp, Rahul Sengottuvelu worked on that problem with **GPT-2** and **BERT**. The practical response was to write code around the models, compensating for behavior they could not reliably produce themselves. As models improved, some of that surrounding code became unnecessary. Deleting it revealed a recurring design question: which parts of an agent help it use greater intelligence, and which parts merely compensate for limitations that may disappear? Sengottuvelu’s [Jsonformer](https://github.com/1rgs/jsonformer) was another instance of this scaffolding, addressing models’ difficulty producing structured JSON. That experience leads into three progressively more flexible architectures for a Ramp agent, then a mail-client experiment that pushes the same idea further. ### Design for more compute [2:27](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=147s) [The Bitter Lesson](https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf), by Rich Sutton, supplies the architectural starting point: general methods that can exploit increasing computation have a long-term advantage. **Build systems whose capabilities can improve when they receive more compute.** Sengottuvelu connects this to the opportunity presented by exponential progress: a system can benefit from improvements that its own developers did not have to produce. The competing approach is to encode human expertise into clever features, heuristics, and carefully structured software. Sengottuvelu invokes chess, Go, computer vision, and Atari games to illustrate the tradeoff. His qualification matters: with compute held fixed, specialized engineering can win. As compute grows, general search and learning can overtake it. The distinction concerns the ability to exploit scale, not a blanket opposition between AI and deterministic computation. ![Three columns labeled CV, Go, and Atari Games compare manually designed features and heuristics with CNNs, AlphaGo Zero, and deep reinforcement learning.](https://ai.engineer/api/talk-essay-media/9d3bfa3f05188c65aaf77c97cd58f0f085ed997eae28623368f60ef9ff0a78dd) Computer vision, Go, and Atari examples contrast hand-engineered methods with learned approaches. ### An arbitrary CSV becomes an onboarding checklist [3:58](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=238s) Ramp’s finance platform covers expenses, payments, procurement, travel, and bookkeeping. Automating those workflows often means interacting with other systems, including legacy systems whose data does not arrive in a convenient format. The switching report is one such integration problem. A customer joining Ramp uploads transactions exported from another card provider. Ramp wants to turn those transactions into a checklist that helps the customer move spending onto its platform. The difficulty is that the incoming CSV can have an arbitrary schema: different providers arrange and name their fields differently. The agent’s job is to transform that input into a format Ramp understands. The checklist is simple; reliably interpreting its source data is the hard part. ### First, write a parser for every vendor [5:30](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=330s) The first approach is conventional integration work. Sengottuvelu proposes supporting the fifty most common third-party card vendors: collect their CSV exports, inspect each schema, and write a dedicated mapping into Ramp’s format. For known inputs, this can work well. But the supported formats are bounded by the integrations engineers have implemented. When a provider changes its export, a parser may break and someone must repair it. The ongoing cost is maintaining knowledge of external schemas in application code. ### Then, let a model identify the columns [6:01](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=361s) The second approach introduces inference inside the existing pipeline. Instead of specifying every provider’s mapping, conventional code can call a model or use embedding similarity to classify incoming columns. Is this a date, a transaction amount, a merchant name, or a user’s name? Once those roles are identified, the pipeline maps the columns into the target schema. This is more general than maintaining a parser for each vendor, but **conventional code still controls the workflow**. The model supplies an answer to a predefined subproblem, and the program resumes execution with that answer. Most computation remains in the fixed pipeline; model inference occupies selected steps. ### Give the model the file, tools, and a verifier [6:57](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=417s) The third approach gives the model control over how to solve the transformation. Its working environment includes: - **The input CSV:** the model can inspect the head, tail, or whichever rows it needs. - **A code interpreter:** it can write Python using pandas or other available packages to perform the transformation. - **A target format:** the required output is a CSV with a specified structure. - **A verifier:** a unit test or other check lets the agent assess whether its output meets the requirement. The application specifies the desired result and a way to check it, while the model chooses the inspection and transformation steps. Sengottuvelu reports that a single attempt was unreliable, while fifty parallel attempts worked well across many CSV formats. Parallel attempts provide more opportunities to find a working transformation, with the verifier supplying a check on the result. The talk gives no success percentage, evaluation dataset, or procedure for selecting among candidates, so this is an implementation report rather than a reproducible benchmark. ![Slide titled “Approach 3: Flexible Agent with Code Interpreter & Verifier,” with explanatory bullets and a flowchart from CSV upload through parallel processing and verification to structured output.](https://ai.engineer/api/talk-essay-media/93422265dc1cb64de7d66fd1c26cd50a91eccfc8a56a5b4649c84bd8b00923da) A flexible CSV agent combines a code interpreter, parallel attempts, and a verifier. Sengottuvelu estimates roughly 10,000 times the compute of vendor-specific parsing, with a probable cost below one dollar for this scenario. Those are tentative figures from the talk, not a current pricing promise. His economic argument is that engineer time is scarce, and a failed CSV import can cost Ramp more in lost business value than the additional computation. A cheaper execution path is not necessarily a cheaper system to build and operate. ### Which system decides what happens next? [8:10](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=490s) The three architectures differ in who controls execution. In the talk’s diagrams, black arrows represent conventional computation and blue arrows represent model computation. | Architecture | Controller | Role of model inference | | --- | --- | --- | | Vendor-specific parsers | Conventional code | None | | Column-mapping agent | Conventional code | Answer selected classification questions | | Flexible agent | Model | Choose when and how to invoke code | In the second architecture, code calls the model. In the third, the model calls code: it decides when Python or another deterministic operation is useful. The parallel version branches into multiple attempts rather than following a single model-driven path. This distinction extends beyond CSV parsing. A backend accepts a request—such as a read, update, or other CRUD operation—performs the necessary work, and returns a response. Conventionally, programmers specify the execution path. A model-controlled backend preserves that request–response boundary while delegating more of the decisions inside it to the model. Sengottuvelu reports that parts of Ramp’s codebase are moving toward the third approach. His expectation is that delegating more work to models exposes more of the application to future capability improvements funded by model developers. The blue arrows may get better without the application team rewriting each individual workflow. That is the proposed architectural advantage, rather than a guarantee that every model upgrade automatically improves the whole application. ### The model executes the backend [10:34](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=634s) Push that architecture further and consider a webmail application. In the conventional version, a server sends JavaScript, HTML, and CSS to the browser. The browser renders an inbox. Clicking an email causes the frontend to request its contents from the backend, which queries storage and returns the result. An LLM might have helped an engineer write that application, but once deployed, its execution is still conventional computation. The alternative places the LLM in the execution path of each request. **The model is doing backend work at runtime**, using a code interpreter, network requests, and potentially database access. To explore that idea, Sengottuvelu introduces an experimental mail client and invites the audience to send messages to a test address. Regular JavaScript still connects the browser to the LLM; the experiment changes who decides what to execute, rather than eliminating all application code. ### An inbox becomes a model conversation [13:59](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=839s) After login, the client takes time to respond. Sengottuvelu explains that the Gmail token is being sent into an LLM chat session with code-interpreter access. The model is instructed to simulate a Gmail client and render a reasonable homepage using that access. The result is a Markdown inbox containing messages sent by audience members; he clarifies that Markdown rendering is probably specified in the instructions. This describes the experiment’s credential handling, not a validated security design. Opening a message follows a different path from a conventional application-specific route: 1. The user clicks the inbox entry labeled **Hello from California**. 2. The client sends the clicked text and its identifier to the model. 3. The model receives the interaction and gets another opportunity to render the page. 4. The resulting page displays the email body. Sengottuvelu infers that the model probably makes a `GET` request to retrieve the message. The visible result is the rendered email; the underlying request is not independently established by the demonstration. The model is effectively choosing the next interface in response to an event, much as a web framework updates a view. The opened message also has actions for marking it unread or deleting it. These controls illustrate how the model can supply interface elements beyond the initial inbox. Sengottuvelu tries the delete action, but displaying a control and clicking it do not establish that the corresponding mutation completed. ![Dark browser page showing an opened email with the message “Hello what is this agent going to do? I’m watching you live,” a reply field, and delete, mark unread, and archive controls.](https://ai.engineer/api/talk-essay-media/09264c64374d911210bc2665c176058167f99588f39590e1cd576c461b627347) The mail client displays “Hello from California,” a reply field, and email actions. ### A slow experiment, an open future [15:34](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=934s) The final interaction remains slow, and the model continues trying to respond to a click without a clearly established successful outcome. Sengottuvelu describes this kind of software as barely working today. Unlike the production switching-report case, the mail client demonstrates an experimental direction, with latency and reliability still conspicuous. Its future depends on whether improving models make runtime delegation practical for more of an application. That would extend the same progression seen in CSV parsing: from manually specified behavior, through narrowly delegated inference, toward models deciding which operations to perform. Sengottuvelu leaves the destination unresolved. Whether more software will look like this remains an open question. ## Resources - [Jsonformer](https://github.com/1rgs/jsonformer) — Generate structured JSON by filling fixed syntax tokens and using a language model for content, with installation instructions and examples. - [The Bitter Lesson](https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf) — Rich Sutton's essay on why general methods based on search and learning benefit from increasing computation. ## Slides ### About me The Ramp AI lead began by building customer-support chatbots, writing code around early models and deleting some of it as the models improved. [0:39](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=39s) [Introduction listing the speaker’s work at Ramp, Cohere.io and Jsonformer, and experience with LLMs.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf08b80ad58da3e128bdb6bc8021cdb49a95986c3c75aa97521db3392269a426.webp) About me ramp Cohere Y Head of Applied AI at Ramp Co-founder of Cohere.io, Early AI Agent company Built Jsonformer Fighting with LLMs since 2021 Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 ### Agenda: One idea The talk moves from the Bitter Lesson and agents to a production agent at Ramp, three ways to build it, and a demo intended to challenge how we think about future software. [2:05](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=125s) [Agenda covering the Bitter Lesson, a production agent at Ramp and a demo.](https://assets.ai.engineer/sites/aiecode2025/sha256/a4220975e15b4757454ab6d69c204f2d0088b760cc827f9b9ca50d8bba062995.webp) Agenda One idea Bitter Lesson and how it applies to Agents A Production Agent at Ramp, and three ways of building it Demo to push you to think Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:04 PM ### Systems that scale with compute A system that can use more compute to think longer tends to beat a rigid, fixed system. The speaker presents this as a practical conclusion from the Bitter Lesson. [2:26](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=146s) [Slide stating that systems that scale with compute beat systems that do not.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ef515926250d005d2f9bfa13c6dbdcd91f1c6357d78be88445bb83ba0e43e3e.webp) Systems that scale with compute beat systems that don’t Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 ### Exponentials are rare The speaker urges builders to ride rare exponentials by choosing systems that improve with more compute. [2:52](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=172s) [Projected slide says to take advantage of rare exponentials and the occasional free pass.](https://assets.ai.engineer/sites/aiecode2025/sha256/a98f5044c54bae411f440895e29a178ae6516533c72bc0052c16fe92735bbd5b.webp) Exponentials are rare, when you find one, just hop on for the ride. Sometimes, you really do get a free pass. Other visible text: AGENTS AT WORK AI Engineer ### Computer vision, Go, and Atari The speaker argues that general methods win as compute scales, even if engineered systems win at fixed compute. The slide contrasts CNNs, AlphaGo Zero and DQN with hand-built approaches. [3:18](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=198s) [Three columns compare hand-engineered approaches with neural networks and reinforcement learning.](https://assets.ai.engineer/sites/aiecode2025/sha256/51801729386f0cc6f254231e40688a00d9b6713f41bccd2b5130885daef952cd.webp) CV Earlier computer vision systems that relied on hand-engineered features (such as SIFT or HOG) and a series of manually tuned processing steps. Deep convolutional neural networks (CNNs) that learn feature hierarchies automatically from large labeled datasets. Go Early computer Go programs that depended on Monte Carlo tree search augmented by human-designed rules and heuristics. Systems like AlphaGo Zero, which rely on deep reinforcement learning and self-play to discover strategies without prior human guidance. Atari Games Early approaches to playing Atari games often relied on hand-crafted features and domain-specific heuristics. These methods required experts to design representations of the game state and tailor strategies for each game. Deep Reinforcement Learning systems, like Deep Q-Networks (DQN), learn directly from raw pixel inputs and game scores. Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:05 PM ### Ramp product overview Ramp’s card sits among tools for expenses, bills, procurement, travel, and business accounts. The speaker describes AI automating finance and employee tasks across the platform. [4:01](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=241s) [Ramp card surrounded by panels for expenses, bills, procurement, travel, and business accounts.](https://assets.ai.engineer/sites/aiecode2025/sha256/45c6546e82a5006eb785dda0ec187de108d9915d78d4bc54fe1939cd82b2d405.webp) ramp Expenses that submit themselves. Modern expense management designed to reduce manual processes. Bills that get processed in seconds. Smart accounts payable that eliminates repetitive tasks. Procurement that runs without delays. Intake-to-pay simplified and automated every step of the way. Travel that's always in-policy. Corporate travel policies built into a consumer-grade booking experience. Business accounts that earn more. Business and investment accounts that maximize returns and minimize work. AIE AGENT ENGINEERING HTTPS://AI.ENGINEER Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:06 PM ### Switching Report Agent The agent parses third-party CSVs into a structured checklist of transactions, helping customers move their transaction history to Ramp—even when the input schema varies or the file is large. [4:39](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=279s) [Slide explains how the agent converts third-party CSV transaction data into a structured checklist.](https://assets.ai.engineer/sites/aiecode2025/sha256/80d3f33c494ee30bebe5a0dad1b999d11fc77aa404d04b16d0af9638ce9ba4e8.webp) Switching Report Agent Why We Need It: Companies must run their businesses efficiently; many expenses reside on third-party platforms. The Switching Report Agent helps migrate transacting history to Ramp, enabling companies to consolidate and streamline operations. How It Works: Users upload a CSV file from a third-party platform. The agent parses the CSV and produces a checklist of items (e.g., user’s name, merchant name, total dollar amount, date) based on a required schema. The Goal: Transform arbitrary CSV input into a predictable, structured output—even when the CSV is very large. Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:06 PM ### Approach 1: Over-Engineered Code Manually supporting 50 third-party platforms could mean maintaining 10,000+ lines of code. A platform changing its CSV format could break its parser. [5:32](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=332s) [Slide describes the maintenance and fragility problems of manually coding many CSV formats.](https://assets.ai.engineer/sites/aiecode2025/sha256/440e1798a0737d11f756c0346fbc5beb8318ec5dd2fe97af28823b06f0e08b6b.webp) Approach 1: Over-Engineered Code • Manually code for every possible CSV format (imagine supporting 50 different third-party platforms). • Results in thousands of lines of code (e.g., 10,000+ lines) that are hard to maintain. • Fragile: A minor format change can break the system. Koding Other visible text: AGENTS AT WORK [5:38](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=338s) [Slide describing the maintenance burden and fragility of manually coding for many CSV formats.](https://assets.ai.engineer/sites/aiecode2025/sha256/8fbc6544689026f8a36b05870553b6b1a615db8f0cd4c28d835174fe361de9f1.webp) Approach 1: Over-Engineered Code • Manually code for every possible CSV format (imagine supporting 50 different third-party platforms). • Results in thousands of lines of code (e.g., 10,000+ lines) that are hard to maintain. • Fragile: A minor format change can break the system. Koding Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:07 PM ### Approach 2: Constrained Agent The agent classifies CSV headers using vector similarity, maps them to expected fields, and routes detected edge cases through a fallback before producing structured output. [6:07](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=367s) [Slide describing CSV column classification and mapping, with a flowchart for handling edge cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab62905931c55509c37cd00b4e0334fbe430fd84179e72b3cd865ffd77d53636.webp) Approach 2: Constrained Agent • Use an LLM to classify each CSV column based on header names. • Leverage vector similarity to map columns to expected fields. • More scalable than manual coding—but still vulnerable to edge cases (e.g., split names or currency symbols). Flowchart: CSV File Uploaded Extract CSV Header LLM Classifies Columns (using vector similarity) Map Columns to Expected Fields (First Name, Merchant, Amount, Date) Check Edge Cases Detected? (e.g., split names, currency symbols) Yes → Handle Exceptions / Fallback → After Handling → Produce Structured Output No → Produce Structured Output Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:08 PM ### Approach 3: Flexible Agent with Code Interpreter & Verifier The LLM writes code to process the CSV, then a verifier selects an output from parallel runs. The speaker says one run failed, while 50 parallel runs were much more likely to work across different formats. [6:59](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=419s) [Slide describing parallel CSV processing with a code interpreter and a verifier that selects an output.](https://assets.ai.engineer/sites/aiecode2025/sha256/804d496a33c7c38d80a00c62d5b55e1239ac2da2372a1b63efca85785864d81b.webp) Approach 3: Flexible Agent with Code Interpreter & Verifier • Leverage a code interpreter for dynamic CSV processing. • The LLM is given the CSV’s header and asked to “just go do it.” • Added Mechanism: • Run this approach in parallel (e.g., 50 times) and use a verifier to select the successful output. • Although it may seem less reliable at first, the parallel approach and verifier make it extremely robust. Flowchart: CSV File Uploaded ↓ Extract CSV Header ↓ Pass CSV to Code Interpreter-enabled LLM ↓ Run Multiple Instances in Parallel (e.g. 50 runs) ↓ Dynamic CSV Processing (per instance) ↓ Aggregate Results Verifier Evaluates Outputs Select Best Result ↓ Select Best Output Produce Structured Output Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:08 PM ### Comparing Three Approaches Compute rises from 1x for over-engineered code to 100x for a constrained agent and 10000x for the flexible agent, which needs parallel runs and verifier logic. The speaker values engineer time over the extra compute. [7:46](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=466s) [Table compares over-engineered code, a constrained agent, and a flexible agent with a code interpreter and verifier, including tradeoffs and relative compute.](https://assets.ai.engineer/sites/aiecode2025/sha256/63f4d51bb9a1756c380d0afcf0f55f1b52a0b013a5c2998b9b4ea8f7fb6c82e0.webp) Approach | Pros | Cons Over-Engineered Code | Precise control; domain-specific tweaks | Brittle, 10K+ lines of code; hard to maintain | 1x compute Constrained Agent | Automates column classification | Fixed schema; struggles with edge cases | 100x compute Flexible Agent with Code Interpreter & Verifier | Highly adaptable; scales with compute; dynamic processing | Needs parallel runs & verifier logic | 10000x compute AIE AGENT ENGINEERING HTTPS://AI.ENGINEER Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:09 PM ### Classical and Fuzzy Computation Over-engineered code stays in classical computation. The constrained agent calls fuzzy computation from classical code; the flexible agent mostly uses fuzzy computation and writes code when it needs classical computation. [8:27](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=507s) [Three request-to-response diagrams compare over-engineered code, a constrained agent, and a flexible agent; black arrows denote classical computation and purple arrows denote fuzzy computation.](https://assets.ai.engineer/sites/aiecode2025/sha256/366827954d94b1083f2a5f75ebc03666b6e42b3c20d2f1fe7b3b6432b340453b.webp) Over-Engineered Code Request → Response Constrained Agent Request → Response Flexible Agent with Code Interpreter & Verifier Request → Response Fuzzy Land Classical Land Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:10 PM ### Parallel Flexible-Agent Runs The speaker proposes running the flexible agent 50 times, shown as many paths from request to response. Most computation is fuzzy; the agent writes classical code when it needs to. [9:11](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=551s) [The three approach diagrams now show many parallel paths for the flexible agent, illustrating multiple runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a2dc6770e6ffb1fc6c443840103dd017037fbfaf573bf01c710d0f72efb8459.webp) Over-Engineered Code Request → Response Constrained Agent Request → Response Flexible Agent with Code Interpreter & Verifier Request → Response Fuzzy Land Classical Land AIE AGENT ENGINEERING HTTPS://AI.ENGINEER Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:11 PM ### This idea generalizes The speaker takes the idea further, beyond the agent scaffolding discussed so far. [10:35](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=635s) [Purple slide stating that the idea generalizes.](https://assets.ai.engineer/sites/aiecode2025/sha256/16ebb7982cd0cbbb1aa056a7cb204b38337a26e08a59c7bcbf17d20181ffd047.webp) what’s the point? this idea generalizes Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/[illegible]/Untitled?node-id=1-337 Fri Feb 21 3:12 PM ### Traditional web app and genweb The diagram contrasts a traditional browser, frontend, backend and database with genweb, where a language model simulates the app layers and exchanges user interactions and HTML with the browser. [10:44](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=644s) [Diagram comparing a traditional browser, frontend, backend and database with a browser connected to a language model simulating those components.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c4c196784ed5a6b27b6676f7d1d34c7c473a59d2f2e546a9cc6f9e8c024c722.webp) Traditional Web App Browser Frontend Frontend Framework Backend Backend Framework Database genweb Browser User Interactions HTML Large Language Model Simulated Frontend Framework Simulated Backend Simulated DB Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Audience · Untitled - Figma Slides figma.com/deck/AUGGvTyBjpQlwwuLwa0KgC/Untitled?node-id=1-337 Fri Feb 21 3:12 PM ### App verification warning During the email client demo, Gmail login opens a Google warning that the app is still being tested and has not been verified. [13:09](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=789s) [A Google warning says the email client app has not been verified.](https://assets.ai.engineer/sites/aiecode2025/sha256/bd7bc0bff7ee05d992f342fd85f74754410da2baa3e6bbd4f8700a10acc8f0af.webp) Login with Gmail Google hasn’t verified this app You’ve been given access to an app that’s currently being tested. You should only continue if you know the developer that invited you. Continue Back to safety English (United States) Help Privacy Terms Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Fri Feb 21 3:14 PM Presenter · Untitled - Figma [illegible] LLM Email Client Genweb App Creator ramp--software-distillation-email-client-serve-index.modal.run Sign in - Google Accounts accounts.google.com/signin/oauth/warning?authuser=4&part=AJ... ## Full transcript [0:00](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=0s) [on-hold music] So a little bit about me. [0:18](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=18s) Um, head of applied AI at Ramp. I've been working on LLMs for four years, which is-- [0:26](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=26s) Well, which is, uh, kind of a long time, I guess, uh, in LLM land. Everything started happening really when ChatGPT came out. Um, so I was trying to build what people would now call an AI agent company. [0:39](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=39s) Back then, we were just doing customer support. We were trying to make our chatbot smarter, and we were trying to figure out what, what models to use to, or what tech to use to get them to respond to customers better. [0:49](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=49s) And we were messing with GPT-2 and BERT, and models were so frustratingly stupid, and the context windows were small, and they were not very smart at reasoning, and it was just incredibly annoying. [0:58](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=58s) And we just wrote lots of code around these models to get them to work at le-at least somewhat reliably. [1:05](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=65s) And along the way, as models got smarter, just kinda had to delete more of that code and just ended up seeing a lot of patterns in what, what code needs to get deleted, how to build agents, in what ways that will scale with more intelligence. [1:19](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=79s) And clearly, we're gonna continue to get a lot more intelligence. And I just wanted to, uh, maybe talk about a single idea throughout the talk, uh, through various examples. [1:27](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=87s) Uh, we'll, we'll do some, uh, some, uh, setting, but I'll also have a bunch of demos to kind of, like, drive home the point, and maybe I can convince you guys that, uh, there's a certain way of building agents that's slightly better than other ways. [1:41](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=101s) I also built a structured extraction library called jsonformer. Um, I think it was the first one. I don't-- I'm not fully sure, but timing-wise, it was before all the other major ones. [1:51](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=111s) Um, and that was also scaffolding around a model. Models were too stupid to output JSON, and we were just really begging it, pleading, pleading it, and forcing it to, uh, act in ways that we want it to be. [2:02](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=122s) So, as I said earlier, just have a one core agenda item here, which is wanna convey one idea. Uh, we'll start off. All, all of you have probably read the essay, The Bitter Lesson. [2:11](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=131s) Just quickly go through what it is. Uh, we'll go through a production agent we have at Ramp and how it works, and three different ways of architecting it. And then I have a demo that to really push maybe how we all think about how software and backends and things will work in the future. [2:27](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=147s) So very simply, the idea is just that systems that scale with compute beat systems that don't. So there's two systems, and, uh, without any effort, the sy- the sy- one of the systems can just think more or use more compute in some way. [2:42](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=162s) That system tends to beat systems that are rigid and fixed and just deterministic. [2:47](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=167s) So from that idea, it's pretty clear, like, if you're building systems, you might as well build systems that improve with more compute. And this, this seems pretty obvious, like, obvious conclusion from The Bitter Lesson. [2:59](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=179s) Taking it a step further, why is this true? It's because exponentials are rare. Like, they just don't exist, and most things in the world aren't exponential. So when you find one, you just should hop on, strap on, and just take the free pass and go for the ride, and you probably shouldn't try too hard. [3:13](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=193s) And there's a lot of examples from history that, um, kind of reflect this. So for, for chess and Go and computer vision, Atari games, like, people have tried to build lots of systems and written a lot of code. [3:24](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=204s) And my way of thinking about rigid systems is just, like, spending a lot of time, grinding weekends, and writing very clever software, well-abstracted, um, maybe trying to synthesize human reasoning and thought process into features, and then using them in clever ways and trying to approximate how a human would think. [3:42](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=222s) And if you actually fix the amount of compute, that approach will win. But if you just-- Turns out, if you end up scaling up how much search you're doing, the general method al-al-al- always ends up winning, even, uh, like, in all these cases, so Atari, Go, and computer vision. [3:58](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=238s) A little bit about Ramp. So Ramp is a finance platform that helps businesses manage expenses, payments, procurement, travel, bookkeeping more efficiently. And we have a ton of AI across the product, so automate a lot of boring stuff the finance teams do and employees do with, uh, submitting expense reports and booking your flights and hotels and, uh, submitting [4:17](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=257s) reimbursements, all that. And so a lot of the work behind the scenes is just we're interacting with other systems, um, and helping, like, legacy systems and helping employees get their work done faster. [4:28](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=268s) So let's actually talk through one of the systems we have today at Ramp and, um, maybe some-- talk through, like, the different versions of the system and how it evolved over time. [4:37](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=277s) So we're gonna talk about something called a switching report. It's a very simple agent. All it needs to do is take in a CSV, a CSV arbitrary format. So the schema could be re- seriously anything from the internet. [4:50](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=290s) And we want these CSVs to come from third-party card providers. So when people onboard to Ramp, we wanna give them a nice checklist and say, "Hey, here are all the transactions you have on other platforms, and we wanna help you move them over. [5:01](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=301s) And the more transactions come on Ramp, the more we can help you, and the more you'll use our software, and the more everyone benefits." And so the switching report is just really a checklist. [5:09](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=309s) But to read people's CSV transactions, we need to understand those, and other platforms have all these kinds of crazy schemas. And so the, the description of the problem we have here is just for an arbitrary, arbitrary, like, CSV, how can we support, um, parsing it and then into some format that we, we understand? [5:30](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=330s) So let's just start with the, the simple approach, right? Is like, let's just take the fifty most common third-party card vendors, um, and let's just manually write code for all of them. [5:39](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=339s) Now, obviously, like, this, this will just work. It is some work, not a lot of work, but [5:46](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=346s) you still have to maybe go to fifty different platforms and download their CSVs, see what schemas they have, and then write code. Maybe if they decide one day they change their format, your thing will break, but that's okay. [5:56](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=356s) You'll get paged, and you can wake up and, and go fix it. [6:01](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=361s) So let's maybe introduce some LLMs in here. So from the over-engineered code where you ended up writing a hundred thousand lines, maybe we don't, we don't-- we want a more general system. [6:11](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=371s) So let's introduce a little bit of LLMs, a little bit of AI in here. And so in the deterministic flow, let's maybe add some, or just like scripting in classical scripting land, let's add some more, um, calls to OpenAI, or you have an embedding model, you wanna do s- uh, semantic similarity or something like that. [6:28](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=388s) So then let's just take every column in the CSV that comes in. Let's try to classify what kind of column it is. Is it a date? Is it a transaction? [6:35](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=395s) Uh, is it a transaction amount? Is it a merchant name, or is it the, uh, user's name? [6:39](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=399s) And then we map it on, and then we probably could, uh, end up in a schema that we're happy with. Again, most of the com- compute is running in classical land. [6:49](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=409s) Some of it is la- running in fuzzy, like, LLM land. But this is somewhat looking like a more general system. [6:57](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=417s) Let's go maybe a different approach when, like, we just go all the way through. Let's just say, we're just gonna literally give the CSV to LLM and say, "You have a code interpreter, so you can write whatever code you want, pandas or all the faster Rust-based ones. [7:11](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=431s) Um, you have all these Python packages. Um, you're allowed to look at the head of the CSV, the tail, whichever rows you want. Um, and then I just want you to give me a CSV, uh, with this specific format, and here's a unit test, here's a verifier that you can use to tell if it's working or not." [7:28](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=448s) Turns out this approach actually doesn't work, like, we tried it, um, if you only run it once. But instead, if you run it fifty times in parallel, it's actually very likely that it works really well and generalizes across a ton of different formats. [7:42](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=462s) The amount of compute here is actually probably, like, what is that number? Ten thousand times more than the, the first approach we came up with. But again, like, what is truly scarce in the world is engineer time. [7:52](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=472s) Maybe not for-- not in a while, but at least today. And we'd rather have a system that works really well, and even with the ten thousand times more compute, it will probably cost less than a dollar. [8:01](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=481s) And every transaction that's switched over, every failed CSV will cost Ramp way more money than whatever money we spend on this exact, this exact architecture. [8:10](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=490s) So this is a very specific, uh, example. It's like, how does this apply to the agents that we all build and maybe the systems we're all working on? Turns out something like this actually generalizes. [8:21](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=501s) So if you're looking at three approaches, and let's assume, like, the black arrow is just classical compute, and then the blue arrows are fuzzy land. So it goes into neural net, and all, all sort of weird matrix multiplication happens, and then we're in latent space, and it gets all alien intelligency, and then comes back to classical land. [8:39](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=519s) First approach, there was no AI. We just wrote code, and it just worked, mostly. The constrained agent, so the second approach, we broke into fuzzy land from classical land when, when we decided we wanted similarity scores or something like that. [8:53](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=533s) And then the third approach is actually flipped, where the LLM decides it needs to go into classical land. So it writes some code, writes some pandas or, uh, Python code, and it decides to break in into this classical land when it needs to, but most of the compute is fuzzy. [9:09](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=549s) Actually, this is maybe not the most accurate graph, like, uh, because I proposed that we run it fifty times. It more so looks like this. But if you look at a back end in general, they're all request response. [9:21](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=561s) So some sort of message is going in, it's like a POST request, or GET, or update, or read, any sort of CRUD operation. And we're a- really just asking the back end to take this piece of information, do whatever you must with it, run whatever mutations you want, and return me a response. [9:36](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=576s) And almost all systems we've built so far is like humanity, I guess, like, look like the first one. But more people are using OpenAI. OpenAI makes billions of dollars, and probably a lot of the systems that use them look like number two, where just regular, uh, programming languages are calling into OpenAI servers, and we're running some fuzzy [9:54](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=594s) compute. What we're seeing in, like, more and more parts of the Ramp code base, we're moving to the third approach because it just tends to work well. Because all the blue arrows, if you did nothing, absolutely nothing, we all went to vacation for the next year, the big labs are still working and spending billions of dollars making [10:13](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=613s) those models better. And so the blue arrows will get better. And so how much blue arrow you're using in your code base actually will help directly your company without much effort from your end. [10:23](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=623s) So this is what I was saying is, like, the bitter lesson is just so powerful, and exponential trends are so powerful that you can just hitch, hitch a ride. [10:34](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=634s) Let's, um, let's take this idea, like, further. [10:40](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=640s) Um, let's actually, like, go all the way, like, something, something crazy. [10:45](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=645s) Um, on the left, you'll see a traditional web app. So usually, the way it works is you open, um, gmail.com, and some, uh, static file server and, and Google sending you a bunch of JavaScript and HTML and CSS. [11:00](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=660s) Your browser renders that, um, and shows you some nice UI, nice HTML that's user-friendly. Maybe you see some emails, maybe you click on one of them. Um, the front end makes a request to the back end, the back asks the front end-- The front end asks the back end, "Give me the content for email," and whatever ID [11:17](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=677s) it is, and then the back end hits the database and gives you the result. And maybe they use codegen, maybe they use all the codegen tools available to make Gmail. [11:25](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=685s) So that, that was probably-- The LLM only worked when the software engineer was writing the code. But once the code is written, and it's, like, pushed to production, it's just classical compute. [11:37](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=697s) And on the right, I'm actually proposing a different model, which is the back end is the LLM. It's not codegen. It's this LLM is doing the execution. It is the back end. [11:47](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=707s) So the LLM has access to tools like code interpreter and potentially has access to, um, through that, making requests, network requests, and also has an access to, uh, DB. [12:00](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=720s) So I have a mail client actually that works with this principle, and this is my test email. So if y'all wanna see any emails you send to me- In a minute or so, you can send me an email. [12:12](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=732s) But please be nice. All right, I think, um, [12:38](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=758s) that's probably enough time. So I'm gonna go over... [12:57](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=777s) So we have this email client. I mean, we still have some regular JavaScript to hook into the LLM, hook the LLM into the browser. But when I do log in, I'm gonna use my email as just showing you... [13:16](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=796s) Hmm. Oh. [inaudible] [13:40](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=820s) What? Oh, it's probably- Yeah. Okay, we're good, we're good. All right, we're saved. [13:49](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=829s) I think. Thankfully, we have a room full of engineers. [laughing] [13:59](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=839s) So there's a dot, but the reason it's so slow is because when I open this page and log into Gmail, the Gmail token is actually being sent to an LLM. [14:07](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=847s) We're saying literally this is a L- LLM chat session. What we're, we're seeing on the screen is like, "Hey, LLM, you're, you're actually simulating a Gmail client. You have access to," oh, all the emails. [14:19](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=859s) Uh, "you have access to, um, Rahul's, uh, Gmail token and a code interpreter. And so just render some UI based on, uh, what you think is reasonable for the homepage for a Gmail client." [14:31](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=871s) And so looks like it decided to render as markdown. Uh, I think we actually tell it to render as markdown. And it's rendering all the emails that a bunch of people sent me from here. [14:40](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=880s) So looks like it says, uh, "Hello from California." So I'm gonna click on that. [14:45](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=885s) When I click on that, we're actually not running, um, any, like, backend calls or anything like that. We're just telling the LLM the user clicked on that piece of text. [14:52](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=892s) In this case, it was, "Hello from California," and the ID number. So the LLM now has the information on what the user clicked on, and it has the chance to re-render the page, much like a web framework would. [15:02](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=902s) So again, it goes back, it probably hits, uh, a GET request for that specific email and pulls the body. [15:09](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=909s) "What is this agent gonna do? I'm watching you live." So the LLM just decided this is the appropriate, uh, UI for a Gmail client. [15:18](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=918s) And also, I have, uh, other features the LLM thought was reasonable, so it looks like I could mark it as unread or, or delete the email if I want to. [15:25](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=925s) Uh, maybe I'll delete it because it's not that good of an email. I'm sorry. [laughing] [15:34](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=934s) It, it is very slow because we're doing a lot. But wanted to push you in this direction because this kind of software barely works. [laughing] [clapping] [15:43](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=943s) Dang. I guess not. Um, also I clicked on it and now the LLM is trying to do something with me clicking on it. But anyway, um, this kinda software barely works today, and it doesn't mean it won't, won't work in the future, uh, but with exponential trends, like, things might just-- like this might just take off. [16:03](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=963s) Um, so just wanted to push you all to think in this direction. Um, yeah. Will software, more software look like this? [upbeat music] I don't know. We'll see. Thank you. [clapping]