AI Engineer Europe 2026
Teaching Coding Agents to do Spreadsheets
Read the talk
Teaching Coding Agents to Work with Spreadsheets
A persistent JavaScript REPL made spreadsheet operations easier to compose, but reliable calculation, rendering, and evaluation made the results worth trusting.
From a talk by Nuno Campos
Before you start: Familiarity with spreadsheet formulas, basic JavaScript, and agents that call tools will help you follow the implementation choices.
What’s the revenue?
What revenue does this spreadsheet report? Answering that question requires more than finding a cell containing a number. Nuno Campos’s team spent four months trying to make coding agents as capable with spreadsheets as with Python or JavaScript. The obstacle starts with how a workbook communicates meaning: a person opening Excel can immediately distinguish a revenue table, an assumptions area, and a chart from their positions and formatting. An agent must recover that structure before it can interpret the contents.
Even after locating revenue, the agent has to distinguish gross from net, identify the relevant quarter or year, and determine whether the cell contains an input or a formula. These are different kinds of uncertainty: what the user means, where the answer lives, and how the workbook derives it. Spreadsheet understanding depends on structure and computation as well as values.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Planning helped; rigid discovery did not
An early design divided the work among three agents. At its center, an edit agent followed a five-step process that included defining the desired end state, planning, executing, and verifying. This changed where failures happened. Without the process, mistakes appeared while the agent constructed a financial model. With it, mistakes could appear in the plan, where they were easier to correct.
The architecture nevertheless proved too rigid. Discovery happened once at the beginning, so the system could not return to exploration when editing exposed a new question. Context also failed to flow adequately between the agents. The useful part was thinking before acting; the failed part was fixing discovery and execution into stages that could not adapt to one another.
The team then explored spreadsheet representations. Familiarity alone did not make a format sufficient: SQL was well represented in model training and designed for structured data, while XML reflected how Excel files were stored. Neither worked adequately as the standalone interface. Two other approaches survived as capabilities inside the eventual system.
| Representation | Appeal | Outcome in this work |
|---|---|---|
| SQL | Familiar structured-data operations | Insufficient alone |
| XML | Excel’s on-disk representation | Ineffective as the interface |
| CSV/TSV | Compact views of selected cells | Frequently useful within a larger system |
| HTML | Layout and formatting | Led toward image rendering |
A CSV or TSV view of a selected region remained useful for inspecting data, even though it could not carry the entire interaction. HTML made layout and formatting more explicit, leading the team to build a rendering engine that could show the agent a spreadsheet range as an image. The successful design accumulated complementary ways to inspect a workbook instead of choosing one representation for every task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn tool calls into composable JavaScript
The largest interface change was replacing approximately fifteen separate tools with one Node.js REPL tool. The operations did not disappear: they became JavaScript functions that the agent could combine inside a single call. Instead of asking the model to orchestrate each operation through a separate tool exchange, the system let it express that orchestration as code.
JavaScript offered a familiar scripting language that the team could sandbox. Campos thought Python would probably work equally well. The spreadsheet implementation itself remained in C#: the scripting language served the agent, while a different language handled the files. Node.js describes the historical architecture discussed here; the companion REPL account records a later migration to QuickJS-Emscripten.
Campos reports that exploring a spreadsheet and reaching an answer previously commonly required ten to fifteen tool calls. Sequential exchanges took time and often timed out. Parallel tool calling did not solve the whole problem because the agent still lacked a way to combine the results within those calls. With the REPL, it could compose several operations in code and return their results together from one tool call. The improvement was composition, not merely concurrency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep state between execution steps
This composition pattern is often called Code Mode: generated code combines multiple tool operations in one call. Campos points to its growing use in the Anthropic API and at Cloudflare. A REPL adds another property: variables survive between calls. The agent can execute a small piece of code, inspect the result, reason about what it means, and continue using the state it already created.
For a small illustration, suppose the agent has already read these two rows from a workbook. The values below are teaching examples, not a reproduced workbook or a Witan API call. In its first REPL call, it keeps the rows available and inspects which revenue labels exist:
javascript
var revenueRows = [
{ metric: "Gross revenue", period: "Q1", value: 1200 },
{ metric: "Net revenue", period: "Q1", value: 1000 }
];
console.log(revenueRows.map(row => row.metric));
After resolving that the question concerns net revenue for Q1, a later call can use the same variable:
javascript
var selectedRevenue = revenueRows.find(
row => row.metric === "Net revenue" && row.period === "Q1"
);
console.log(selectedRevenue);
The second call does not reload or redeclare revenueRows. That is the persistent-state property: execution can pause for interpretation without discarding prior work.
Without REPL semantics, Campos observed agents commonly writing JavaScript scripts of around fifty lines. Those scripts combined substantial work, but committed to a longer sequence before the model saw intermediate results. With persistent state, scripts became shorter, allowing more reasoning between operations. In the team’s observations, that flexibility often produced a better answer faster.
The same interface also simplified adding capabilities. Under the separate-tool design, formula-dependency exploration could require several new tools, new schemas, and attention to how those tools interacted. Under the REPL design, it meant exposing additional JavaScript methods. A TypeScript type definition file placed in the prompt told the agent which methods were available and how to call them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The gains and their evaluation conditions
Campos reports internal financial-analysis benchmark accuracy of approximately 50% before the REPL, 74% after its introduction, and eventually 92% after further changes. Those later changes included better fuzzy search, formula-dependency tracing, system-prompt improvements, and bug fixes. The progression describes the team’s engineering results, not a controlled measurement of the REPL alone: the companion research log uses changing task sets and separately attributes an earlier large improvement to an extraction bug fix.
Under the usual five-minute whole-task limit, Campos reports essentially zero timeouts with the more efficient approach. That is a reported benchmark outcome, not a guarantee for arbitrary workbooks or executions. The operational benefit was that the agent could spend less of its time budget moving through separate tool exchanges.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Calculate and render before trusting the result
Coding agents work better when they can run a compiler, linter, or test suite and respond to the results. Human programmers benefit from the same loop. Spreadsheet agents need equivalent feedback: producing a plausible formula or formatting instruction is not enough to establish that the workbook behaves or looks right.
The team built two engines to supply that feedback:
- Formula calculation: Compute the workbook’s formulas so the agent can inspect their results.
- Range rendering: Produce an image of a selected range, preserving layout and formatting so the agent can inspect its appearance.
Together they support a write–calculate–render loop. The agent makes a change, observes its numerical and visual consequences, and returns to the formula or formatting when the result is wrong.
Verification is only as trustworthy as the engine supplying the feedback. An incomplete calculation engine can turn a correct Excel formula into an apparent failure. The agent writes a valid formula, asks the engine to calculate it, and receives an error because the engine does not implement that operation. It may then try to repair a formula that never needed repair. Campos’s example of an engine supporting only half of Excel’s formulas is hypothetical; the mechanism is the consequential part. Feedback that diverges from the target application can actively make the agent worse.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the interface from the feedback infrastructure
The REPL and the verification engines solve different problems. The REPL is an interface chosen to fit the coding strengths of the models Campos’s team was using. If models become equally capable at mouse-and-keyboard computer use, a different interface could become preferable. That possibility does not remove the need to check the workbook’s behavior.
Calculation and rendering are therefore the more durable investment. Across several model releases during the project, Campos observed that stronger models could get more value from the verification loop. Better reasoning did not make feedback redundant; it made the agent better able to interpret and act on it.
Domain guidance also survived the interface changes. Its purpose was not to teach the model what revenue or ARR means. The model already knows many relevant and irrelevant things; the prompt directs attention toward the knowledge that matters for the current financial task. Campos reports that this guidance improved results across tool iterations, with almost the same prompt working for both individual tools and the REPL.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare spreadsheet behavior, not just judged answers
Reliable evaluation was necessary to decide whether an interface or representation actually helped. The team initially relied entirely on an LLM judge. That supplied a score, but introduced another source of variation: when the score changed, was the agent different, or had the evaluator simply produced a different judgment?
Where possible, the team replaced judgment with deterministic comparison against a golden spreadsheet:
- Select a reference workbook with known input and output locations.
- Put a set of values into its inputs and obtain the outputs.
- Put the same values into the corresponding inputs of the generated workbook.
- Compare the resulting outputs.
This treats each workbook as a black box. The generated spreadsheet need not have identical internal construction to demonstrate matching behavior for the tested inputs. The comparison can be more trustworthy than asking another model to grade the work, but it is not suitable for every task. The companion benchmark account retains LLM grading for QnA text; the headline accuracy progression should not be read as wholly deterministically graded.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A reasoning failure may be a tool failure
Agent failures can originate outside the model. An incorrect example in a skill or prompt may be followed faithfully. A bug in a tool may cause repeated failures, after which the agent keeps retrying or searching for a workaround. From the final conversation, that behavior can look like poor reasoning even when the immediate cause is broken infrastructure.
Inspecting the trace makes that distinction actionable. Follow what the agent was told, what it called, and what the tool returned. The question is whether the model misinterpreted valid feedback or whether the system supplied an incorrect instruction or result. The latter is a defect the application team can fix without waiting for a more capable model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give agents composition and a way to check their work
A long sequence of separate tool calls can become an awkward substitute for a scripting language. Code Mode or a REPL gives the agent an explicit way to combine operations. But composition needs a feedback loop behind it: use existing domain tools when they can check the result, and invest in a calculation or rendering engine when the necessary checks do not exist.
Interface design deserves deliberate work because it changes what the agent can accomplish efficiently. It also deserves revisiting as model capabilities change. The REPL was a strong fit for this system’s models and workload, not a reason to freeze the interface permanently.
The failed three-agent architecture does not invalidate planning before acting. Nor does a powerful model remove the need to focus its domain knowledge or evaluate its outputs. Prefer deterministic checks where they fit, and use an LLM judge where that is the available option. Before attributing persistent confusion to the agent, inspect the traces and the plumbing: sometimes the most effective improvement is fixing the bug the model has been struggling around.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- Teaching Machines to Read SpreadsheetsRepository
Witan's engineering retrospective covering spreadsheet representations, agent architectures, evaluation and the REPL.
- The REPL ToolArticle
Architecture and examples for persistent spreadsheet exploration, typed APIs and composable JavaScript operations.
How Witan evaluates spreadsheet content, layout, formatting, formula behavior and text answers.
An explanation of exposing tools as TypeScript APIs so agents can compose operations in executable code.
Updates since the talk
Current instructions for scripting workbook edits and checking calculation, rendering and lint results.
Read the complete timestamped transcript
- 0:00
[upbeat music] Cool.
- 0:16
Uh, hi, everyone. My name is Nuno, and I wanna talk to you about how, how we spent the last four months teaching coding agents to master spreadsheets.
- 0:27
Um, so, uh, essentially, our goal was to get coding agents to be as good at spreadsheets as they are at, you know, Python, JavaScript or whatever your favorite language is.
- 0:38
Uh, we started at, uh, around fifty percent accuracy on a financial analysis benchmark and got to ninety-two percent. So I'll chat about what actually moved the needle and what didn't and the dead ends and, um, yeah, let's see.
- 0:57
Uh, so spreadsheets are a little bit harder for AI than you might think at first. Um, if you, uh-- if you think about how you would... You know, if you open Excel, uh, how you'd find your way in an Excel file that you don't know, it's actually a very visual thing and you, you know, you just instantly
- 1:15
see the structure. There's a revenue table here, assumptions in there, a chart in there, uh, and it just, you know, feels intuitive, and you don't even think about it.
- 1:23
Uh, and, um, an LLM doesn't really see any of this. It, you know, if you ask it, "What's the revenue?" Then it has to figure out which revenue do you mean.
- 1:34
You know, the net revenue, gross revenue, revenue for this, revenue for that. Uh, which, which, uh, quarter, which year, uh, and then, uh, is the number it found like an actual input?
- 1:46
Is it a formula? Uh, it's-- So it's actually a deceptively hard task.
- 1:53
One, uh, thing we tried close to the beginning was to split the work into three agents. Um, so the, kind of the central one was the edit agent that had, like, a five-step process, um,
- 2:06
uh, that, you know, you'd define the, the end state, you'd do a plan, you'd execute, you'd verify, all, all the things you're supposed to do. And this, uh, kind of changed the kind of errors we got.
- 2:18
Uh, without it, the, the agent would just make mistakes while ac-actually building a, a financial model or something. And with this, it would maybe make those mistakes while planning, which was a lot easier to rectify.
- 2:31
Um, but this architecture in the end was too rigid because, you know, discovery ran once upfront, uh, and then you couldn't revisit it. And, uh, the context wouldn't flow between the different agents.
- 2:42
So it just turned out to be one dead end.
- 2:47
Uh, then some more dead ends. Uh, we, I think, ended up probably trying every conceivable way of representing a spreadsheet to an LLM. Uh, none, uh, really worked as a standalone representation, but two, uh, turned out to be useful as, uh, methods inside the, the REPL that we ended up, uh, creating.
- 3:09
Uh, but they all had something going for them in theory, and that's why we tried it, right? So SQL has obviously been around for decades, so you know, super popular in, uh, LLM training data.
- 3:18
So agents are really good at it. It's supposed to be [chuckles] great way to deal with structured data. Uh, but it turns out that, you know, it doesn't quite work for this.
- 3:28
Uh, XML is how Excel files are represented on disk, so, you know, maybe that was a good idea. It, it wasn't. Um, and, uh, you know, many others. In the end, uh, we did get two, uh, useful things out of this.
- 3:42
One was the concept of, uh, having these CSV or TSV, uh, views of part of a spreadsheet. Uh, this turned out to not be that great as the only way to interact with a spreadsheet.
- 3:55
But as, uh, one piece of, uh, of the larger solution, uh, it turns out to be used very, very often. Uh, and HTML was also a step in the right direction as it, you know, introduced the idea of a layout and formatting and so on.
- 4:09
Uh, so that ended up resulting in us building a rendering engine to let, um, uh, the agent see what the rendered spreadsheet looked like as an image.
- 4:21
Uh, and then, uh, you know, eventually we, uh, hit on, uh, what's-- was probably ended up being the biggest, uh, breakthrough, which was to replace the, you know, the many tools that we accumulated over time.
- 4:34
I think at that time we had around fifteen tools. Uh, with a single, a single tool, which was a, a Node.js REPL. Um, and so, you know, all the, all the fifteen tools that, that we had to start just became different JavaScript functions that, uh, the agent could combine in this one, uh, REPL call.
- 4:54
And why, why JavaScript? Uh, we needed a scripting language that's easy for, uh, to sandbox. It's easy-- LLMs are, you know, uh, very, uh, familiar with it. And, uh, Python would probably work equally well.
- 5:09
We just went with JavaScript. Uh, but the actual implementation of the code that deals with the spreadsheet is actually in a completely different langua-language, in C#. Um, uh, and that's kind of the advantage of this architecture.
- 5:23
You just, you know, use the scripting language for what it's good at, which is letting the agent interact with it, and use the right language to then deal with the actual files.
- 5:34
And what it looked like before and after. So before, it would be you'd have, you know, ten or fifteen tool calls usually for an agent to, uh, to explore a spreadsheet and get to an answer.
- 5:46
Um, and this would actually very often end up timing out and taking a long time because it was just, you know, doing things sequentially. Even parallel tool calling didn't really help 'cause you couldn't combine the results in any way.
- 5:57
And after, you could-- the agent would just combine, you know, the different things it wanted to do in a single tool call and get all the results a-at the same time.
- 6:08
Um, so some of you will be familiar with the idea of Code Mode. Uh, I think that's becoming more popular, uh, showed up in the Anthropic API and Cloudflare has talked a, a bunch about it.
- 6:19
Uh, a REPL is actually-- and the Code Mode is, you know, already super useful, right? 'Cause it's- that's that basic idea of combining multiple tools into a single, uh, into a single tool call.
- 6:30
Uh, but the, a REPL actually goes further, and the difference is it's basically Code Mode with persistent state, so that, uh, you know, the agent calls the REPL tool once, defines a few variables, and then sees the results, spends a few more reasoning tokens, and then the next time it calls the tool those variables all are still
- 6:49
there. So that means, uh, it actually can, uh, you know, build on its work. And what we observed with this is that just pure Code Mode without the REPL semantics, uh, agents would very often write quite long scripts, like 50 lines of JavaScript would be pretty common.
- 7:10
Um, which, which is great, means they're doing many things at the same time. But with a REPL, they would actually write shorter scripts, um, which meant it could, uh, basically do more interleaving of putting reasoning in between each of the things that the, uh, agent was doing.
- 7:27
Uh, which, uh, many times resulted in, uh, the agent getting to a better answer, uh, faster because it was less, uh, static.
- 7:37
And the another nice thing about this design is, um,
- 7:42
in the previous way where we had separate tools, if we, you know, if we figured out, oh, there's a new, uh, a new method we need to give the agent access to, to, I don't know, explore the dependencies between formulas or something.
- 7:54
So that would mean creating several more tools that are gonna go into the tool schema and, uh, we need to see how they play with each other. Uh, whereas with this approach, um, all it means is making a few more methods available in, in the JavaScript REPL and making the agents aware of that is as simple as
- 8:14
creating a TypeScript type d- type definitions file and putting it into the prompt, and that works really well.
- 8:21
And, uh, so the results, uh, out of all of this was, uh, you know, we went from, as I said, uh, 50% before we had the REPL, then 74, and then, you know, over time we made more changes.
- 8:35
None as dramatic as the REPL that eventually got us to 92% on this, uh, internal benchmark we have. And these were changes like giving the agent better fuzzy search or formula tracing, uh, functions for dependencies or improving the system prompt or just fixing bugs.
- 8:53
Um, uh, but you know, it all adds up to a nice result. And, uh, another thing I wanna, uh, call out is, is the timeouts. So, uh, this approach ended up really, uh, you know, fixing the tasks that would time out.
- 9:08
We usually ran tasks with a five-minute timeout 'cause if it takes longer than five minutes to answer a question about a spreadsheet that's not particularly useful. Um, and this approach essentially resulted in zero timeouts because it just was a lot more efficient for the agent to, to do its thing.
- 9:28
Um, there's a lot of parallels between, uh, spreadsheets and, and coding. Um, I'm sure you all use Claude Code or Codex or whatever coding agent, uh, every day and it does a much better job when it can, say, run the compiler for your language or the linter or your tests and then iterate based on those results.
- 9:52
And, uh, you know, when we write code manually that's true as well, right? If we're not allowed to compile or lint or test the code then it's not gonna produce a great result.
- 10:02
And the same is true of spreadsheets and, but to, to enable that for, uh, for spreadsheet work, uh, we had to build, uh, uh, a couple of, uh, engines that, uh, can close that feedback loop.
- 10:14
The two most important ones is one is a formula engine to calculate the, the formulas, and another one is a, a render engine to render the contents of a, of a range into like a, uh, a, uh, an image, uh, with all the formatting and layout and so on.
- 10:31
And that's kind of the source of truth. It's the verification loop that, uh, that makes the agent, uh, you know, confirm that it did the right thing and when it didn't do the right thing go and fix the formula or go and fix the formatting, uh, in order to, to make it correct.
- 10:46
But it only really works, uh, if the engine is actually high fidelity. Uh, so if you use, um, an incomplete engine that implements, say, 50% of the formulas in Excel, then what you end up with is actually worse results because the en- the agent is gonna write a formula that it thinks would work and in practice would
- 11:07
work, and then it's gonna try and compute it and it's gonna get the wrong result or gonna get an error because it's not implemented in the engine. Uh, so the f-- that verification loop is really only as good as the engines that power it.
- 11:21
Uh, so this ends up with, you know, two, two different things. One is the REPL and that's an interface. It's how we present, uh, uh, our, uh, our tools to the agent.
- 11:36
And, uh, it, you know, a REPL is the best interface that we could c- come up with today because coding is what, uh, the current state-of-the-art models are the best at.
- 11:47
But that's not necessarily gonna be true forever, right? Uh, pe-- the, the labs are working on computer use a lot so, you know, eventually maybe the models will be as good as at computer use with a mouse and keyboard as they are at, at coding.
- 12:00
And at that point maybe a REPL is not going to be the best interface. Um,
- 12:06
but it's the best one today. What won't change is the need for, uh, for, you know- For that verification loop. And, uh, what's behind it is actually, uh, I think the, the more durable part because the more capable the models are, like there, there have been, I don't know, four or five, uh, model releases while we've been
- 12:27
doing this work. And every time we've seen the more capable the model is, uh, the more they can get out of that verification loop.
- 12:37
And, uh, another thing we ended up doing is, um, uh, adding domain knowledge, uh, to, to the prompts. Um, and, and that actually ended up, you know, surviving, uh, uh, all, all of the different iterations of the tools.
- 12:52
And, and it always, you know, produced, um, improved results. Um, and this is not so much because the a- the LLMs out of the box don't know what, I don't know, revenue or ARR means.
- 13:03
It's more because they, you know, know many, many things and you kinda need to, uh, pigeonhole them a bit- little bit into what you're, what you want them to focus on, um, for, for the specific task that you have.
- 13:17
And, uh, and it's actually super portable, like this-- uh, almost the exact same, uh, prompt would work for the REPL or the individual tools or, uh, or, or any of the other approaches.
- 13:29
Uh, I also wanna touch a little bit on evaluation. Uh, it ended up being, uh, a lot of work, uh, to evaluate this stuff, and it was actually, you know, a really important part of what actually enabled us to be sure whether, you know, the CSV or the SQL representation were good is if we can actually evaluate
- 13:50
it. And evaluating it, um, uh, correctly, uh, turned out to be a bit of a journey as well. We started with LLM as, LLM as a judge, um, only, and, you know, that works to some extent and sometimes it's the only option you really have.
- 14:08
Uh, but, uh, the annoying part is sometimes you can't really tell if when a score changes, is it because the, you know, the agent changed something or the evaluator, uh, changed, uh, what it outputs?
- 14:23
Uh, so we ended up doing a bunch of work to replace it with deterministic, uh, comparisons wherever that was possible, which is not always possible. Um, but where we could, uh, for instance, you know, take a golden spreadsheet that had a set of inputs and a set of outputs and then use that as kind of a black,
- 14:43
black box to test a spreadsheet that the model produced saying, "Hey, if you put some numbers into these inputs, you get something out of these outputs." And then you put the same numbers into the spreadsheet the model produced, and you see if you get the same, uh, outputs.
- 14:58
Uh, and that, uh, ends up being, uh, you know, sometimes more trustworthy than just using an LLM to grade that work.
- 15:08
And, uh, as with everything, there's, uh, bugs and infrastructure bugs, uh, when you're building agents, uh, many times end up looking like, uh, you know, reasoning failures and it may seem like, oh, the model is doing something wrong.
- 15:23
Um, and, uh, but actually many times it turns out, you know, it's a, it's a bug where, uh, you're, you just have a bug in the code or the, the sc- the skill or the prompt has the wrong example and the model is, uh, following that very faithfully or, uh, uh, there's actually, you know, a bug in
- 15:41
the tools and they fail. Um, and then the model keeps retrying and it seems like the model is being dumb but, you know, it's just trying to work around the issue.
- 15:49
Uh, so there's, you know, a lot of juice to get out of just really looking at those traces and seeing w- what is going wrong and trying to figure out is this, you know, the model not getting it quite right or is this something we can actually fix?
- 16:07
Oh. Sorry. Um, so I wanted to kind of end with, um, uh, a summary of what I think generalizes to other tasks. And, uh, I think the first thing is if your agent is making many sequential tool calls or even parallel tool calls, uh, then you've kind of invented a bad scripting language,
- 16:32
so you might as well just give the agent a real one, and that can be Code Mode or REPL or whatever you want. Uh, the second one is I think feedback loops really matter.
- 16:43
And if you happen to be working in a domain where those, uh, where you can build those feedback loops with ex- existing tools, then great. Uh, less work for you. [chuckles]
- 16:53
Uh, but if you, if you're not, i- if you're in a domain where those, uh, feedback loops don't actually exist, uh, I think it's actually really worth spending the time to build that rendering engine or calculation engine or whatever applies to your particular domain.
- 17:08
Uh, the third is, um, I think interfaces are super important and as, you know, as I explained before, the, the REPL really changed the results we got. Um, so you should really spend the time figuring out what the best interface is.
- 17:24
But you should expect to have to revisit that, um, because the, the models, the capability of the models is gonna keep changing and as they get better at other things, you may find that the best interface, uh, is something else and you need to, uh, find what that next one is.
- 17:41
Uh, next I think, you know, we shouldn't really underestimate the power of planning and think before, uh, think before you act. Uh, and yeah, uh, sometimes the simple things really do make a difference, so don't, you know, actually spend the time on those as well.
- 18:01
Uh, domain knowledge I think is, uh, really important and you, you need, really need to spend a, a bunch of time thinking about what's the, what's the things that you need to re- remind the model about.
- 18:12
It's not so much teaching the model, it's more reminding it to pay more attention to that than other things. And, uh, lastly, yeah, evaluation. I think the more you can do deterministic evaluation, the better, which doesn't mean that you should, you know, avoid LLM as a judge, it just means, uh,
- 18:31
if that's the only option you get, then that's exactly what you should do. But if you can, uh, evaluate in some other way, do that.
- 18:38
And, uh, always check your traces and your plumbing because, uh, sometimes agent confusion is just bugs and you should fix that. Thank you. [audience applauding] [upbeat music]