AI Engineer World's Fair 2025
MCPs are Boring (or: Why we are losing the Sparkle of LLMs)
Read the talk
Beyond fixed tools: let the model write the software
A weather lookup, a CRM query and a JavaScript sandbox show how generated code can reduce model round trips and turn one-off answers into reusable software.
From a talk by Manuel Odendahl
Before you start: Basic familiarity with LLM tool calling, SQL joins and JavaScript functions will help you follow the examples.
What happens when you ask for the weather?
Ask an LLM for the weather in San Francisco. Instead of writing an API integration yourself, you want to describe the request and let the application do the work. That promise is the starting point for Manuel Odendahl’s critique of tool calling. Known as ProgramWithAi on Twitter and wesen on GitHub, Odendahl brings a background spanning Common Lisp, embedded systems, search, databases and backend engineering. He reports using LLMs for coding since 2022 and adopting them for essentially all his coding after ChatGPT appeared.
The basic mechanism is language generation. Models have learned about tools from descriptions, literature and code, with reinforcement further shaping their use. A model emits structured language; the host maps that output to a function, an API request or an MCP server. The returned text becomes material for another inference. It does not need a rigid response schema merely for the model to interpret it, which is part of the appeal: a person can ask for information without writing all the parsing code themselves.
For the San Francisco request, the loop is straightforward:
- The application supplies tool descriptions and argument schemas in the model’s context.
- The model selects the current-weather tool and produces its arguments.
- Deterministic code calls the weather API and returns the conditions, perhaps as JSON.
- A new inference turns that response into an answer for the user.
The software executes the action; the model chooses and describes it. Once several useful tools are connected, that ordinary plumbing produces a remarkably flexible interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model becomes a data courier
MCP makes these connections reusable across applications. A host can connect to tools for GitHub, Blender, local files or even a room-temperature sensor. But a larger inventory also creates more decisions. In Odendahl’s hypothetical collection of 100 tools, both a weather function and an internet-search function might appear to answer the San Francisco question. The model can choose poorly, confuse schemas or supply the wrong parameters.
Arguments have another cost: the model must generate them. Suppose a document fragment is already in the conversation and the next task is to find similar material. Passing that fragment to a search tool can require the model to emit the whole thing again. The user waits while generation performs what is effectively a copy operation. Repeating San Francisco as an argument is the same mechanism on a smaller scale.
Results create the reverse problem. If the user only wants wind speed, a weather response may still include dates and information about the sun and moon. Odendahl illustrates the waste as 2,000 tokens processed to recover a single wind-speed value. That is an illustrative estimate, not a measured token count for a specified API response. The expensive boundary is the model’s context: data crosses it even when ordinary code could select the relevant field first.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One email address, an entire CRM response
Now ask a CRM for OpenAI’s contact information. In the example, the available Get CRM Companies tool returns all 36 companies so the model can extract OpenAI’s email address. A narrower Get CRM Company tool seems like the obvious repair, until a misspelled company name fails to match. Add another option for contact information, such as contact=true, and the interface starts accumulating special cases.
Continue covering every possible request and the tool begins to resemble a query language such as GraphQL—with another syntax the model can get wrong. Yet the broad response has a real advantage: once the company records are in context, the user can ask an unanticipated question such as which companies are in San Francisco.
| Interface | What it buys | What it costs |
|---|---|---|
| All company records | Flexible follow-up questions | Large model inputs |
| One company by name | Smaller responses | Matching sensitivity |
| Specialized flags | Targeted fields | More interface cases |
| General query language | Expressive selection | More query construction |
Odendahl’s illustrative bulk-query scenario uses 20,000 tokens, costs about $0.50 and takes five minutes. Those figures express the proposed tradeoff; the talk does not establish model, pricing or measurement conditions. The design problem is how to retain flexible questions without sending every record through inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let tools use context the application already has
One way to avoid regenerating information is to make existing application context available to the tool. Odendahl proposes several forms:
- Chat history: Include history as tool-call metadata. A tool could ignore it, reuse arguments from previous calls or derive a search query from the conversation.
- Persistent memory: A weather tool could read a stored location of San Francisco instead of requiring the model to repeat it.
- Attachments: Pass files, their paths or their metadata so the tool can work with the original material.
These proposals move information through the application instead of requiring the model to reconstruct it as output tokens.
That access has security implications, and applications do not all represent context alike. A drawing interface may have images that do not fit a conventional attachment model; a graph-RAG application may need to convey graph structure. A universal protocol for all these forms can lead back toward the ontologies, semantic-web triples and XML schemas of earlier interoperability efforts. The easy local extension becomes a difficult shared contract.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Approval should not be the only intervention
A more immediate improvement is to let the user edit a proposed call. Before execution, a small interface could expose attachments and arguments. If the model misspells OpenAI—or decides to search for Oracle—the user could correct the request directly instead of approving it or starting another conversation turn. Approval-only interfaces and permanently static tool inventories are application choices, not MCP requirements: the March 2025 tools specification leaves interaction patterns to implementations and supports notifications when tools change.
The same intervention belongs on the return path. Let the user trim a large response before it enters the model’s context. Odendahl contrasts manually trimming 30 pages with an illustrative ten-minute, $0.50 model pass. A person who can see the relevant records may finish the selection faster than a model can regenerate an explanation of them.
The overdue-invoice example makes the proposed workflow concrete:
- The user asks for all customers with overdue invoices.
- The model proposes a SQL-like CRM query and filter.
- The host offers approve, reject and edit actions before the query executes.
- The user inspects the returned records and edits what will be sent onward.
- If the query was wrong, the user returns to its arguments, changes them and runs it again.
The important distinction is between a proposed call, an executed query and the result admitted into context. Each is a separate opportunity for intervention. A query that returns thousands of unwanted records should not force the user to rewind the entire agent run. This is a local repair loop; Odendahl leaves the broader problem of editing conversation context outside the talk’s scope.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask for the code that performs the action
There is a larger opportunity behind these interface changes. A language model can write a poem about Oracle or imitate a terminal because its output is not confined to function arguments. It can also write programs. Odendahl praises Sonnet 4 for generating whole applications and working with recently introduced APIs. His references to thousands of APIs and knowledge of work only months old express his experience of that capability, rather than results from a named evaluation. The question is why an application should expose that coding ability through only a fixed menu of function schemas.
A SQL query is already code. A file-editing tool can write code that invokes other tools, evaluates more code or runs another agent. Given a database schema, a model can generate a Get Company Contact function at the moment it is needed. Odendahl even speculates that a local three-billion-parameter model could handle that task. The capability need not exist as a separately designed tool before the conversation begins.
The resulting instruction is simple: ask the model to write the code to do the task. Odendahl points to Voyager and recalls a second paper approximately; the likely match is Executable Code Actions Elicit Better LLM Agents. Both are relevant to treating generated programs as agent actions rather than restricting each action to a single predefined call.
His own workflow began with a manual version of that loop. ChatGPT generated shell scripts, small applications or SQL queries; he ran them and pasted the results back. The human was the tool caller. He describes formal tool calling as still feeling closer to GPT-3.5-era intelligence than the coding abilities he now sees. That is a subjective comparison, but it motivates a concrete simplification: replace the copying with an evaluator.
An eval tool lets the model compose actions in code. Bash is one possible execution language. Coding agents already use grep, find and ls, and Odendahl points to their fallback to sed when a dedicated file-editing tool fails. A shell program can combine existing utilities without introducing a new model-facing schema for every combination.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover the schema, then let SQL count
Consider the question: how many orders did John Smith place last month? In this example, the model initially receives only a SQL evaluation tool, not the database structure. It first discovers the tables, finds customers and orders, and inspects their schemas. A customer name, an order’s customer identifier and a date field provide enough information to construct the join and aggregation.
For a SQLite version of that workflow, suppose inspection reveals customers(id, name) and orders(id, customer_id, order_date), with dates stored as ISO-formatted text. The discovery and answer queries can take this shape:
sql
SELECT name
FROM sqlite_master
WHERE type = 'table';
PRAGMA table_info(customers);
PRAGMA table_info(orders);
SELECT COUNT(*) AS order_count
FROM orders AS o
JOIN customers AS c ON c.id = o.customer_id
WHERE c.name = 'John Smith'
AND o.order_date >= date('now', 'start of month', '-1 month')
AND o.order_date < date('now', 'start of month');
Here, last month means the previous calendar month. The database performs the join, filtering and count; only the count needs to return to the model. The schema and query remain inspectable, and the same query can be reused.
A specialized orders-for-customer tool can fail at several different boundaries. It might expect separate first and last names. A retry might then fail because the date argument needs a different format, such as a relative offset. Once the request succeeds, it might return invoice items that the model must aggregate itself. Odendahl estimates 5,000 tokens for that failure-prone path versus roughly 500 for the SQL approach. These are illustrative estimates, but the mechanism is precise: SQL performs deterministic arithmetic on selected rows instead of asking the model to add values from a large response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the useful code
Once the query works, save it. Odendahl proposes storing a customer-order calculation as a SQL view so later requests can select the relevant value without rediscovering the tables and joins. Exploration may require several attempts; the durable result can be one reusable abstraction. He describes consolidating a sequence of queries into a view, although the screenshots shown at this point are not the intended view example.
A collection of reusable functions and views is a library. Instead of exposing every GitHub operation as a separate tool, the evaluator could have access to a preconfigured GitHub library. Authentication belongs in that environment, so the generated program does not need to manage OAuth tokens. The model can compose library operations using the same coding patterns it already knows.
Odendahl claims models handle a library with a very large function surface more effectively than a comparable inventory of tool definitions. He acknowledges improved tool calling in Sonnet 4 and GPT-4.1, while reporting continued parameter mistakes. In contrast, he finds that pointing models at function declarations in his own header files often provides enough information to use those libraries. The practical distinction is between selecting one schema-bound action and writing a program against an API.
For a CRM, import the existing functionality and generate the missing composition. A get-company-info operation may simply call the company-list function, loop over the results and select the requested fields. There is no need to publish each such composition as a new MCP tool before the model can use it. The generated function can become another reusable part of the library.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate an interface for the person, too
Generated code is useful to more than the model. The next step is to build interfaces for the people using the tools. A JavaScript interpreter with library access still leaves the user with a poor experience if the only visible interaction is a function call and a small JSON editor. Odendahl proposes giving the host an easy way to render generated interfaces.
Return to the editable overdue-invoice query. The model could emit a UI description language that the host renders into controls suited to the arguments. Instead of changing raw JSON, the user could adjust sliders, choose dropdown values, attach a file or refuse access to memories. Preferences could be saved for subsequent calls.
The output side could provide a scrollable, filterable view of results. The user could remove information the model should not see, add relevant material or return to the previous inputs and run the tool again. The interface makes context selection an explicit part of the workflow rather than an invisible consequence of whatever the tool happens to return.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A JavaScript evaluator with a database and a server
Odendahl’s prototype makes the execution environment concrete. It is an MCP implementation written in Go, with a JavaScript sandbox he calls Sandbox.js. The environment loads a SQLite library and a web server that exposes a handler-registration function. Evaluated JavaScript can therefore query data and register HTTP handlers. The interface supports executing JavaScript directly or loading an existing JavaScript file, including one written in Cursor.
Using Sonnet 4 with deliberately aggressive prototype prompting, the company lookup begins like the earlier SQL example: inspect the database. But the generated JavaScript can filter the table listing for company-related tables before returning anything. It can also fetch the relevant schemas inside the same evaluation. Odendahl reports saving two tool calls by combining table filtering and schema retrieval in code. The underlying database query operation has not changed; the model can now compose several uses of it before crossing back into inference.
The generated program can continue into querying during that evaluation. The narrated sequence includes an initial result containing ten companies and a subsequent call for all companies. Even the response formatting is rough: ordinary Go-style output appears rather than carefully serialized JSON, yet the model can interpret it.
Odendahl reports a query response in two or three seconds and describes it as one tool call rather than 15. The timing and call-count boundary are unclear because the preceding sequence includes a follow-up query; these numbers should not be read as the cost of the entire demonstration. What the example makes visible is the architectural change: discovery, filtering and querying can happen within generated code instead of requiring a model turn between each operation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a query to a usable CRM
The next request is to save the useful code as a global function. Sonnet goes further than requested. Odendahl reports that the save-function step produced 15 functions in two tool calls, with one call needed to correct a syntax error. The resulting operations include retrieving company information, looking up a company by ID and searching companies. He attributes the error to the prototype’s prompting.
Those functions can now become HTTP endpoints. Because the evaluator has access to handler registration, the model can connect the generated functions to REST routes. It then generates a website, registering separate handlers for JavaScript, CSS and HTML. The existing REST endpoints supply the data. At this point, the user has an interface to work in rather than another answer to read in chat.
This changes what the model is being asked to deliver. A fixed-tool agent performs operations through capabilities supplied beforehand. Here, the model helps construct the capabilities and the interface around them. Odendahl describes the resulting CRM as supporting company edits and new records stored in the database. Once generated, that interface can be used without continued LLM interaction or a separate coding environment such as Cursor. A single kind of capability—evaluation—underpins the workflow, but building the application involves a sequence of requests and calls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Code that creates the next capability
The final step is to think recursively. Ask for code that performs a task, then ask for code that creates or packages that capability. The JavaScript sandbox is not merely an interpreter with some libraries loaded: it can create libraries that later evaluations load and reuse. Generated APIs can also serve other systems, extending the usefulness of the work beyond the original conversation.
Language produces code; that code can produce more code, invoke another model or expose another interface. Eventually the chain makes something happen outside the conversation. Keeping those intermediate artifacts matters because they are where a one-off request becomes durable software.
The LLM writes code that does the work. Calling the surrounding system an agent does not remove the engineering. The database still aggregates records, the server still handles requests and the interface still lets a person complete a task. Odendahl’s closing appeal to recover the sparkle of LLMs is an invitation to build these successive capabilities: let generated software become the material for the next piece of software, instead of ending every interaction at a predefined function call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A Minecraft agent that generates executable skills, improves them through environmental feedback and stores them for reuse.
Introduces CodeAct, which represents agent actions as executable Python so tools can be composed and revised using execution feedback.
Further reading
Historical specification covering tool discovery, invocation, changing tool inventories and application-controlled interaction patterns.
Updates since the talk
- Code execution with MCPArticle
A November 2025 explanation of combining MCP with generated code to load tools selectively, filter intermediate data and save reusable functions.
Read the complete timestamped transcript
- 0:01
I'm, uh, Manuel, and I'm, uh, gonna be talking about MCP is boring, or why we're losing the sparkle of LLMs.
- 0:10
So just a little bit about me. I'm, uh, Manuel. I'm ProgramWithAi on Twitter. My GitHub's, uh, wesen, and I've been a software engineer for 25 years, probably a veteran coder at this point, I guess.
- 0:22
And, uh, I've been wr- writing a lot of Common Lisp, and I'm bringing that up 'cause it's gonna be relevant for, for this talk. But I've been, uh, an embedded engineer.
- 0:29
I love, uh, search engine. I love, uh, databases. I love back-end coding. I've, um, all the, all the coding basically that's like boilerplate and back-end and those kinds of things.
- 0:41
I've been coding with LLM since 2022, since the alpha of Copilot, I think. Uh, and then when ChatGPT came out, I decided to do all my coding with LLM, and I've been pretty much obsessed with it since then.
- 0:53
Thus my ProgramWithAi Twitter account, which, uh where I share all my tips and all the things I've discovered. And we've been using tool calling in LLMs for I, I guess it came out like two years ago.
- 1:05
We've been, uh, using it increasingly. They're great, right? 'Cause, uh, LLMs are language producers. So what that means, we've written a lot about tools, so per se, LLMs know a lot about calling tools.
- 1:18
And I, I mean, we've written a lot about tool using in our literature, uh, in our code, and they've been reinforced, so we have a much, much content to talk about tool calling.
- 1:29
And what that means is that LLMs can produce lang- language that, uh, calls tools. In structured format is what we usually do, and then, uh, we map the structured format that comes out to just basically call a function, call an API, call an MCP server, and it's magical.
- 1:46
The text come back. We don't really need a schema. The LLM can continue working on it, can answer questions about it. And what that means is that now we can, instead of writing code, having to parse schemas, having to validate them, having to be really careful, we can just talk to the machine, say like, "Please," you know,
- 2:02
"Please call the API and then, uh, give me that information," and we're done. Pretty amazing, right? I think, uh, everybody here uses agents and knows pretty much what I, what I mean.
- 2:14
Uh, what does that look like in practice? I think, uh, this is something that is, uh, fairly straightforward. It's the standard example of what's the weather like in San Francisco.
- 2:24
The assistant can do some chain of thought, for example, and, uh, checks what tools it has available, which are provided as a schema in the context that's being sent to the LLM.
- 2:33
It basically says in a special token format, "These are your tools. This is how you call them. This is what it means to call them." And the LLM then decides that, you know, getting the current weather might be a pretty good tool to use, uh, to get the, the weather from in San Francisco, right?
- 2:50
And the tool then, which is a deterministic piece of code, is going to call whatever weather API you have, fetch your current conditions, return that as JSON maybe, and the LLM recei- receiving this JSON does a new inference and outputs an answer to the user's request.
- 3:09
No magic there. Um, well, there's no magic in how it works, but as soon as you've, uh, wired up a few tools, especially your own tools, you realize how magical that is.
- 3:19
Um, but great, we have MCP now, which allows any kind of LLM app to interact with tools that other people have built, so you can download an MCP to interact with GitHub.
- 3:31
You can download one to interact with Blender. You can download one to interact with your own files. You can download one to interact with the temperature in your room if you have, uh, if you have built such an MCP.
- 3:41
It's, uh, absolutely mind-boggling possibilities. The problem is when you give an MCP too many tools, right? Um, so I'm, I'm just going through like we... It's absolutely magic. It's amazing.
- 3:54
Um, but if you give an LLM too many tools, if you give it like 100 tools, it's going to be, "Well, should I call get weather or should I call like, uh, search internet for weather in San Francisco?"
- 4:04
It's kind of hard to decide and, and the LLM like ends up calling the wrong tool or, uh, uses the wrong schema or like doesn't really understand what parameters to give to a certain tool.
- 4:18
So it's not great. And the other o- the other problem is that when you call a tool, that means the arguments that the LLM is going to give that tool and then per induction, those that we are going to provide to that tool have to be inferenced by the LLM.
- 4:34
So if we, if we want to call the tool and say like, you know, "We have this like fragment of document. Please search for something similar somewhere else," you have to emit this part of document, which probably is already in the context higher above anyway, and repeat that to give it to the tool, uh, which is pretty
- 4:54
expensive. It's pretty slow. Uh, I think you all know that, where it's going to be like spinny, spinny, spinny, spinny. And actually it's just copy-pasting something that's literally just above.
- 5:05
If you, if you go back to the San Francisco example, it's like a, a lower,
- 5:10
l- l- less impressive, um, example of that, but it repeats the San Francisco, right?
- 5:17
So the problem is that all the tokens that come back from that tool also have to be put in the LLM. So if you do something like, um,
- 5:27
getting the weather and you only want the wind speed, you'll still have the whole JSON that gives you the date and gives you all kinds of information about the sun and the moon and...
- 5:37
But you actually only care about like a single number that's like a single token. So you've wasted 2,000 tokens on something that could have been answered much more qu- quickly.
- 5:47
Um, what does that look like is, uh, imagine if you have a CRM and you ask for the contact information of OpenAI, uh, what it's going to do is going to call its tool that's called Get CRM Companies.
- 5:57
That's, that's like the tool that exists. And this is a massive response 'cause there's 36 companies in your database, so you end up having this like insane list of companies And, um, then at the end, you basically just look at the email of OpenAI.
- 6:13
And you could say, "Oh, bad tool," right? Like, get a tool that's called Get CRM Company, but then if you misspell OpenAI, maybe that won't work, right? Like, you have all these, like, little...
- 6:24
You're starting to, you're starting to optimize for a certain case where you, like, only want one company, then maybe you only want the contact information for the company, so you add a flag, like contact is equal to true, and you end up with, like, something crazy like GraphQL at the end of the day.
- 6:38
If you want to cover all your bases, and then the model's gonna create wrong GraphQL, and it's gonna go downhill. So
- 6:45
tools have limitations, right? It's great to have all the information of the companies, because suddenly I can ask questions like, "Who's in San Francisco?" And it's going to be able to answer these, like, fuzzy queries that I maybe never asked for.
- 7:01
Um, the problem is you end up having 20,000 tokens. It costs you, like, 50 cents. You're waiting for five minutes before you get the answer to your query, and
- 7:11
that's not, it's not great. So we can engineer around it, but really what we're doing is, uh, why are MCPs basically so restricted? And I think it's because they're boring.
- 7:25
Like, we are not leveraging all the things we could actually do with LLM, with code generated by LLM, with the coding, the tool calling around it. For example, why don't we pass the chat history along with LLM calls?
- 7:39
The LLM call can decide to, uh, the tool can just decide to, to discard it, right? But we already have all this information in the context of an LLM application.
- 7:49
Why not also pass it to the tool call so that we can say, "You know, we've called this thing, like, 15 times. Just reuse the arguments from before," for example, or use this as, you know, a search query.
- 8:03
Um, so that would be one example. It's pretty, pretty simple. It's like, you, you know, please just give me the s- the chat history that is present as metadata to your tool call.
- 8:12
You could also give me your memories if you have an LLM application that has memories, and you have persistent knowledge. Like, why not give that to my tool to say, like, "Well, you have already called me 15 times."
- 8:22
Like, maybe my location is San Francisco, then I can just call Get Weather, and Get Weather will look at my memories and say like, "Oh, there's a location memory.
- 8:31
There you go." Um, pretty easy extension that allows you to do so, so, so much more at a very, very low cost. And another thing is that if you can attach files in your LLM application, like, why not pass these files along, or at least their path, or, like, at least their metadata?
- 8:47
But dangerous, right? Like, the security implications, and the problem is also there's many, many, many different LLM apps that all have different m- modalities, so suddenly you have to design a protocol that covers all these different cases.
- 9:04
Maybe I have an application where people can, like, draw little images, so suddenly how do I pass that as a, as an attached file? Like, maybe I have a graph rag.
- 9:12
How do I pass my graph structure? Lots of questions, lot of boring engineering. We've done that in the '90s and the 2000s with ontologies and semantic web and graphs and triplets and XML schemas.
- 9:28
Could go back to that. Um, but one very easy thing we could also do, um, is, uh, you know, before you call the tool, why not give the, the user a little UI on what they wanna attach and how they wanna attach and maybe edit the arguments?
- 9:44
Because why not? Very, very easy to at least, you know, if, if I have the LLM write a whole tool call, and I, it opens up and it says, like, "Well, I'm searching for the company," like,
- 9:55
OpenAI, and it's misspelled. Then the user can go in and say like, "Nuh-uh. This is OpenAI. It's different." Or maybe it's like suddenly it's searching for Oracle for some reason.
- 10:04
You can say like, "No, no, no, no, this is not the tool call that I want." Currently, the only interaction we have is, like, allow, which is boring, is square.
- 10:12
It's like, why not think wilder? The next thing we can do is, like, well, let the user edit the tool result before [chuckles] we paste all of this stuff back, right?
- 10:19
Like, if I get 30 pages of results, as a user, it's actually faster for me to just edit down these results, and maybe the UI is even nice to do it, instead of waiting 10 minutes and spending 50 cents to have the LLM do it.
- 10:33
Um, so to show you an example of what that can look like is, uh, you can
- 10:40
say, "I wanna find all the customers with overdue invoices," right? We get a tool call. Imagine we have a SQL-like or, like, a SQL tool, so it's able to write SQL queries, and it's going to have this query CRM thing with a, with a filter.
- 10:54
And before we, before it gets call- called, we have this little approve, reject, which we know, and a little edit dialog, which, and here's a very raw way of editing it, but it allows you to, to edit the arguments before they go, before they go further.
- 11:11
Um, so suddenly when the call comes back, you can actually edit the results as well and
- 11:20
tune your, tune your, your query or, like, realize, "Oh, it's the wrong call. Like, let me go back to the edit step. Let me do the call again." You, I don't need the LLM to do that kind of stuff, right?
- 11:29
It's, uh... And that to me, especially when I'm doing, uh, a lot of, you know, database queries, those kinds of things, if I do the wrong query and I have the, only the option allow, and I get back 10,000 things, I'm like, "Damn, I'm, like, messed up."
- 11:47
And if it's an agent thing, I have to rewind a whole agent run instead of being able to edit this one tool. I've got a whole set of thoughts around context editing, but, uh, this goes beyond the, the, the scope of this talk, and you're welcome to come, uh, talk to me to, to get more info.
- 12:05
Um, so- LLMs are so much more, right? They're language producers. They can create every word under the sun. They've been trained under every word under the sun, and you can do the weirdest crap with them.
- 12:18
You can do poems about Oracle. You can do... Right? Like, those are even boring things you can do. If you look at everything like AI re- red teams are doing, and if you're going to starting to talk to it like a terminal, it's a terminal.
- 12:30
Like, basically everything you tell the LLM is going to be what the LLM's gonna pretend to be. So this gives us, like, a lot of leverage because they've been trained so hard and really reinforced to learn about code, right?
- 12:43
They're great at coding these days. You can, like, one-shot entire applications. You can, uh, call, like, 15,000 APIs that have been, like, recently built. Like, Sonnet 4 is amazing at knowing stuff that was just, like, built two, two months ago.
- 12:57
It one-shots things that I never thought would be possible before. But why are we stuck with, like, tools that don't e- that don't even work that well, right? Like, I know a tool call is a function call.
- 13:10
Why are they so bad at function calling when at the same time they can generate code that is so much better? Um,
- 13:18
so code is a tool, right? There's many things that are called, like even if I have just a SQL tool, I'm basically giving it code, 'cause a SQL query is code.
- 13:28
If I have an edit file tool, I can give it code, and you can write code that calls tools, that calls code, that runs an agent. You can do all this, like, infinite recursion stuff at the inference time.
- 13:42
And, um, basically, you can tell an LLM, "Please create the tool that I want," right? If I, if I give it a database schema, I can say like, "Well, please create the tool Get Company Contact."
- 13:55
No problem. Every LLM by now can do that. It can probably do it with, like, a three billion model, three billion parameter model locally. Um, and so they're these kind of like magical genies that can just, like, create whatever you want at the moment you need it, in the way you want it, and modify it, right?
- 14:12
So why don't we leverage that instead of being stuck with this, like, you can only call functions, you can only call functions with this schema that we've given you, and it's, like, static, and you can't even modify it.
- 14:23
So the only prompt engineering you actually kind of need to do agents is, like, write the code to do X, right? And there, there's a couple of papers around it.
- 14:32
Um, I haven't linked them, but there's the Voyager paper, uh, from two years ago already, and there's the, I think, Code Elicits Better Tool Actions, something like that, which is a very short paper that basically says, like, you know, just, like, write code to do tools.
- 14:47
Um, and I've been on the LLM stuff pretty early to do code, like, once the instruct versions came out. And, and writing my little tools was the first thing I did.
- 14:59
I was like, "I want to write a shell script to do this XYZ, and I want to do the shell script to do XYZ." I would run it, I would paste the result back into ChatGPT, which is basically like a,
- 15:09
like, um, I'm the tool caller at that point. And that was really, really useful. So the whole time before I read these papers, before MCP came out, before tool calling came out, I was, like, generating these shell scripts or generating little applications to do these kinds of queries, or generating SQL queries that would push back.
- 15:27
And so it was already crappy back then, but it hasn't really gotten significantly better, right? Like, it feels that tool calling is still stuck at this, like, GPT-3.5 kind of intelligence.
- 15:38
I, I don't know why, but they're not that great. [chuckles]
- 15:42
However, they, they're so good at writing code that the only MCP I think I need is, like, eval, right? Instead of copy-pasting things, putting it in a shell, copy-pasting it back, all I need is, like, eval around it, and it can be Bash, and you can look at it.
- 15:57
Every coding agent does most of its work with actually Bash calls. They call grep, they call find, they call ls, they call sed when they're, like, struggling with their edit file tool, right?
- 16:06
It's like just editing a file, they actually don't know how to do it. And when they really fail at calling their tool, they're just like, "Ah, fuck it, I'm gonna write said code," which is an insane tool anyway as well.
- 16:18
Um, so you can realize, like, why do we even need MCPs when, uh, we just have eval? And so here's an example where eval is actually SQL, right? And I just ask it, like, "How many orders did customer John Smith place the last month?"
- 16:33
And I don't even tell it what the database is. I just say like, "You have a SQL evaluation tool." So the first thing it's gonna do is gonna be like, "Well, what tables do I have?"
- 16:42
Right? And it says like, "Oh, well, I found customers and orders, so, you know, let me look at their schema." And it says like, "Oh, well, customers, you know, they have, like, a field called name, and then there's, like, a field customer ID in the table orders with a date.
- 16:55
Like, yeah, I know how to do a SQL query to do that," right? And so it's, it's doing the structure loading. It looks at the schema of customers and orders and says like, "Oh, look, there's customer ID."
- 17:07
And now it can just write a SQL query that does the result of the orders, the join that it needs with the aggregation that it needs, and just returns the result.
- 17:16
And the crazy thing here is, like, if you had a thing that's called, like, orders for customer placed last month, maybe the LLM won't realize that actually it has to pass, you know, uh, first name, last name, for example.
- 17:30
So the first tool call fails, and then it, like, repeats its tool call and says like, "Well, oh, the date format is wrong. You have to do, like, minus 30 days."
- 17:37
So it tries with minus 30 days, and then suddenly it gets, like, a huge table with all the invoice items, and it's like, "Okay, cool, now I have the information.
- 17:44
I'm gonna aggregate as an LLM," get the wrong number 'cause they can't do addition, and you wasted 5,000 tokens, and you get, like, a wrong response, while this actually probably takes like, you know, 500 tokens and you get a deterministic, repeatable kind of query that can reuse, right?
- 18:04
So this is why eval is such a, such a nice tool, is that... Oops, why did this go so fast? But if we take it to the next step, it's like, once it works, you know, why not store this query?
- 18:15
Uh, why not say like, "Oh, now we have like a get customer order amount query." Um, and so the way you can do that, for example, with SQL, is that you can just create a view to do it, and then suddenly you don't even need to look at the tables, you don't need to do complicated SQL.
- 18:30
You just do like, "Oh, select amount from view." And so this is what it looks like, is, um, this thing is going to run. It's like couple of queries, like maybe it even needs to do things in sequence where it's like gonna be like, "Oh, I'm gonna select the orders."
- 18:43
And then I see like, "Oh, okay, I have to join this table. Okay, let's, let me do it again, and then I have to do this," has this complicated code.
- 18:51
But then being good coders, it knows how to turn like 15 queries into a single view. And, um, at that point, this looks like not the example of the view that [chuckles] I was trying to show.
- 19:03
So, um, I, I can show you that live. I, I used the wrong, uh, the wrong screenshots. Um,
- 19:11
but that shows you that you can easily create tools and functions and views and whatever. And so when you create a lot of functions and views and make them nice to reuse, that's called a library, right?
- 19:24
So instead of like exposing a GitHub tool, you just say like, "Well, your eval now has access to the GitHub library." And it's pre-configured. You don't need to deal with like OAuth tokens and whatever.
- 19:34
You just have like this whole API to do interactions with GitHub. And funnily is like if you put 10,000 functions into an API, the LLM actually knows how to use them.
- 19:46
If you put them as tool calls, it doesn't. Which
- 19:50
I don't know why they get so bad once you add tool calls. Um, I mean, they've gotten better, right? Like Sonnet 4, uh, GPT 4.1, like all these newer models have been trained to be a little bit better at tool calling, but you still like very, very quickly run into like weird things where it doesn't understand the
- 20:06
parameters. And then, however, they're so good at writing code these days that I rarely have to fix anything in code, even for my own libraries, right? Like, I just point them at my set of functions in a header file, and then it works.
- 20:23
So why instead of doing the CRM MCP, you know, just do like import star from CRM and then you're done. You can, not only do you have all the tools that you used to have as a, as a MCP, but now you can create your own tools that are really rich.
- 20:37
So, you know, uh, just build the get company info tool. Um,
- 20:45
um, because you, you... Well, you don't need to build an MCP get company info tool because you can actually just generate the code to call CRM list info, and then you have like, uh, you know, it's able to put a for loop on around it.
- 20:57
It's able to put any kind of code around it. And so if you think a step further is that these tools and the code that's generated is like not just for the LLM, but like a lot of it is for us as well.
- 21:12
So why don't we use the fact that we can use code now to build tools that are much richer than just a function call with like a little JSON window that you have to click in to edit it, um, if you can even edit it, but instead have tools that build UIs for us, right?
- 21:27
So instead of just incorporating a JavaScript interpreter that has like under-the-hood access to libraries, but then still on the surface just calls like functions, um, why not have something in the LLM host application that allows us to do UIs very easily?
- 21:44
And so I've, I've built a couple of prototypes around it, but just to show you what this would look like, if we go back to editing, you know, the tool input and the tool output.
- 21:53
If you give the LLM the opportunity to say like, "Well, if the user wants to edit my input," you know, give it like a good UI instead of a little text window where you can edit JSON.
- 22:03
And so this could, if we go back to the previous example, it could like output some kind of UI DSL that's rendered by the LLM host. And suddenly, instead of having to edit like JSON fields, you get a slider, you get like drop-downs, you get all kinds of things which the user can validate or tweak or, you
- 22:20
know, attach a file, say like, "No, I don't want it to know about my memories," and then call the tool. Um, and maybe there's like preferences you can save.
- 22:28
Like there's all kinds of things you can do around it, right? And then similarly, you can have a UI that allows you to edit the output, um, with like maybe a scroll view and a filter, and you can say like, "Well, remove this, add this.
- 22:38
The LLM shouldn't know about this. Recall the tool by modifying my previous inputs," and you get this like rich UI to do your work. Um,
- 22:47
and so what I built, and I, I can show you that live, is like a very simple MCP, which has a Sandbox.js. It's written in Go, and it has two libraries.
- 22:56
It has SQLite library, which is loaded, and it has a web server that basically has a single function that's called register handler. And then you just write JavaScript for the handler, right?
- 23:07
Um, so there's a, a single call, it's called eval, and then the LLM, when it calls it, can also register rest handlers. So what this looks like, right, like there's execute JS, and if you want to load a file that you've already written, if you're like saying cursor, you can use execute JS file.
- 23:24
Um, what this means, if I use the same query and y- write as a prototype, so I'm a little bit aggressive on the prompting, it will suddenly write JavaScript to what it's gonna be like, you know, let, let me look at the tables, so exactly the stuff from before.
- 23:36
But it's already clever 'cause it's, [chuckles] 'cause it's Sonnet 4, so it will actually look for the company table in code already. It won't even wait for the result, right?
- 23:47
Like it won't stream back all the tables in my database. It will actually already filter them. Uh, so it saves on tokens and whatever. Once they return these tables in their schema, so you save two tool calls, right?
- 23:59
And you save a lot of tokens just by the virtue of having eval.
- 24:03
Just by the virtue of having it. It's the same tool. It has like db SQLite query, but suddenly we're already like saving money. Um,
- 24:14
and then in the same call, actually, I forgot about it. In the same call, it actually already does the querying of it. And it's well possible that, you know, you already get like some, some stuff at the beginning.
- 24:27
Suddenly you get like the first 10 companies, and then it calls it again with all the companies, and then the LLM is able to, to show the result, right?
- 24:35
And you can see the schema is super ugly. It like actually doesn't even take care of printing it as JSON apparently. Just like literally logs it out with the standard like Go syntax.
- 24:46
And then you get the query response, and this took two seconds, right? Or like three seconds, not 15 tool calls. It took one tool call. Um,
- 24:59
and so what I can do now is like just save it as a global function, it being sonnet and being kind of on crack cocaine and just like deciding to do 15 things, it like generated 15 functions, sure.
- 25:10
So now I have all my tools, right? I've get companies info. I can get a company by its ID. I can search companies. I can do all kinds of things by the virtue of just like two tool calls.
- 25:20
Uh, and one of them was just because of a syntax error because my prompting's bad, because it's a prototype. So why not create a REST API endpoint, right? Because it has register handler, so why not hook up all of these tools to all of these functions to a REST API, which like, all right, here you go, right?
- 25:39
Like, just like not really hard. It just like calls the thing. Um, because it's sonnet, again, it like generated like even more. [laughs]
- 25:48
So, and then I asked it to generate like a website, right? So it's registering a handler for the JavaScript, it's registering a handler for the CSS, and it's registering a handler for the HTML.
- 25:57
It already has the REST endpoints, and boom, now I have a whole CRM. I don't even need the LLM anymore. I can just start working with it, and that was a single tool, eval, right?
- 26:10
So I think we're leaving so much on the table by focusing on tool calling and saying like, "Oh, there's like agents, there are these little widgets with these little creatures with tools," and that's our mental model.
- 26:20
Instead of being like, "No, this is like a magical genie that can create anything I want when I want it without even needing any big information because it's all in training corpus."
- 26:31
You can edit the companies, you can add new ones, which will be stored in the database. Like this is a real, real CRM. This, this, this is just, I don't even...
- 26:41
But I don't need Cursor. I don't need anything. I can just say like, "Mm, do it."
- 26:48
So to close this off, LLMs are absolute magic, and I think you should think, get used to thinking recursively, right? It's like if you ask the LLM to do something, ask it to do the code to do something, and then once you have something that writes the code to do a certain task, ask it to write the
- 27:03
code to write the code, right? Which is kind of what I did with the JS sandbox. I didn't just give it like JavaScript with loaded libraries, is that suddenly I have a JavaScript sandbox that you can use to create libraries that can then be loaded later on.
- 27:17
And those are APIs that I can then reuse in like different systems, and it's like all very circular. I can create everything I want all the time.
- 27:28
And right, they create words, and those words create more words, and then I can create words that create an LLM that create words that create words. Is, it's, it's infinite.
- 27:38
Um, and all of these word ultimately are going to make things happen in the real world. But if you focus on just the thing that you need to make happen in the real world, you tend to forget that it's not just tool calling, it's actually we are engineers, the LLMs are engineers.
- 27:56
Everything's just engineering in this case, so just write the code to solve the problem as we have been used to instead of saying like, "We have an agent," and suddenly it does everything.
- 28:05
That's not true. The LLM writes code that does everything. Um,
- 28:11
so yeah, uh, infinite loops of creation. I hope, uh, you enjoyed this talk, and I hope that, uh, you are able to bring back the magic into LLMs, right?
- 28:20
Like the sparkle, sparkle, sparkle because, um, they're so much more than what we're trying, than what we're thinking of them these days.