AI Engineer Code 2025
How Claude Code Works
Read the talk
How Claude Code Works
A coding agent can have a simple control loop while still needing careful tools, context management, permissions, and tests. The engineering lies in deciding where flexibility should end.
From a talk by Jared Zoneraich
Before you start: Familiarity with command-line tools, basic Python, and language-model tool calling will help you follow the architecture and examples.
What made coding agents useful?
What changed between coding agents that struggled to finish a task and agents useful enough to reorganize an engineering team around? That is the practical question behind Jared Zoneraich’s workshop. His account is an independent investigation, not an Anthropic-endorsed description of Claude Code’s internals.
At the time of the talk, Zoneraich was building PromptLayer with a small team in New York. He reports that the product had launched three years earlier and was processing millions of LLM requests daily. Its organizing principle was collaboration: engineers and product managers should build with domain experts, just as an AI lawyer should involve lawyers. His own work alternated between using agents and using his company’s tools to build them.
That experience led to a concrete engineering rule: if a task could be completed with Claude Code in less than an hour, do it without putting it through prioritization. Small platform problems—an upload that fails, an awkward edge case—otherwise accumulate faster than a small team can schedule them. Understanding why agents became useful therefore matters beyond choosing a coding assistant: it suggests how to build agents for other work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From copying code to delegating work
The workflow changed in stages. First, developers copied code into ChatGPT and pasted answers back into their editor. Cursor brought that interaction inside a VS Code fork with Command+K. Its assistant then handled a longer back-and-forth, followed by Claude Code’s terminal-oriented workflow, where a developer could delegate work without touching each edit. Each step transferred more of the execution process to the agent.
Zoneraich attributes the improvement to two connected changes: simpler architecture and better models. Tool calling gives the model a structured way to request actions, extending earlier efforts to obtain reliable JSON such as Jsonformer. As models improve at selecting tools and recovering from mistakes, the application needs fewer branching prompts designed to compensate for particular model weaknesses. Give the model useful tools, then let it choose the next step.
This changes the expected lifetime of scaffolding. A classifier, special-case prompt, or routing branch may solve a real problem today but become unnecessary as the model improves. In Zoneraich’s interpretation, Claude Code exemplifies this shift by relying on runtime searches such as grep rather than making an embedding index and a complex retrieval pipeline the center of its architecture. He contrasts that with Cursor’s mixture of retrieval approaches; this is a preference for simpler exploration, not a claim that retrieval has no place.
The engineering philosophy is familiar from Python’s import this: prefer simple designs, avoid unnecessary nesting, and distinguish complexity inherent in the problem from complications introduced by the implementation. Agent architecture does not escape those principles just because the component making decisions is a language model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repository instructions and one master loop
CLAUDE.md, and the analogous AGENTS.md used by other agents, provide a simple place for repository instructions. Instead of requiring a separate system to infer every convention, a Markdown file can state how the project works. Users can edit it; agents can update it when appropriate. Zoneraich contrasts this with his description of Cursor 1.0 building a local vector database to understand a repository. Both approaches supply context, but the file makes an important part of that context explicit and editable.
The control flow is equally direct: ask the model what to do, execute its tool calls, append the results, and ask again. When the model returns without tool calls, return control to the user. Zoneraich tentatively calls the internal loop N0; the related PromptLayer architecture analysis spells it nO. These names come from an analysis of the implementation, rather than a stable public interface.
A Python harness can express that control flow without hard-coding a route through the tools. Here model_turn returns a dictionary containing text and tool_calls; each call carries an id, name, and arguments. The host supplies tool handlers, including their permission checks.
python
def run_agent(model_turn, tools, messages):
history = list(messages)
while True:
reply = model_turn(history)
history.append({"role": "assistant", **reply})
calls = reply.get("tool_calls", [])
if not calls:
return reply.get("text", ""), history
for call in calls:
try:
handler = tools[call["name"]]
result = handler(**call["arguments"])
except Exception as error:
result = {"error": str(error)}
history.append({
"role": "tool",
"tool_call_id": call["id"],
"content": result,
})
Returning an error as a tool result gives the model information it can use to correct its next action. The loop does not need a separate branch for every possible failed search, malformed command, or mistaken assumption. That flexibility is the central mechanism, although the host still controls what actions its tools permit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why keep dedicated tools when Bash can do so much?
The tool inventory changes frequently, but each tool in the workshop’s snapshot has a reason to exist beyond merely exposing a command.
| Tool | Purpose of the boundary |
|---|---|
| Read | Limit how much file content enters context |
| Grep / glob | Search contents and paths with familiar operations |
| Edit | Express targeted changes instead of replacing files |
| Bash | Compose commands and run temporary programs |
| Web search / fetch | Delegate web work to a cheaper, faster model |
| To-dos | Maintain a visible plan and support steering |
| Tasks | Isolate work in a separate context |
A dedicated read tool can reject an oversized file rather than dump it into the conversation. Search tools reproduce what a developer would do at a terminal. An edit tool can describe the few lines that change, avoiding the burden of generating every unchanged line again. Zoneraich compares that to crossing out parts of a slide deck instead of recopying the entire deck to revise it.
Bash is a universal adapter to existing software. An agent can create a Python file, run it, inspect its output, and delete it afterward. It can compose shell utilities, generate tests, or work out how to start a local development environment. That replaces the fragile practice of keeping a small list of startup commands in a note that gradually becomes stale. Zoneraich also attributes Bash’s usefulness to the abundance of familiar command-line patterns in training data; he uses less common languages such as Rust as a contrasting example.
Versatility does not eliminate the need for operational guidance. The system prompt can tell the agent to read before editing, prefer the dedicated Grep tool for searches, run independent operations in parallel, and quote paths containing spaces. Zoneraich suggests that security, sandboxing, and output limits help explain the dedicated-tool preference. These are small corrections informed by repeated use, rather than an attempt to precompute the agent’s entire workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Structured plans without a hard-coded workflow
To-do lists occupy an interesting middle ground: the data is structured, but the workflow rules need not be enforced in code. Zoneraich describes instructions to keep one task in progress, mark completed tasks, and leave a task in progress when it encounters a blocker or error. In his account, the model follows those rules because they appear in the prompt and tool descriptions, not because the client rejects every invalid transition. A schema can organize state without enforcing the policy for changing it.
The agent emits a function call carrying task data. The example structure includes a version, an identifier, a human-readable title, and potentially evidence blobs. Zoneraich describes the identifiers as hashes that can be referenced later. Those fields make a plan addressable and legible; they do not by themselves prove that the underlying work has happened. His suggestion that this might have been a quick implementation is speculation, not an account of Anthropic’s development process.
The benefits extend beyond planning. Recorded task state helps resume interrupted work, gives the user a way to see progress, and provides something concrete to redirect. A long agent run with no visible signal may be capable but difficult to supervise. The to-do list improves the working relationship between user and agent even when it does not change the model’s coding ability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep accumulated output out of the reasoning path
A simple loop can still drown in its own history. Every terminal log, file read, and intermediate investigation competes for the model’s attention. Zoneraich describes an AsyncBuffer, tentatively called H2A in the talk and h2A in the related analysis, that decouples I/O from reasoning. He also describes a compressor that drops middle material and summarizes the head and tail of accumulated context. These are his account of the internals, rather than a public contract for how every release compacts a conversation.
Zoneraich estimates that compression begins at roughly 92% context capacity; he does not identify a tested release or measurement procedure. The broader design problem survives any particular threshold: the agent needs somewhere to keep useful material without carrying all of it in every model request. A sandbox filesystem supplies that storage. For deep research, he explicitly asks the agent to save Markdown files that it can consult later, and predicts that ordinary chat interfaces will increasingly include sandboxes for the same reason.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move rigor into tools, not every decision
The alternative to an exploratory loop is often a directed acyclic graph, or DAG, of prompts and routing rules. Zoneraich describes customer-support systems built over the preceding two to two-and-a-half years with hundreds of nodes: classify the request, route refund questions to one prompt, route another intent somewhere else, and keep subdividing the workflow. Narrow classifiers and discarded context can restrict what each step is able to do.
He argues that an X/Y classification step with discarded context can prevent certain unwanted refunds and injection paths. That should be read as a claim about restricting a particular workflow, not as a general guarantee that DAGs solve prompt injection. Replacing those restrictions with a flexible agent restores an attack surface along with the ability to explore. Zoneraich characterizes loops as 10× easier to develop and 10× more maintainable, without supplying measurements.
Even apparently helpful guidance can interfere with exploration. In a PromptLayer browser-agent experiment, he added button titles and prescriptive navigation instructions to help the agent operate the dashboard. It navigated worse: the instructions seemed to distract it when the prescribed sequence did not fit. He explicitly allows that the experiment could have been flawed and worth rerunning. The useful lesson is to test whether additional guidance helps the actual task, rather than assuming more instructions must improve it.
An audience member raises the obvious objection: even if scaffolding becomes obsolete in a few months, what if it solves a problem now? Zoneraich’s answer is use-case dependent. A bank chatbot warrants more care than an open-ended development assistant. His preferred compromise is a flexible master loop with rigorous tools. Put the edge-case workflow inside a tool that can be evaluated and versioned, while leaving discovery and navigation to the model. A tool may contain substantial deterministic logic without forcing the entire agent into a predetermined graph.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adjust reasoning instead of adding another router
This preference also explains Zoneraich’s skepticism about embedding intent classifiers and regex rules throughout an agent. PromptLayer prototyped a non-LLM classifier for prompt pipelines but never released it. He recognizes that classifiers can work and may still make sense when cost dominates; his concern is their diminishing value as smaller models become cheaper and more capable. The point is not to ban regex from software, but to avoid making brittle intent routing the agent’s organizing principle.
Reasoning effort supplies another control. The workshop shows the historical trigger phrases think, think hard, think harder, and ultrathink as ways to request more reasoning. Its slide lists 4,000 tokens for think, 10,000 for think hard, and 31,999 for both think harder and ultrathink. Treat this as the keyword ladder shown in the recording, not current configuration guidance. Architecturally, the distinction is between adjusting the reasoning budget of the ongoing agent and dispatching a separate hard-planning tool.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A simple loop still needs execution boundaries
Shell access makes mistakes consequential. Zoneraich jokes about using YOLO mode, then gives the counterexample: teammates had dropped their local databases. He says enterprise customer workflows do not use that approach. The more dangerous combination is an agent that can both fetch untrusted internet content and execute commands; instructions encountered on the web may influence actions on the machine.
He describes URL approval, isolated web work, containerization, and a command-gating pipeline that considers Bash prefixes. These belong to related but distinct layers: permissions decide whether an action is allowed; sandboxing limits what execution can affect. His later contrast with Codex should not imply that OS-enforced isolation was exclusive to Codex: Anthropic had already documented Claude Code filesystem and network sandboxing using macOS Seatbelt and Linux bubblewrap. Much of a production agent’s complexity can live in these execution boundaries while its reasoning loop remains small.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Delegate the investigation, return the result
Sub-agents address the context problem by giving a bounded task its own conversation. A researcher, documentation reader, test runner, or reviewer can consume detailed material without adding every intermediate step to the parent’s history. Only the result needs to return. In the dashboard experiment, for example, Zoneraich asked the coding agent to read the documentation before changing the site; a documentation sub-agent could perform that preparation separately.
The Task interface shown in the talk has two important arguments: a short description that the user sees and a longer prompt that tells the sub-agent what to do. An example description is finding the default chat-context instantiation. The parent is therefore not merely selecting a predefined specialist: it is writing instructions for another agent. If a task fails, it can try again with more information. Zoneraich suggests that some applications may benefit from a structured object instead of one long prompt string.
The audience asks where those instructions come from. The clarification is that the main agent has the tool schema in its context and generates the description and prompt dynamically when it calls Task. The task call itself is part of the main interaction; the delegated investigation has its own context. Independent task calls can run in parallel and return their results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tune behavior and load specialized instructions
Published system-prompt leaks form part of Zoneraich’s research. The instructions he highlights are pragmatic: keep output concise, avoid announcing the work instead of doing it, use tools rather than merely propose commands, match existing code, avoid unnecessary comments, run independent commands in parallel, and maintain to-dos. He notes that the comment instruction does not reliably work for him. Prompting is useful here because these are often preferences discovered through repeated use, not invariants that must reject an operation whenever they are violated.
Skills extend the available instructions without loading everything at once. A documentation-update skill can contain product knowledge and writing style; a design skill can carry a style guide. Zoneraich also describes seeing skills used to edit Word and Excel documents, though that is not his own workflow. For deep research, he supplied an article or repository explaining the process and asked Claude Code to turn it into a skill. The common mechanism is selective context: load the detailed instructions when the task needs them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Small edits and the limits of skill discovery
Unified diffing applies the same economy to output. Revising an essay with red lines requires less reproduction than rewriting the entire essay; editing code through a patch similarly concentrates generation on the intended change. Unified diff is an established format, although some agents use variations that omit details such as line numbers when their editing mechanism does not need them.
For example, a proposed correction to a Python predicate can preserve the surrounding function while identifying the exact replacement:
diff
--- a/eligibility.py
+++ b/eligibility.py
@@ -1,2 +1,2 @@
def is_eligible(age):
- return age > 18
+ return age >= 18
The patch says what should change; applying it and checking the resulting behavior remain separate operations. The unchanged function signature provides context while the deletion and addition identify the edit.
On-demand instructions introduce a different failure mode: a skill that never loads cannot help. An audience member reports seeing a warning when CLAUDE.md exceeded roughly 40K characters, splitting it into skills, and then finding those skills ignored. Zoneraich explains that short skill descriptions should help the model select the relevant instructions, but says he generally invokes skills manually. Better prompting, explicit workflow rules, and model post-training are possible remedies; the exchange does not demonstrate a fix.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fewer tools, adjustable models, new primitives
Two possible directions follow from this architecture. One is a master loop with hundreds of increasingly well-selected tools. Zoneraich favors the other: a small tool set centered on Bash, with scripts in the local directory extending what the agent can do. His preference for one broad tool is not absolute; he considers the earlier core inventory a reasonable boundary.
Model choice can also become part of the agent’s workflow. Zoneraich proposes a hypothetical default model that is 20× faster but somewhat weaker, with a stronger reasoning model available as a tool. Planning may deserve the stronger model first—he names GPT-5.1-Codex and Opus as possibilities—while cheaper or faster models execute more routine work. To-dos and skills suggest another direction: new first-class structures that help an agent organize work without prescribing every action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Different agents optimize different working relationships
Zoneraich calls this the AI therapist problem: there may be several useful approaches without one global best answer. His examples—meditation, CBT, and ayahuasca—illustrate different approaches rather than recommendations. The analogy carries into product design: taste, domain knowledge, and the desired interaction matter alongside capability. He personally uses Claude Code for local environments and Git, Codex for harder problems, and Cursor’s Composer for speed. He explicitly lacks evidence that his impression of Codex’s greater power establishes comparative superiority.
His comparison is a map of product emphases, not a leaderboard:
| Agent | Emphasis in the workshop |
|---|---|
| Claude Code | Simplicity, Git, interactive terminal work |
| Codex | Context management and difficult problems |
| Cursor | IDE interaction, speed, model choice |
| Factory Droid | Specialized sub-agents |
| Cognition Devin | End-to-end autonomy and self-reflection |
| Amp | Model flexibility and distinctive user experience |
These differences leave room to combine ideas across products rather than copy one architecture wholesale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Codex: similar loop, different execution machinery
Codex’s public repository makes the agent itself available for investigation. Zoneraich describes a Rust core with the same broad master-loop pattern, but more explicitly event-driven concurrency: submission queues carry work in, and event outputs carry progress and results out. In his comparison, threading, permissions, and the underlying model distinguish it more than the high-level loop does.
The workshop names macOS Seatbelt and Linux Landlock as Codex sandbox mechanisms. Its architecture slide also includes persistent state and resume/fork workflows. These details explain how an agent can manage ongoing sessions and concurrent activity around a small decision loop; they do not make OS isolation unique to Codex. Zoneraich then shows Claude Code researching Codex through Explore sub-agents—using one coding agent to inspect another, with delegated investigations keeping the parent’s context focused.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Amp: improve the environment and start fresh threads
In the workshop’s snapshot, Sourcegraph’s Amp offers a free tier supported by advertising and what Zoneraich describes as excess provider tokens; PromptLayer is an advertiser. Amp also avoids making a model selector the center of the experience. That lets its developers change the underlying models without promising users the behavior of one fixed model.
The more consequential design question is whether the repository is a good environment for an agent. A self-contained project with runnable tests gives the agent a feedback loop: change something, run the checks, inspect the failure, and improve the result. Zoneraich connects this to Factory’s thinking and wants an equivalent loop for frontend design, where an agent can inspect its own output and revise it. Better autonomy depends partly on making success observable.
Handoff addresses accumulated context by starting a fresh thread with information selected for the next task. Compaction, by comparison, summarizes an existing conversation so work can continue within it. Zoneraich likens Handoff to switching weapons instead of reloading in Call of Duty: change to a fresh working context rather than keep repairing the old one. He allows that both approaches may be useful. Amp’s fast, smart, and oracle choices extend the same product philosophy to model routing, with oracle reserved for difficult problems and its underlying model replaceable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cursor: fast interaction and specialized training
Cursor’s distinction begins with the IDE experience. Zoneraich finds Composer fast enough to change his daily preferences and sees its specialization as evidence that product data can support defensibility through model training. He calls Composer distilled, although Cursor’s original report describes a mixture-of-experts model trained with reinforcement learning and does not establish that characterization. The practical appeal is shorter interaction cycles; the corresponding hazard appears in his anecdote about accidentally pushing to master on a personal project.
Cursor’s gradual improvement from a rough VS Code fork matters to this account: it accumulated a useful product while improving its models and interaction design. Zoneraich also describes Codex models as distilled and optimized for coding agents, and predicts faster OpenAI models; those are his characterization and forecast. Cursor meanwhile continues to offer alternative models alongside Composer. The blog screenshot discussed in the talk includes GPT-5, while he names GPT-5.1 as a newer possible planning choice. Specialization and access to other frontier models can coexist in one product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate outcomes, decisions, and execution traces
How should a team choose an architecture or know that a change helped? Zoneraich distrusts public benchmark marketing, but the more useful problem is local: a flexible loop may solve the same task through several valid paths. Testing one exact sequence risks rejecting good behavior, while testing only the final answer can conceal expensive or fragile execution.
He separates several evaluation approaches:
| Evaluation | Question |
|---|---|
| End-to-end | Did the agent solve the problem? |
| Point-in-time | Given this context, does it choose the needed tool? |
| Historical backtest | How does a change affect previously observed work? |
| Trace sanity checks | Did calls, retries, or elapsed time become suspicious? |
A point-in-time test can begin halfway through a conversation where the next action is known. Backtesting replays historical inputs and can reveal changes in tool selection. Tool-call counts, retries, and elapsed time are what Zoneraich calls agent smell: useful indicators that warrant investigation, not substitutes for correctness. His recommended starting point is to capture historical data and rerun it.
The concrete example uses PromptLayer’s evaluation product as a batch runner. Each row runs headless Claude Code, with a provider supplied through file variables. The instruction is to search the web, find that provider’s most recent and largest released model, and return its name. The internal search path is left open. This makes it an end-to-end task: assess the returned result, then inspect row-level statistics such as tool calls to understand how the agent got there.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make exact output requirements testable
The flexible loop does not remove the value of function-level tests. A tool with a clear input and output can be tested rigorously. If the tool is itself a sub-agent, evaluation becomes recursive: test that sub-agent’s end-to-end behavior as a component of the larger agent. This is where Zoneraich puts work that must conform to a specific voice, email format, or blog structure.
His email workflow first checks whether the message meets the required standards. A satisfactory email proceeds to revision; an unsatisfactory one first gets missing parts, such as a header, added and then undergoes revision. The displayed workflow makes those branches visible through nodes labeled check-email-parts, add-email-parts, revise new email, and revise good email. Exploration is useful for deciding what to write, but this tool constrains how the deliverable is prepared.
A related SEO workflow has roughly twenty nodes, including deep research, outline generation, conclusion repair, and adding links. That degree of structure makes sense when the desired output is specific. To evaluate the email version, Zoneraich runs sample emails through the workflow and uses heuristics and an LLM judge to check for a greeting such as Hi, Jared,, an email body, and a signature. Code-based checks are another option. He reports an evaluation result of “a hundred” for this example; the sample size, scoring scale, and held-out status are not supplied.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the agent as a component in a larger workflow
A headless coding agent can supply an existing harness instead of requiring every application to build its own. Zoneraich describes a daily GitHub Action for documentation maintenance:
- Pull the relevant repositories and inspect newly pushed commits.
- Determine what changed.
- Read
CLAUDE.mdto decide whether those changes warrant documentation updates. - Make the updates and create a pull request.
An audience member asks whether he reviews the results. He confirms that the agent creates the PR but does not merge it. The automation therefore ends with a reviewable proposal, not an unattended publication.
This is a higher level of abstraction: an application can delegate repository navigation, tools, and orchestration to an existing agent. Zoneraich also imagines a team that combines Claude Code, Codex, and other agents, possibly exchanging messages through Slack. That is an aspiration, not a system demonstrated in the workshop. Its motivation is that different agent perspectives may complement one another, while context management and a restrained tool set remain important within each agent.
The slide deck itself is a smaller example of composition. Zoneraich asked Claude Code to research Slidev and build a skill for it. A deep-research skill investigated the coding agents, while a design skill helped refine boxes and accent colors. He supplied visual judgment—whether something looked good—without needing to specify every design operation himself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where fixed sequences and direct APIs still matter
The closing questions sharpen the limits of the simple-loop approach. If customer support must collect a name and then an email, how is that order enforced without a DAG? Zoneraich distinguishes general-purpose problem solving from a repeatable deliverable. Coding has no universal sequence. A travel itinerary has a more consistent output, but researching each city may require different actions. He would let the agent explore during research, then use a DAG-backed tool to create the standardized plan or output file. A prompt can request that the agent finish with that output; ordering that must be guaranteed belongs inside the constrained tool.
Another question asks whether direct model calls will give way to agent SDKs: instead of one LLM call per document, could a Claude Code loop process documents, save files, and produce a final summary? The attraction is simpler development through an existing harness. Zoneraich compares this, at a deliberately simplified level, to reasoning models such as o1 moving more work behind the provider’s interface. He can imagine providers exposing agentic endpoints, but direct model access remains useful when a task requires tighter control. His analogy to the move beyond completions interfaces is a forecast about abstraction, not a claim that every pipeline should become a coding-agent session.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tests, specifications, and the limits of reverse engineering
For coding with agents, good tests provide the feedback loop emphasized by Amp and Factory. Zoneraich relies heavily on planning and specifications for substantial work, but skips that phase for very simple edits. He does not prescribe test-driven development for every task. The useful question is whether the specification and checks make the desired result clear enough for the agent to improve its own work.
The final technical exchange also corrects the workshop’s research framing. An audience member asks whether Claude Code’s system prompt can be found in the downloaded JavaScript bundle or lives behind a server endpoint. Zoneraich initially speculates that it is hidden and mentions an account of modifying open-source Codex to send a custom prompt before its model was separately available. Pressed on Claude Code’s prompt location, he admits he does not know. He then relays Nico’s statement that it is on the local machine and acknowledges that the leaked prompt he used may be old. That leaves the workshop’s internal names and prompt-derived details as a historical investigation, rather than an authoritative implementation specification.
Asked how people can help, Zoneraich closes with an invitation to the New York team, which was hiring, and gives @jaredz as a contact. He identifies PromptLayer as the source of the evaluation screenshots and describes its scope: prompt management, logging, evaluations, auditability, and governance, with collaboration among engineers, product managers, and nontechnical experts. That collaboration is the practical setting for the architecture: people define the result they need, the agent explores how to produce it, and tools and evaluations make the important requirements inspectable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Introduces moving task-relevant context into a new thread instead of repeatedly compacting an existing one.
Public implementation and setup instructions for OpenAI's terminal coding agent.
A library for generating structured JSON by constraining language-model output.
Developer-oriented tools for creating presentations with Markdown and web technologies.
Further reading
Anthropic's explanation of selective retrieval, tool design and managing limited context in agents.
- Claude Code sandboxingArticle
Explains filesystem and network isolation for shell execution, including OS-level enforcement.
- Agent Skills architectureArticle
Explains how agents load specialized instructions and supporting resources as needed.
Cursor's original Composer announcement explains reinforcement learning and the conditions behind its reported generation speed.
Updates since the talk
Practical guidance for repository instructions, verification, skills, subagents and context management.
Read the complete timestamped transcript
- 0:00
[upbeat electronic music] So welcome to the last workshop.
- 0:22
Um, you made it. Congrats. [laughs] Yay. [audience clapping] Out of, like, 800, uh, people, you're the, you're the last standing, uh, so the very, very dedicated engineers. Uh, yeah, so this one's a weird one.
- 0:36
I got in trouble with Anthropic, with Anthropic on this one, uh, obviously because of the title. [laughs] I actually also gave him the title, and I was like, "Do you wanna change it?"
- 0:43
And he was like, "No, I just roll with it." [laughs] It's kind of funny. [laughs]
- 0:47
Uh, uh, so, so yeah, this is not officially endorsed by Anthropic, but we're hackers, right? And Jared is, like, super dedicated. He's, um... Uh, and the other thing I also, like, really enjoy is featuring, like, notable New York AI people, right?
- 1:01
Like, so don't take this as, like, this was the only thing that Jared does. He has a whole startup that you should definitely ask him about. Um, but, like, you know, I'm just really excited to feature more content from local people.
- 1:13
So yeah, Jared, take it away.
- 1:15
Thank you very much. Thank you very much, and what an amazing conference. Very sad we're ending it, but hopefully it'll be a good ending here. Um, and yeah, uh, my name is Jared.
- 1:26
Uh, this'll be a talk on how Claude Code works. Again, not affiliated with Anthropic. Uh, they don't pay me. I would take money, but they don't. Um, but we're gonna talk about a few other coding agents as well, and kind of the high level goal that I'll go into is, me personally, I, I'm a big user of
- 1:47
all the coding agents, as is everyone here, and they kind of exploded recently. And as a developer, I was curious what changed, what made it finally... What made coding agents finally be good.
- 2:00
So let's get started. I'll start about me. I'm Jared. You can find me, I'm [REDACTED:username] on X, on Twitter, whatever. Um, I'm building the workbench for AI engineering, so, uh, my company's called PromptLayer.
- 2:13
We're based in New York. You can kind of see our office here. It's like a little building, so it's blocked by a few of the other buildings. So we're, we're a small team.
- 2:21
We launched the product three years ago, so, uh, long for AI, but small for everything else. And, uh, yeah, what, kind of our core thesis is that we believe in rigorous prompt engineering, rigorous agent developing, development, and we believe that the product team should be involved, the engineering team should be v- involved.
- 2:40
We believe if you're building AI lawyers, you should have lawyers involved, as well as engineers. Um, so that's kinda what we do, uh, processing millions of LLM requests a day, and a lot of the insights in this talk come from just conversations we have with our customers on how to build coding agents and stuff like that.
- 2:56
And al- also feel free throughout the talk, we can make this casual, so if there's anything I say, if you have a question, feel free to just throw it in.
- 3:04
Uh, and I spend a lot of my time kind of dogfooding the product. It's kinda weird, the job of, of a founder these days, because it's half, like, kicking off agents and then half just using my own product to build agents.
- 3:15
And it feels weird, but it's kinda fun. And, uh, yeah, the last thing I'll add here is I'm a big enthusiast. We literally rebuilt our engineering org around Claude Code.
- 3:26
I think the hard part about building a platform is that you have to deal with all these edge cases and, oh, uh, we're uploading data sets here, it doesn't work, and you could die a death by a thousand cuts.
- 3:38
So we made a rule for our engineering organization, if you can complete something in less than an hour using Claude Code, just do it. Don't prioritize it. And we're a small team on purpose, but, uh, it's helped us a lot, and I think it's really taken us to the next level.
- 3:54
So I'm a big fan, and let's dive into how these things work. So this is what I, as I was saying, the goal of this talk. First, why have these things exploded?
- 4:03
What is the... What was the innovation? What was the invention that made coding agents finally work? If you've been around this field for a little bit, you know that, uh, a lot of these autonomous coding agents sucked at the beginning.
- 4:17
And, uh, we all tried to use them, uh, but it's, it's night and day. Uh, we'll dive into the internals. And, and lastly, we'll, like, everything in this talk is oriented around how do you build your own agents and how do you use this to do AI engineering for yourself.
- 4:36
So let's just go, uh, talk about history for a second here. How did we get here? Uh, everybody knows, started with, uh, you remember the workflow of you just copy and paste your code back from ChatGPT back and forth, and that was great, and that was kind of revolutionary when it happened.
- 4:53
Uh, step two, when Cursor came out, if we all remember, it was not, not great software at the beginning. It [laughs] was just the VS Code fork with the Command+K, and we all loved it.
- 5:06
But, uh, now, now we'll no- we're not gonna be doing Command+K anymore. Then we got the Cursor assistant, so that little agent back and forth, and then Claude Code.
- 5:15
And honestly, in the last few days since I made this slide, maybe there's a new version we could talk about here. And, uh, at the end I'll talk about, like, kind of what's next.
- 5:23
But this is how we got here, and this is really, I think the Claude Code is kind of this headless, uh, not even, this, this new workflow of not even touching code.
- 5:33
And it has to be really good. So why is it so good? What, what was, uh, what was the big breakthrough here? Let's try to figure that out. And again, throw this in one more time, these are all my opinions, uh, and what I think is the breakthrough.
- 5:47
Maybe there's other things, but simple architecture, I think a lot of things were simplified with how the agent was designed. And then better models, better models, and better models.
- 5:58
Uh, I think the- A lot of the breakthrough is kind of boring in that it's just Anthropic releasing a better model that works better for th- these type of tooling calls and these type of things.
- 6:10
But the simple architecture relates to that. So we could di- dive into that. The architecture, and, and this is our little... You'll see, uh, prompt wrangler is our little mascot for our company, so we made a lot of graphics for these slides.
- 6:24
But, uh, basically, give it tools and then get out of the way is what a one-liner of the architecture is today. I think
- 6:35
if you've been building on top of LMS for a little bit, this has not always been true. Obviously, tool calls haven't always existed, and tool calls is kind of this new abstraction for JSON formatting, and if you remember the GitHub libraries like JSON Former and stuff like that in the olden days.
- 6:51
But give it tools, get out of the way. Uh, the models are built for these things and being trained to get better at tool calling and better at this.
- 7:01
So the more you wanna over-optimize, and every engineer, uh, including my- especially myself, loves to over-optimize. And when you first have an idea of how to build the agent, you're gonna sit down and say, "Oh, and then I'm gonna prevent this hallucination by doing this prompt, and then this prompt, and then this prompt."
- 7:17
Don't do that. Just a simple loop and get out of the way, and just delete scaffolding, and less, less scaffolding, more model is kind of the tagline here. And, you know, this is, uh, the leaderboard from this week.
- 7:34
Obviously, these models are getting better and better. Uh, we could have a whole conversation, and I'm sure there's been many conversations about, is it slowing down, is it plateauing?
- 7:43
It doesn't really matter for th- [laughs] this talk. It-- we know it's getting better, and they're getting better at tool calling, and they're getting better optimized for running autonomously. And don't...
- 7:54
This is... I, I think Anthropic calls this, like, the AGI pill to-- way to think about it, is don't try to over-engineer around model flaws today, because a lot of the things will just get better, and you'll be wasting your time.
- 8:09
So here's the philosophy the way I see it of Claude Code. Ignoring embeddings, ignoring classifiers, ignoring pattern matching. The-- we had this whole RAG thing. Actually, Cursor's bringing back a little bit of RAG in how they're doing it, and they're mixing and matching, but I think the genius with Claude Code is th- they, they scratched all this
- 8:29
and they said, "We don't need all these fancy, uh, paradigms to get a- around how the model's bad. Let's just make a better model and then let it, let it cook." [laughs]
- 8:39
And, uh, just leaning, uh, uh, on these tool calls and
- 8:46
simplifying the tool calls, which is a very important part, part. Instead of having a workflow where the master prompt can break into three different branches and then go into four different branches, there, there's really just a few simple tool calls, uh, including Grep instead of RAG, and, uh...
- 9:06
Yeah, and that's kinda what it's trained on. So, uh, these are very optimized tool-calling models. So this is,
- 9:15
uh, [laughs] the zen of Python if, if you guys are familiar, if you do import this in Python. This is... I love this philosophy when it comes to building systems, and I think it's really
- 9:26
apt for how Claude Code was built. So really just simple is better than complex. Complex is better than complicated. Flat is better than nested. This is, this is all you need to...
- 9:36
This is the whole talk. [laughs] This is all you need to know about how Claude Code works and why it works specifically. That just, in eng-- We're going back to engineering principles such that simple design is better design.
- 9:49
Uh, I think this is true whether you're building a,
- 9:55
uh, database schema, uh, but this is also true when you're building these auten- au- autonomous coding agents. So let's... I'm gonna now kinda break down all the specific parts of this coding agent and, uh, why I think they're interesting.
- 10:12
So the first is the constitution. Now, a lot of this stuff we kinda take for granted, even though they started doing it a month or two ago, or maybe three or four months ago.
- 10:20
So this is the Claude MD, Codex or others use Agents MD. The interesting thing, uh, I think, I assume most of you know what it is. Uh, it's, again, it's where you put the instructions for your library, but the interesting thing about this is
- 10:36
it's basically the team saying, we don't need to over-engineer a system where the model f- first researches the repo and Cursor, uh, like Cursor 1.0, a- as you know, makes a vector DB locally to understand the repo and kind of does all this research.
- 10:55
They're just saying, "Eh, just put a markdown file. Let the user change stuff when they need. Let the agent change stuff when they need." Very simple and kind of goes back to prompt engineering, which I'm a little biased towards because Prompt Layer is a prompt engineering platform, but, uh, everything's prompt engineering at the end of the day,
- 11:13
or context engineering. Everything is how do you, uh, how do you adapt these general purpose models for your usage? And the simplest answer is the best one here, I think.
- 11:24
So this, this is the core of the system. It's just a simple master loop. Uh, and, and this is actually kind of revolutionary considering how we used to build agents.
- 11:38
Everything in Claude Code and, and all the coding agents today, Codex and, and, and, uh, the new Cursor and AMP and all that, it's just one while loop with tool calls just running the master while loop, calling the tools, and going back to the master while loop.
- 11:52
This is [laughs] basically four lines of what it's called. I think they call it N0 internally, uh, at least based on my research. But while there are tool calls, run the tool, give the tool results to the model, and do it again until there's no tool calls, and then Ask the user what to do.
- 12:11
The first time I did this, uh, the first time I used tool calls, it was very shocking to me that the models are so good at just knowing when to keep calling the tool and knowing when to fix their mistake.
- 12:22
And I think that's one of the most interesting thing about LLMs, just they're really good at fixing mistakes and being flexible. And the more, just going back, the more you lean on the model to explore and, uh, figure it out, the better and more robust your system is gonna be when it comes to better models.
- 12:45
So, so these are the core tools, uh, we have in Claude Code today, and to be honest, these change every day. You know, they're doing new releases every few days, but these are the core ones that I found most interesting to talk about.
- 13:01
Uh, there could be 15 tomorrow. There could be down to five tomorrow, but this is what I find interesting. So first of all, read. Uh, yeah, they could just do a cat, uh, but what's interesting is read is we have token limits.
- 13:16
So y- if you've used Claude Code a lot, you've seen that sometimes it'll say, "This file's too big," or something like that. That's why it's worth building this read tool.
- 13:25
Grep glob. Uh, this one's very interesting too because it goes against a lot of the wisdom at the time of using RAG and using vectors. And I'm not saying RAG has no place by the way either, but in these general purpose agents, Grep is good and, and, and Grep is, uh, how users would do it.
- 13:44
And I think that's actually a high level point here. As y- as you're, a- as I'm talking about these tools, remember, these are all human tasks. They're not... We're not making up a brand-new tool for the model to use.
- 13:56
We're kind of just mimicking the human actions and what you and I would do if we were at a terminal trying to fix a problem. Edit. Edit makes sense.
- 14:04
I think the interesting thing to note in edit is it's using diffs, and it's not rewriting files most of the time. Uh, way faster, way m- way, uh, less context used, but also way less,
- 14:17
uh, issues. Uh, if, if I asked you to... If I, if I gave you these slides and asked you to review the slides, and you read it and had to write down all the slides for me in your new revisions versus if you could just cross out things in the paper, the crossing out is way easier.
- 14:33
Diff is kind of a natural thing to prevent mistakes. Bash. Bash is, uh, Bash is the core thing here, I think. You could probably get rid of all these tools and only have Bash, and the first time I saw this, when, when you run something in Claude Code and Claude Code creates a Python file and then runs
- 14:53
the Python file, then deletes the Python file, that's, that's the beauty of why this thing works. So Bash is the most important, I'd say. Web search, web fetch. Uh, the interesting thing about these is they mo- move it to a cheaper and faster model.
- 15:08
So for example, if you're building a, some sort of agent maybe on your platform, and you're building an agent, and it needs to connect to some endpoints, some list of endpoints, might be worth to bring that into a kind of sub tier as opposed to that master while loop.
- 15:24
That's why this is its own tool. To-dos. Uh, we've all seen, seen to-dos. I'll talk about it a little bit more later, but keeping the model on track, steerability, and then tasks.
- 15:34
Tasks is very interesting. It's context management. It's how do we, how do we run this long process, read this whole file without cluttering the context? Because the biggest enemy here is when your context is full, the model gets stupid [laughs] for lack of better word.
- 15:50
So basically, Bash is all you need. Uh, I think this is the one thing I wanna drill down. The amazing thing about... There's two amazing things about Bash for coding agents.
- 15:59
The first is that it's simple, uh, and it does everything. It's, it's very robust, but the second thing that's equally important is there's so much training data on it because that's what we use.
- 16:11
It's not... It's the reason that models are not as good at Rust or less common programming languages, just because there's less people doing it.
- 16:22
So it's really the universal adapter. Um, you've thousands of tools. You could do anything. Uh, this is that Python example I gave. I, I, I always find it so cool when it does the Python script thing or creates tests, and I always have to tell it not to.
- 16:35
But it... All these shell tools are in it, and this is... I mean, I find myself using Claude Code to spin up local environments where normally I'd have like five commands written down on some file somewhere, and then they get out of date.
- 16:49
It's really good at figuring this stuff out and running the stuff you'd wanna do. Uh, and it specifically lets the model try things.
- 16:58
So, uh, yeah, the other suggestions here and the tool usage, uh, I think
- 17:06
there's a little bit of a system prompt, uh, that tells it which to use and when to use which tool over which, and this changes a lot, but the, these are kind of like the edge cases and the corners you find the model getting stuck in.
- 17:17
So reading before editing, uh, they actually mas- make you do that using Grep, the tool, instead of the Bash. So if you look at the tool list here, there's a special Grep tool.
- 17:30
Uh, there could be a lot of reasons for that. I think security is a big one, uh, and sandboxing, but then also just that token limit thing. Running independent operations in parallel, uh, so kind of pushing the model to do that more, and then also, like, these trivial things like quoting paths with spaces.
- 17:47
It's just the common, common things I'm sure they're just dogfooding a lot at Anthropic, and they find it, and they're like, "All right, we'll throw it in the system prompt."
- 17:55
Okay, so let's talk about to-do lists. Uh, now again, a very common thing, but was not a common thing before. The, the [laughs]... So this is actually, uh, I think a to-do list for, from some of my research for this slide deck.
- 18:09
Um, but- The really interesting thing about to-do lists is that they're structured, but not structurally enforced. So here are the rules, one task at a time, uh, mark them completed.
- 18:24
This is kind of stuff you would expect, uh, keep working on the in progress if there's block, uh, blocks or errors, and kind of break up the tasks into different instructions.
- 18:36
But the most interesting thing to me is it's not enforced deterministically. It's purely prompt-based. It's purely in the system prompt. It's purely because our models are just good at instruction following now, and this would not have worked a year ago.
- 18:51
This would not have worked two years ago. Um, there's tool descriptions at the top of the system prompt. We're kind of, uh, injecting the to-dos into the system prompt.
- 19:02
Uh, there's... They're not, uh, but it, but it's not enforced in actual code. And again, uh, maybe there's other agents that take an opposite path. Uh, I just found this pretty interesting that this, at least as a user, makes a big difference, and it doesn't even see...
- 19:18
It seems, it was, it seems like it was very simple to implement, almost a, [chuckles] a weekend project someone did and seemed to work. Could be wrong about that, about that as well.
- 19:27
But, uh, um, so yeah, it, it's literally a function call. Uh, it, it's the first time you ask something, the reasoning exports this to-do block, and I'll show you what the structure is on the next slide.
- 19:40
Uh, there's IDs. There, there's some kind of structured schema and determinism, but
- 19:47
it, it's just injected there. So here's a example of what it could look like. You get a version, you get your ID, uh, a title of the to-do, and then it could actually inject evidence.
- 19:59
So this is, uh, seemingly arbitrary blobs of data it could use, and the IDs are hashes that it could then refer to, title something human readable. But this is, uh, just another way to structure the data.
- 20:14
And in the same way that you're gonna organize your desk when you work, this is how we're trying to organize the model.
- 20:21
So I think there's, uh, these are kind of the four benefits we're getting. We're forcing it to plan. Uh, we get to resume after crashes, uh, Claude Code fails.
- 20:33
I think UX is a big part of this. As a user, you know how it's going. It's not just running off in a loop for forty minutes without any, uh, signal to you.
- 20:42
So UX is non-negligible, even though UX might not make it a better coding agent, it might make it better for us all to use, and, uh, the steerability one.
- 20:52
So here's two other parts that were under the hood. AsyncBuffer, so they called it H2A. Uh, it's kind of, uh, the IO process and how to decouple it from reasoning, and, and how to manage context in a way that you're not just stuffing everything you're seeing in the terminal and everything back into the model, which again, context
- 21:12
is our biggest enemy here. It's gonna make the model stupider. So we need to, uh, be a little bit smart about that and, and how we do compact and how we do summarization.
- 21:22
So here you see when it reaches capacity, it kind of drops the middle, summarizes the head and tail. Um, then we have the, uh, uh, that's the context compressor there.
- 21:32
So what is the limit? Ninety-two percent it seems like something like that. Uh, and, and how does it, how does it save long-term storage? That's actually another kind of advantage of Bash, in my opinion, and having a sandbox.
- 21:47
I would even make a prediction here that all your, all ChatGPT windows, all Claude windows are gonna come with a sandbox in the near future. It's just so much better because you can store that long-term memory, and I do this all the time.
- 22:00
I have, I have Claude Code skills for deep research and stuff like that, and I'm always instructing it, "Save markdown files," because the shorter the context, the quicker it is and the smarter it is.
- 22:12
So this is what I'm most excited about. We don't need DAGs like this. We...
- 22:18
I'll give you, I'll give you a real example. Uh, so some users at Prompt Layer, uh, different agents like customer support agent, basically everybody was building DAGs like this for the last two, two and a half years.
- 22:32
Uh, and it was crazy. Hundreds of nodes of, okay, this, if this user wants a refund, route them to this prompt, if they want this, and a lot of, uh, classifying prompts.
- 22:45
The advantage of this is you can kind of guarantee there's not gonna be hallucinations or guarantee there's not gonna be
- 22:52
refunds to people who shouldn't be having refunds or kind of the, that pro... It, it solves the prompt injection problem because if you're in a prompt that purely classifies it as X or Y, injecting doesn't really matter, especially if you throw out the context.
- 23:05
Now, we kind of brought back, bring back that attack vector, but the, but the major benefit is we don't have to deal with this web of engineering, uh, madness. [chuckles]
- 23:15
And, uh, it just, it's 10X easier to develop these things, 10X more maintainable, and it actually works way better because our models are just good now.
- 23:24
So this is, this is kind of a takeaway is rely on the model. Uh, when in doubt, don't, don't try to think through every edge case and think through every if statement.
- 23:36
Just rely on the model to explore and figure it out. And I was actually, two days ago, I think, or yesterday, sometime this week, I was doing a, uh, a experiment on our dashboard to add like trying these browser agents, and I wanted to see if I could add little titles to all our buttons, and it would
- 23:55
help the agent navigate our website automatically. And it actually made it worse, surprisingly. Uh, and maybe I could run it again, and maybe I did something wrong with this test, but it made the agent navigate Prompt Layer worse because it was getting distracted because I was telling it, "You have to click this button, then you have to
- 24:11
click this button," and then- It's, it, it didn't know what to do. So it's better to rely on exploration. You have a question?
- 24:19
Yeah. I'll, I'll push back a little bit.
- 24:22
Please.
- 24:23
I'll admit any scaffolding we create today to resolve the idiosyncrasies of limitations will be, like, that'll be obsolete three to six months. Even if that's the case, they help a little bit today, I don't...
- 24:40
How do you balance that, like, wasted engineering to solve a problem we only have for three months?
- 24:46
It's a great question. So just to repeat, uh, the question is basically,
- 24:51
uh, what is the trade-off between solving the actual problems we have today, and if you're relying on the model that can't do it yet, but it'll be able to do it in three months, right?
- 25:00
Um, it's case by case. It depends what you're building. If you're building a chatbot for a bank, you probably [laughs] do wanna be a little bit more comp- be careful.
- 25:09
To me, the happy middle ground is to use this agent paradigm of a master while loop and tool calls, but make your tool calls very rigorous. So I think it's okay to have a tool call that looks like this or looks like half of this, uh, in the same way that Claude Code uses read as a tool
- 25:29
call or grep as a tool call. So for the edge cases,
- 25:34
throw it in a structured tool that you can then eval in version and stuff like that, and I could talk, I'm gonna talk a little bit more about that later.
- 25:41
But throw it in that structured tool, but for everything else, uh, for the exploration phase, leave it to the model or throw some system prompt. Uh, so it's a trade-off, and it's very use case dependent, but I think it's a good question.
- 25:58
Thank you. So yeah, uh, just back to Claude Code, uh, we're, we're getting rid of all this stuff. We're saying we don't want ML-based intent detection. We don't want regex.
- 26:08
We don't want the... I mean, it uses regex a little bit, but we don't want regex baked into it. We don't want classifiers. And, and there was a long time, we actually built a product for PromptLayer.
- 26:17
We never released it because it was only a prototype of using a ML-based, like a non-LLM-based classifier in your prompt pipeline instead of LLMs. And a lot of people have a lot of su- success with it, but it, it feels more and more like it's not gonna be that helpful unless cost is a huge concern for you.
- 26:36
And even then, cost is, the smaller models, is going less and less as, uh, kind of financial engineering between all these companies pays for our tokens. [laughs] Um, so Claude does also s- this smart thing, I think, with the trigger phases.
- 26:52
You know, you have think, think hard, think harder, and ultra think is my favorite. Uh, and this lets us use the reasoning budget, the reasoning token budget, as another parameter that the model can adjust.
- 27:05
And this is actually ... The model can adjust this, but this is how we force it to adjust. And as opposed to you could make a tool call for hard planning, and actually there's some coding agents that do this, or you can, uh, let the user specify it and then just on the fly change it.
- 27:23
So this is, this is one of the biggest topics here, sandboxing and permissions. I'm gonna be completely honest, it's the most boring part of this [laughs] to me because I just run it on YOLO mode half the time. [laughs]
- 27:35
Um, it's, uh, [laughs] some people on our team actually dropped all their local databases, so you do have to be careful. Uh, so, uh, you know, w- we don't YOLO mode with our enterprise customers, obviously.
- 27:49
But, uh, I, but, but I think this stuff is ... It feels like it's gonna be solved, but, but we do need to know how it works a little bit.
- 27:57
So there's a big issue of inj- prompt injection from the internet. If you're connecting this agent that has shell access and you're doing a web fetch, that's a pretty big attack vector.
- 28:11
Uh, so there's some containerization of that. There's s- blocking URLs. You could see Claude Code's pretty annoying about can I fetch from this URL? Can I do this? And it kind of puts it into a sub-agent.
- 28:23
And, uh, yeah, uh, most of the, most of the complex code here is in this sandboxing and permission set. I think there's this whole pipeline to gate batch commands, so
- 28:36
it, depending on the prefix, is how it goes through the sandboxing environment. And a lot of the other models work differently here, uh, but this is how Claude Code does it.
- 28:46
I'll explain the other ones later at the end.
- 28:51
The next topic, uh, of relevance here is sub-agents. Uh, so this is going back to context management and this, this problem we keep going back to of the longer context the, the stupider our agent is.
- 29:02
This is a, this is an answer to it. So using sub-agents for specific tasks, and the key with the sub-agent is it has its own context, and it feeds back only the results.
- 29:12
And this is how you don't clutter it. So we got the researcher. These are just four examples, researcher, docs reader, test runner, code reviewer. In that example I was talking about earlier when I added all the tags to our website to let the agent do it better, I, obviously, I used a coding agent to do that, and
- 29:29
I said, "Read our docs first and then do it." And it's gonna do this in a sub-agent. It's gonna feed back the information. And the, the key thing here is the forks of the agent and how we aggregate it back into our main context.
- 29:44
So here's an example. I think this is actually very interesting. I wanna call out a thing or two here. So task is what a sub-agent is. We're giving task two things, description and a prompt.
- 29:56
The description is what the user's gonna see, so you're gonna say, "Task, uh, find default chat context instantiation," or something. And then the prompt, you're gonna give a long string, which is really interesting because now we have the coding agent prompting its own agents.
- 30:13
And I've actually used this paradigm in agents I've built for our product. Uh, if you can- You can just have the agent stuff as much information as it wants in this string, and if we're going back to relying on the model, if this task returns an error, now it could stuff even more information, and let it solve
- 30:33
the problems. It's better to be flexible rather than rigid.
- 30:37
If I was building this, I would consider switching a string to maybe an object here, uh, depending on what you're building, and maybe let it give actually more structured data.
- 30:46
Yes.
- 30:47
So I can see this prompt has quite a couple sentences. Is that in the main agent? Is that taking the context of the main agent, or is there, like, you know, some sort of intermediate step where, um, the sub-agent double reads over, you know, like, what the main agent is doing, and then, you know, kind of generates
- 31:04
its own slightly longer prompt or something?
- 31:06
Right. So the question is: Does the task just get the prompt here, or does it also get your chat history? Is that the question?
- 31:15
Yeah.
- 31:15
No, the, the question is, is, are all of the... I have my main agent. Are, is all of this in the system prompt of the main agent to inform how that prompts the sub-agent?
- 31:24
No, no.
- 31:25
Okay.
- 31:25
Like, it's not in the system prompt. It's in the whole context. Is the, all of this the context of the main agent?
- 31:33
The task it calls, or, or the, you're saying the structure for the task.
- 31:37
This whole JSON array, like, or, or-
- 31:39
Yes
- 31:40
...
- 31:40
So this is a tool call. So the tool call structure of what a task is, is in the main agent. Uh, and then these are generated on the fly.
- 31:49
Uh, so as you wanna write a task, it's generating the description and the prompt. Task is a tool call that could be run in parallel, and then they're returning the results of it.
- 32:00
Hopefully, that helps.
- 32:02
Thank you.
- 32:03
Um, so we could go back to the system prompt. So there's some leaks of the Claude Code system prompt, so that's what I'm basing this on. Uh, you can find it online.
- 32:13
Um, here are some things I, I noted from it. Uh, concise outputs, uh, obviously, don't give anything too long. No, "Here is" or "I will," just do the, do the task the user wants.
- 32:28
Uh, kind of pushing it to m- use tools more, more instead of text explanations, obviously. I think when we, we've all built coding agents, and when we do it, it usually says, "Hey, I wanna run this SQL."
- 32:40
No, push it to use the tool. Um, matching the existing code, not adding comments. This one does not work for me, but, uh- [laughs] ... running commands in parallel extensively, and then the to-dos and stuff like that.
- 32:54
There's a lot that you can nudge it to do with the system prompts, but as you see, I think there's a really interesting point to the earlier question you had about where, what's the trade-off between DAGs and loops.
- 33:08
A lot of these things, you could see are, feel like they came from someone using it, Claude Code, and saying, "Oh, if only it did this a little less," or, "If it did this a little bit more."
- 33:19
That's where prompting comes in, because it's so easy to iterate, and it's not, you're not, it's not a hard requirement, but if only it said, "Here is a little bit more."
- 33:28
It's okay to say it sometimes, but... All right, skills. Skills is great. It's, uh, slightly newer. I've, I honestly got convinced of it only recently. So good. I built these slides with skills.
- 33:41
Uh, it's basically, I think in the context of this talk about architecture, let's think of it as a extendable system prompt. So in the same way that we don't wanna clutter the context, there's a lot of different type of tasks you're gonna need to do where you want a lot more context.
- 33:58
So this is how we give Claude Code a few options of how it could tap into more information. Here are some examples. Uh, I use this for, I have a skill for docs updates to tell it my writing style and h- and my product.
- 34:14
So if I wanna do a docs update, I say use that skill, load in that skill. Uh, editing Microsoft Office, uh, uh, Microsoft Do- Microsoft Word and Excel, um, I, I don't use this, but I've seen a lot of people using it.
- 34:27
It kind of, like, decompiles the p- fi- it's really cool. Uh, but it lets Claude Co- Code do this. Design style guide, this is a common one, deep research.
- 34:36
I, the other day, I threw in a, like, article or GitHub, uh, repo on how deep research works, and I said, "Rebuild this as a Claude Code skill." Works so well.
- 34:46
It's amazing. So unified diffing, I think this is worth its own slide. Uh, it's very obvious, probably not too much we need to talk about here, but it makes this so much better, and it makes the token limit s- shorter, it makes it faster, it makes it less prone to mistakes, like I gave with that example, when
- 35:07
you rewrite an essay verse marking it with a red line. It's just better. I highly recommend using diffing in any agents you're doing. Unified diff is a standard. When I looked into a lot of these coding agents, some actually built their own kind of standard, uh, and, like, with slight variations on unified diff, because you don't always
- 35:27
need the line numbers, and... But unified diff works. You had a question?
- 35:32
To go back to skills, I are, uh... I don't know if anyone's seen the Claude, the Claude Code warns you in, in yellow text if your Claude MD gets, like, greater than 40K characters.
- 35:44
And so I was like, "Okay, I'm fucking up. Let me break this down into skills." So I be- spent some time, and then Claude ignored all of my skills, and, uh, so I put them somewhere else.
- 35:54
So what am I... I don't know. Skills feel globally, uh, misunderstood or, uh, like, not... I don't know. I'm missing something. Um, help me understand. [laughs]
- 36:05
Same.
- 36:06
Yeah, so the, the question was on... Okay, so Claude Code system, Claude MD, it tells you when it's too long, so, uh, you move it into skills, and then it's not recognizing the skills and not picking it up when it's needed.
- 36:20
It ignores.
- 36:21
Yeah. Take that up with the Anthropic team, I'd say. [laughs] Uh, but that's also a good example of maybe the system prompt-
- 36:31
You need to invoke them and, like, the agent itself shouldn't, like, just call them all the time or-
- 36:39
Right
- 36:39
... do the other thing.
- 36:39
It does give a desc- description of each skill to the model, or it should, uh, tell it, "Okay, here's, like, a one-liner about each skill." So theoretically, in a perfect world, it would pick up all the skills all the time, but you're right, I generally have to call the skill myself manually.
- 36:56
I- but I think this is a good tie-back into when is prompting the right solution or when is the DAG the right solution, or maybe this is a model training problem.
- 37:07
Maybe they need to do a little bit more in post-training of getting the model to call the skills is almost like calling a tool call. Uh, you have to know when to call it.
- 37:18
So maybe this is just a, a functionality that's not that good yet, but I think the paradigm is very interesting. But it's not perfect [laughs] as we're learning.
- 37:28
So diffing, we just talked about. What's next? So this is more opinion based, but where I see these things going and where the next kind of innovations might likely be.
- 37:39
So I, I th- there's two schools of thoughts here. A lot of people think we're gonna have one master loop with hundreds of tool calls and just tool calling is gonna get much better.
- 37:50
That's highly likely. Uh, I take the alternate view, which I think we need to reduce r- the tool calls as much as possible and just go back to just bash and maybe even put scripts in the local directory.
- 38:04
I think I am on the proponent of one mega tool call instead of a lot of tool calls. Maybe not actually one. I actually think that slide I showed you before is probably a good list, but a lot of people think we need hundreds of tool calls.
- 38:17
I just don't think it's going there. Adaptive budgets, uh, adjusting reasoning. We do this a little bit, uh, the thinking and ultrathink and stuff like that, but I, I think reasoning models as a tool makes a lot of sense as a paradigm.
- 38:32
Can you use ... I think a lot of us would make a trade-off of a twenty times quicker model with slightly stupider results and being able to call a tool call for a very good model.
- 38:44
I think that's a trade-off we'd, we'd make in a lot of cases. Maybe not our planner. Maybe we go to the planner first with GPT 5.1 Codex or Opus or whatever if the, when the new Opus comes out.
- 38:54
Uh, but I think, I think there's a lot of, uh, mix and matching we can do, and that's, I think, the next frontier. And I think the last frontier, I think there's a lot we can learn from to-do lists and, and new first class paradigms we can build.
- 39:08
Skills is another example of a first class paradigm we can kind of try to build into it. Maybe it doesn't work perfectly, uh, but I think there's a, I think there's a lot of new discoveries to be made there, in my opinion.
- 39:19
Do I have them? I don't know. Uh, so now I, I wanna h- for the, for the l- latter part of this talk, I wanna talk about the other frontier agents and the other philosophies they've designed, phils- philosophies they've chosen.
- 39:33
And we all have the benefit, we can mix and match. When we're building our agent, we can do whatever we want and learn from the best. And the frontier labs are very good at this.
- 39:43
So, uh, something I like to go back to a lot, I call it the AI therapist problem. May- maybe there's a better name to give it. Uh, but I believe there's a lot of problems, the most interesting AI problems around, there isn't a global maximum.
- 39:58
Meaning, all right, we're in New York City. If I need to see a therapist, there's six on every block here. There's no global answer for what the best therapist is.
- 40:09
There's different strategies. There's a therapist that does meditation or CBT or maybe one that gives you ayahuasca. A- and these are just kind of, like, different strategies for the same goal, in the same way that if you're building an AI therapist, there isn't a global maxima.
- 40:25
This is kind of my anti-AGI take, but this is also the take to say that when you're building these applications, taste comes into it a lot and design architecture matters a lot.
- 40:36
You can have five different coding agents that are all amazing. Nobody knows which one ... Today, nobody knows which one's best, to be honest. I don't think Anthropic knows.
- 40:44
I don't think OpenAI knows. I don't think Sourcegraph knows. Nobody knows whose has the best, but some are better at some things. I personally like Claude Code for, I said, like, running my local dev environment or using Git or using these kind of, like, human actions that require back and forth.
- 40:59
But I go to Codex for the hard problems, or I go to Composer from Cursor because it's faster. And there's a lot ... Basically, all this to say there's value in having different philosophies here, and I don't think there's gonna be one winner to this.
- 41:14
I think there's gonna be different winners for different use cases. And, and this is not just coding agents, by the way. This is all AI products. This is, this is kind of why our whole company focuses on domain experts and bringing in the PM and the, the, the subject matter expert into it, because that's how you build
- 41:30
defensibility. So here are the perspectives the way I see it. This is not a complete list of coding agents, but these are the ones that I think are the most interesting.
- 41:39
Claude Code, I think, I think to me, it wins in user-friendliness and simplicity. Uh, like I said, if I'm doing something that requires a lot of applications that ...
- 41:49
Git. Git's just the best example. If I wanna make a PR, I'm going to Claude Code. Codex, uh, context. It's really good at context management. Uh, it feels powerful.
- 41:59
Do I have the evidence to show you that it's more powerful? Probably not. But, uh, it feels that way to me. And the market feel ... There's a whole another conversation here to say the market knows best and what people talk about knows best, but I don't know if they know either.
- 42:14
Cursor IDE is kind of that perspective, model agnostic. It's faster. Factory, uh, makes Droid. Uh, great team. They were here too. Uh, they have multiple ... They, they really specialize these Droid sub-agents they have.
- 42:28
So that's kind of their edge, and that's maybe a DAG conversation too, or maybe a model training. Uh, cognition, uh, so Devin, uh, kind of this end-to-end autonomy, self-reflection, AMP, which I'll talk about more in a second.
- 42:42
They have a lot of interesting perspectives, and actually I find them very exciting these days. Free, it's model agnostic, uh, and there's a lot of UX sugar for users.
- 42:52
And I- actually, I love their design. Their, their talks at this, this conference, they, they, they have very, very unique perspectives. So let's start with Codex because it's a popular one.
- 43:03
So it's pretty similar to Claude Code. Uh, same master while loop. Most of these do, because that's just the winning architecture. Uh, interestingly, Rust core, uh, the cool thing is it's open source, so you can actually use Codex to understand how Codex works, which is kind of what I did.
- 43:20
Um, it's a little more event-driven, a little more, uh, work went into concurrent threading here, uh, kind of submission queues, event outputs, kind of the, the thing I was talking about with the IO buffer in Claude Code, I think they do it a little bit differently.
- 43:39
Uh, sandboxing is very different, so theirs is more, y- I mean, you could see here macOS seatbelt and Linux Landa. Theirs is more kernel-based. Uh, and then
- 43:50
sta- kind of this, it's all under threading and, and permissions is how I'd say it's mostly different. And then the real difference is the model [laughs] to be honest. Uh, so this is a, this is actually me using Claude Code to understand how Codex works.
- 44:08
Uh, so you see we have a few Explore. I didn't talk about Explore, but, uh, it's, uh, it's a, it's another sub-agent type. As y- as I, as I mentioned, these go in and out.
- 44:18
Uh, but yeah, this is researching Codex with Claude Code. It's always a fun thing to do. So let's talk about AMP.
- 44:25
So this is Sourcegraph's coding agent. I, it has a free tier. That's just a cool perspective in my opinion. Uh, they leverage kind of these excess tokens, uh, from providers and they give ads.
- 44:37
So we actually have an ad on them. I think it's a cool... I'm pro-ad. A lot of people are anti-ad. I think it's one of my hot takes, but I like it.
- 44:45
They don't have a model selector. This is very interesting too. This is its own perspective. Uh, it actually helps them move faster because you're, you have less of an exact expectation of what the output is because you know they might be switching models here and there.
- 45:00
So that changes how they develop. And then, uh, I think their vision is pretty interesting. Uh, their vision is, how do we build not just the best agent, but how do we build the agent that works with the most agent-friendly environments?
- 45:18
And actually, Factory gave a talk similar to this as well. But how do, how do you build a hermetically sealed, uh, a, like, coding repo that the agent can run tests on?
- 45:28
How do you build the feedback loop? Because that's kind of the Holy Grail. That's how we build an autonomous agent. And how do we, uh, I'd love to see the front end version of this.
- 45:35
How do we let it look at its own design and make it better and go back and forth? And this is kind of their guiding philosophy, and you could boil it down to the agent perspective, as I've been calling it.
- 45:48
I think they do interesting stuff with context. So we're all familiar with Compact. It's the worst. You have to wait ten min- I don't know why it takes so long.
- 45:56
Uh, and if you're not familiar, it's summarizing your chat window when the context gets too high and giving the summary. So they have something called Handoff, which makes me think of, if you, if any was a, anyone was a Call of Duty player back in the day, switch weapons.
- 46:09
It's faster than reloading. And, uh, that's what Handoff is. It, you're, you're just starting a new thread and you're giving it the information it needs for a new thread.
- 46:18
That feels like the winning strategy to me. Could be wrong, but maybe you need both. That's where they're pushing it. And I, I kind of like that. I, they give, they give a very fresh perspective.
- 46:28
So the second thing is model choice. This is the reasoning knobs, uh, and their view on it. They have fast, smart, and oracle. So they lean even more heavily into, we have different models.
- 46:41
We're not telling you what oracle is. They tell you, but we're willing to switch what oracle is, but we're gonna use oracle when we have a very hard problem.
- 46:50
So yeah, so that's AMP. Let's go to Cursor's Agent. I think Cursor's Agent has a very interesting perspective here. First, obviously, it's UI. Uh, UI first, not CLI. I think they might have a CLI, not entirely sure, but the UI is the interesting part.
- 47:06
It's just so fast, their new model composer. It's distilled. They have, they have the data. They actually made, in my opinion, people interested in fine-tuning again. Fine-tuning, it was almost, uh, we'd never recommend it to our customers, but Composer shows you that you can actually build defensibility based on your data again, which [laughs], which is, uh, surprising.
- 47:28
But, uh, yeah, Cursor's Agent Composer, I've been almost switching completely to it since because it's just so fast. It's almost too fast. Accidentally pushed to master on one of my personal projects, uh, so you don't, you don't want that always.
- 47:40
Uh, but Cursor was just the crowd favorite. And, and I want to give a lot of, uh, props to their team. They built iteratively. The first version of Cursor was so bad and it was...
- 47:52
And we all used, I used it because it's a VS code for, for it, I have nothing to lose, and it's gotten so good. It's such a good piece of software and it's a great team.
- 48:00
And, uh, but I, I'll say the same thing can be said about OpenAI's Codex models. They're not quite as fast, but they are optimized for these coding agents and they are distilled.
- 48:11
And I could see OpenAI coming out with a really fast model here because they also have the data.
- 48:18
So h- here's a picture. Um, I think you could... This is a picture they put on their blog, and you could see what their perspective is on coding agents here, just based on the fact that they show you the three models they're running.
- 48:30
So they're offering Composer, but they're letting you use the state of the art because they know that maybe GPT 5.1 is better at planning, or here it's 5, but now we have 5.1
- 48:42
So here begs the big question, which one should we all use? Which architecture is best? What should we do? And, uh, my opinion here is that benchmarks are pretty useless.
- 48:55
Benchmarks have become marketing [chuckles] for a lot of these model providers. Every model beats the benchmarks. I don't know how that happens, but [chuckles]
- 49:04
I think there's a, there's world where evals matter here, and
- 49:09
the question is what you can eval. The question is how this whole simplis- so simple while loop architecture that I've been kind of trying to push based on my understanding of it actually makes it harder to eval because if we're relying more on model flexibility, how do you test it?
- 49:26
You could run an integration test, kind of this end-to-end test, and just say, "Does it fix the problem?" That's one way to do it. You could break it up.
- 49:33
You could kind of do point in time snapshots and say, "Hey, I'm gonna give a context to my chatbot from like a half-finished conversation where I know it should be running a specific tool call."
- 49:43
I could run those, uh, I c- or I could maybe just run a back test and say, how, how often does it change the tools? I think there's also another concept here that's starting to be developed called agent smell, or at least I'm calling it agent smell.
- 49:57
So run an agent and see how many times does it call a tool call, how many times does it retry, how long does it take? And these are all surface level metrics, but it's really good for sanity checking.
- 50:06
And these things are hard to eval. There's a lot that goes into it. I'll show you an example of what I did, uh, just to kind of dive into it.
- 50:15
But, but on that subject, maybe I'll just say one more thing. I would break it down ment- me- my mental model is you could do an end-to-end test, you can do a point in time test, or what I most often recommend is just do a back test.
- 50:31
Start with back test, start capturing historical data, and then just rerun it. So yeah, let me give you, uh, this example. So basically, what I have here... So this is a screenshot of PromptLayer.
- 50:42
This is our... Our evals product is also just a batch runner, so you could kind of just run a bunch of columns through a prompt. But in this case, I'm running it through not a prompt, but Claude Code.
- 50:52
So I just have like a headless Claude Code, and I'm taking all these providers, and I just, my headless Claude Code says, I think I have it on the next slide, "Search the web for the model provider."
- 51:02
It's given to you in a file variables. Find the most recent and largest model released and then return the name. So I don't know what it's doing. It's doing web search.
- 51:10
I'm not even caring about that. This is an end-to-end test. This is how we kind of try doing Claude Code, and I actually think there's a lot about putting Claude Code into your workflows and those type of headless SDKs.
- 51:23
I'll talk about that, I think, next slide. But
- 51:27
kind of main takeaway here is you can kind of start to do end-to-end tests. You can look at it from a high level, do a model smell, and then kind of look into the statistics on each row and see how many times it called a tool.
- 51:41
And going back, and we, we've talked about this a lot in this talk, rigorous tools. The tools can be rigorously tested. You can... This is how you offload the deter- this is how you offload the determinism to different parts of your model.
- 51:55
It's you test the tools. You, you test the shit out of your tools. Look at them like functions. It's an input and an output. If your tool is a sub-agent that runs, then we're in a kind of recursion here because then you have to go back and test the end-to-end thing.
- 52:08
But for your tools, I'll give you this example. If I... So the, in my coding agents, or my agents in general, my autonomous agents, if there's something very specific that I want to output, so in this case, if I have a very specific type of email format or type of blog post that I want to write, and
- 52:30
I really want it to get my voice right, I don't want to rely on the model exploration. I want to actually build a tool that I can rigorously test.
- 52:38
So in this case, this is also just a PromptLayer screenshot, but this is a, like a workflow I've built. It has an LLM assertion where it says, "Check if the email is good to my standards."
- 52:48
If it's good, it revises it. If it's not good, it adds the parts, so like the header that it missed, and it revises it with the same step. This is obviously a s- a very simple example, but
- 53:00
in, I, we have another version for some of our SEO blog posts that has like twenty different nodes and writes an outline from a deep research and then fixes a conclusion and adds links.
- 53:12
And for the stuff that you have a very specific vision, that's when testing it just gets so much easier. Because as you can see, obviously, testing this sort of workflow has less steps and less flexibility.
- 53:26
So this is an eval I made. I start with just a bunch of sample emails. I run the prompt, or actually, I run the, the agentic workflow here, and I'm just adding a bunch of heuristics.
- 53:37
So this is a very simple LLM-as-judge. Does it include three parts in it? So this is what I was testing for, like the, "Hi, Jared," email body and the signature.
- 53:46
You can get a lot more complicated. You could do a code execution. You can do... I don't know. LLM-as-judge is usually the easiest, but now obviously you could see, I could keep running this until it's correct on all of them and kind of, uh, see my eval over time.
- 54:00
This is just from this example. I got it to a hundred, so that was fun.
- 54:04
Uh, and then I wanna, I wanna add another future-looking thing. Keep an eye on headless, uh, Claude Code SDK. I know there was a talk about it this morning, um, so I don't wanna, I won't spend too much time on it, but
- 54:19
it's amazing. You just give a simple prompt, and it's just another part of your pipeline. I use it for... I think I have it on the next slide. I have a GitHub action that updates my docs every day and just reads all the commits we've pushed to our other repos.
- 54:34
And we have a lot of commits going, and it just runs Claude Code. Claude Code pulls down all the repos, checks what's updated, reads our Claude MD to see if it should even update the docs.
- 54:46
And then creates a PR. So I think this unlocks a lot of things, and there's a possibility that we're gonna start building agents at a higher order of abstraction and just rely on Claude Code and these other agents to do a lot of the harnesses and orchestration.
- 55:02
Are you reviewing those?
- 55:03
Yeah. [laughs] I- it creates a PR. It doesn't, uh, it doesn't merge the PR.
- 55:12
So here are my takeaways. Number one, trust in the model. Uh, when in doubt, rely on the model when you're building agents. Number two, simple design wins. Number one and number two kind of go together here.
- 55:26
Number three, Bash is all you need. Go simple with your tools. Don't have forty tools, have ten or five tools. Four, context management matters. This is the boogeyman we're running from all the time in agents at this point.
- 55:43
Maybe there'll be new models in the future that are just so much better at context, but there's always gonna be a limit because, ah, you're talking to a human.
- 55:52
I forget people's names if I meet too many in one day. That's context management [laughs], or my stupidity, I don't know. And number five, different perspectives matter in agents.
- 56:02
I think this is... The engineering brain doesn't always comprehend this as much as it should, especially in, and I'm an engineer, so I'm also talking about myself. But the- different perspectives matter such that there's different, uh, ways to solve a problem where there's not one is better than the other, and you kind of, m- you probably want
- 56:23
a mixture of experts agent. I, I would love to have mine run Claude Code and Codex and this and give me the output and consider it a team and maybe have them talk to each other in a Slack-based message channel.
- 56:34
I'm waiting for someone to build that. That would be great. But these are my takeaways. Uh, my bonus thing that I'll show you is how I built this slide deck using Claude Code.
- 56:43
So, uh, I built a slide dev skill. So I, I basically told Claude Code to research how slide dev works and how it can, uh, and that's kind of just a library that I made this in.
- 56:54
I built a deep research skill to research all these agents and how they work. I built a design skill because I, I know if a thing looks terrible or looks good, but I'm not a good designer to figure it out.
- 57:05
So these boxes even, I was just like, "Oh, ma- make the box a little nicer. Give it an accent color." Uh, so yeah, this is how I built it.
- 57:13
But again, thank you for listening. Uh, happy to answer any questions. I'm Jared, founder of PromptLayer. [clapping] Find me there.
- 57:24
Yes.
- 57:25
Thank you. Great talk. Um, so you mentioned, um, regarding DAGs, basically like let's get rid of them, right? But, um, DAG's gonna enforce this like sequential, uh, execution, right, of paths, I don't know, customer service, like agent asks the name, email, right, like in some sort of, um, uh, sequence.
- 57:47
Um, so are you saying just write this out, um, like th- this is now, th- this should be, uh, just written out as a plan for an agent to execute and just trust that the model is gonna be calling those tools in that sequence?
- 58:04
Like how do we enforce, uh, the order?
- 58:07
Right. So the question was, why do I keep talking about getting rid of DAGs? How else am, are you supposed to enforce a specific order for solving a problem?
- 58:19
So I think there are different types of problems. So the problem of building a general purpose coding agent that we can all use to do our work and even non-technical people can use, there's no specific step to solving that problem, which is why it's better to rely on the model.
- 58:37
If your problem was to build, let's say, a travel itinerary,
- 58:46
it's more of a specific step because you have a deliverable that's always the same. So there's a little bit more of a DAG that could matter, but in the research step of traveling, you probably don't want a DAG because every city is gonna be different.
- 58:58
So it really depends on the problem you're solving. I would, if I wanted to make an agent for a travel itinerary, I'd probably have my tool call,
- 59:07
w- one of my tool calls be a DAG of creating the output file because I want the output to look the same, or creating the plan. And then in the system prompt, I could say, "Always end with the output," for example.
- 59:17
But you need to mix and match. There's a, every use case is different, but if you wanna make something general purpose, my take is to rely more on the model, on simple loops, and less on a DAG.
- 59:31
Cool. Any other questions? Yes.
- 59:34
Yeah, building on that point, like do you think we're heading towards a world where most of, you're not actually gonna call the API through code and that most LLM calls are by triggering Claude Code and is write, just writing the files instead?
- 59:50
So the question is, are we gonna move away from calling models directly and just calls call, call like headless Claude Code? Right?
- 59:58
Yeah. Like if I had a, like I have a pipeline that does one LLM call per document, summarizes it at the end. You could make a while loop Claude Code that saves a file every time.
- 1:00:10
You never call the API besides using Claude Code in a, in a while loop.
- 1:00:16
Potentially. Uh, I'll give you the pro and the con there.
- 1:00:20
Yeah.
- 1:00:21
The pro is it's easier to develop, and we can kind of rely on the frontier mo-- I mean, if you think about it, a reasoning model is just that.
- 1:00:30
The reasoning models didn't always exist. We just had normal LLM model and then, oh, now we have O1 and reasoning models. All that is is a, I mean, it's a little more complicated than this, but it's basically just a while loop on OpenAI's server that keeps running the context and then eventually gives you the output.
- 1:00:44
In the same way that Claude Code SDK Is a while loop with a bunch of more things. So I could totally see m- a lot of builders only touching these agentic endpoints, maybe even seeing a model provider release a model as a agentic endpoint.
- 1:01:01
But for a lot of tasks, you're gonna want a little bit more control, and they're probab- and probably you'd still wanna go as close to metal as possible. Having said that, there's, there's a lotta people who still wanted completions models, and that never happened, and nobody really talks about that anymore.
- 1:01:20
So it's very likely that everything just g- becomes this SDK. But I don't have a crystal ball, but those are, that is how I, I would think about it.
- 1:01:29
Yes.
- 1:01:30
Thanks for your talk. Um, I know you said the simpler the better, but, um, what's your thoughts about test-driven development, uh, spec-driven development in AI? Have you tried it?
- 1:01:40
What do you think about it?
- 1:01:42
For building agents, or for getting work done?
- 1:01:45
For coding.
- 1:01:46
Okay. So the question on spec-driven development, test-driven development for coding with agents.
- 1:01:55
When in doubt, go back to good engineering practices is what I would say. So i- if you
- 1:02:04
... [laughs] A- and there's, there's whole engineering debates on if test-driven development is the right way, and some people swear by it, and some people don't. So I don't think there's an answer.
- 1:02:13
I think coding agents, clearly test-driven development makes it easier. I think, as I was showing you, that's AMP's, Sourcegraph's, whole philosophy that if you can build good tests, and Factory I think thinks this as well, if you could build good tests, your coding agent can work much better.
- 1:02:29
So it makes sense to me. When I'm working personally, I rely pretty heavily on the planning phase and the spec-driven development phase, and I think the simpler tasks are pretty easy for the model, but if I'm doing a very simple edit, I'll skip that step.
- 1:02:43
So no one size fits all, but return to the engineering principles that you believe when in doubt, I'd say.
- 1:02:52
Yes.
- 1:02:53
So early, ear- early you talked about sy- about, uh, system prompt leaks. Um, is it not possible to just look at the, uh,
- 1:03:02
at the downloaded JavaScript bundle, or do they have a special endpoint that has some prompts behind the endpoint?
- 1:03:08
Yeah. Uh, I think, I think they hide it. I think they hide it. There was a, there was actually a interesting article of someone ... Because Codex is open source, they w- before OpenAI released the Codex model that it was using, they were able to hack together the open source Codex to give a custom prompt to the
- 1:03:28
model and be able to use the model without it. So yeah, you can dive into it, but, uh, generally it's tried to be hidden, and also laziness of someone posted it, so there you go.
- 1:03:39
That's the work. [laughs]
- 1:03:40
Yeah.
- 1:03:40
But someone had to have found it, right?
- 1:03:41
Yeah, but like, is the system prompt somewhere on your machine?
- 1:03:46
I actually don't know that answer. [laughs]
- 1:03:49
Just have it.
- 1:03:49
Do you know that answer? Yeah. Yes? Yeah, it's on your machine. It's on your machine. Nico says it's on your machine. So there we go. So maybe the prompt I was looking at is a little bit old, and I have to update it.
- 1:03:59
But the sy- but, uh, the question was does a,
- 1:04:04
is the prompt hidden on their servers, or can you find it if you are so determined, and the answer seems to be yes.
- 1:04:13
Any other questions? Yes.
- 1:04:16
Is this the last one? Is this the last question?
- 1:04:19
Can be.
- 1:04:21
Um, you talk about Prompt Layer. How can people help you?
- 1:04:24
Yes. [laughs] That's a good one. I forgot about that, thank you. [laughs] [laughs]
- 1:04:29
Um, so yeah. My ... One, we're hiring. Uh, so if you're looking for coding jobs at a very fun and fast-moving team in New York, you can reach out to me on X or email [REDACTED:email_address]
- 1:04:45
We're based in New York. We are, uh, yeah, we're, we're a platform for building and testing AI products for prompt management, auditability, governance, all that fun stuff, but also logging and evals, and those screenshots I showed you came from Prompt Layer.
- 1:05:00
So if you're building an AI application and you're building it with a team, you should probably try Prompt Layer. It'll make your life easier, uh, especially the bigger your team is, the more you wanna collaborate, the more you wanna collaborate with PMs and non-technical users, and, or if you're just technical users.
- 1:05:14
It's a great tool. It'll make your life better. Highly recommend it, promptlayer.com, and it's easy to do. And that was my shill. [laughs] [laughs]
- 1:05:22
Thank you for listening. [upbeat music]