AI Engineer Summit 2025
AI Agents, Meet Test Driven Development
Read the talk
AI Agents, Meet Test-Driven Development
Reliable agents need more than capable models: representative tests, observable decisions, and bounded feedback loops turn promising examples into workflows that can improve in production.
From a talk by Anita Kirkovska
Before you start: Familiarity with LLM prompts, API calls, and basic software testing is helpful; the article introduces retrieval, model judges, and agentic workflow levels as they appear.
What makes an AI wrapper work in production?
What turns an application wrapped around an AI model into a durable product? Anita Kirkovska approaches that question through her work leading GenAI Growth and Education at Vellum. She reports that, across hundreds of companies the team worked with, those adopting test-driven development built stronger, more reliable production systems. The contrast starts with the skepticism of 2023, when AI wrappers were widely dismissed as lacking defensibility. Kirkovska reports that Cursor subsequently reached $100 million in annual recurring revenue in 12 months, describing it as the fastest-growing SaaS business. That is her framing of the market’s reversal, rather than an independently established growth comparison.
Better coding models helped. So did broader AI adoption and the suitability of programming as an early application. But another change happened around the models: developers learned to orchestrate calls, connect models to application data, and make the resulting systems operate effectively in production. Model capability and application engineering advance together. A stronger model does not remove the need to organize its work.
That surrounding engineering matters because hallucinations, overfitting, and inconsistent output structure remain practical problems. Kirkovska describes the apparent slowdown after the leap from GPT-3.5 to GPT-4, then questions whether it represents a real ceiling. Scaling training data and model size was no longer the only avenue worth considering.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Better reasoning still needs a surrounding system
Reinforcement learning and additional computation before answering opened another path. The talk uses DeepSeek-R1 to illustrate this shift, but its description of training without labeled data needs a distinction: the original DeepSeek paper describes DeepSeek-R1-Zero as using reinforcement learning without preliminary supervised fine-tuning, whereas R1 incorporates cold-start data and multiple training stages. Neither description means learning without pretraining data.
Kirkovska connects that development to OpenAI’s o1 and o3 reasoning models. OpenAI’s account of training o1 supports the general connection between reinforcement learning, chain-of-thought reasoning, and additional inference computation; it does not establish an identical training recipe for all these models. The application-level consequence is straightforward: spending more computation before answering can help a model tackle more complex reasoning tasks.
Tool use, research capabilities, and document understanding also expand what workflows can do. Kirkovska characterizes Gemini 2.0 Flash as having near-perfect OCR accuracy, without specifying a dataset or scoring conditions. That makes it a candidate to test on the documents a product actually handles, not a universal OCR guarantee. Meanwhile, Humanity’s Last Exam illustrates why harder benchmarks emerged as older ones became saturated: even capable contemporary models struggled with its difficult questions. Better models expand the available building blocks, but production success still depends on how those blocks are assembled.
The application toolkit developed in parallel. Chain-of-thought prompting encouraged more explicit reasoning; retrieval-augmented generation, or RAG, grounded answers in an organization’s data. Memory retained context across conversations, while longer context windows opened additional use cases. GraphRAG brought graph structure and hierarchy into retrieval. Reasoning models added more time spent thinking at response time, and agentic RAG introduced decisions about retrieval into the workflow itself. The engineering task is to choose the combination of techniques, models, and logic that fits a well-understood problem—not to accumulate every available technique.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Establish feasibility before committing engineering time
The development cycle starts with experimentation, moves into evaluation at scale, and continues after deployment through monitoring and improvement. In this setting, test-driven development is a continuing feedback process: demonstrate that a workflow can solve the problem, test it against a broader range of cases, then use production behavior to improve the next version. A successful prototype is the beginning of that process.
During experimentation, compare approaches that change how the work is divided:
- Few-shot prompting: Supply examples that show the model the desired task and response pattern.
- Chain-of-thought prompting: Explore whether a reasoning-oriented prompt helps on more complex tasks.
- Prompt chaining: Split a large instruction set across multiple prompts, giving each stage a narrower job.
- ReAct: Interleave reasoning and actions so feedback from the environment can inform the next step. The ReAct paper describes this interaction more specifically than a single planning stage followed by an answer.
These are alternatives to evaluate against the task, rather than a progression every workflow must follow.
Domain experts belong in this phase because they know what a useful answer looks like and which omissions matter. Kirkovska argues against making engineers solely responsible for prompt tuning: expert participation can establish feasibility before the team commits substantial implementation time. Remain model-agnostic, too. Compare models on the actual job they must perform; Vellum’s favorable experience with Gemini 2.0 Flash for OCR is her example of matching a model to a particular task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move beyond a handful of successful examples
A few good outputs establish possibility, not production coverage. Kirkovska asks what happens when a system faces hundreds, thousands, or even millions of requests per minute. Those are hypothetical demand levels, not measured throughput. Her next step is to assemble an evaluation dataset containing hundreds of examples and use it to expose the compromises that a small demonstration can hide.
Define priorities before selecting the final model and workflow:
| Priority | Decision it influences |
|---|---|
| Quality | Whether slower, more deliberative responses are acceptable |
| Cost | Whether a lighter, cheaper model is sufficient |
| Latency | How much processing the response can afford |
| Privacy | Which data-handling choices are acceptable |
No configuration optimizes every dimension simultaneously. A high-quality result may justify a longer wait; a cost-sensitive task may require a smaller model. Making those priorities explicit lets the evaluation answer a product question rather than merely produce a score.
Use ground-truth examples where possible, with subject-matter experts helping build the dataset and assess the expected behavior. Generic or synthetic benchmarks can be useful, but they do not necessarily represent the product’s inputs or acceptance criteria. Without ground truth, an LLM judge can evaluate another model’s response. Kirkovska presents this as a standard, reliable option; in practice, the judge itself needs scrutiny. Research on LLM judges documents position, verbosity, self-enhancement, and reasoning biases, so an editorial or correctness rubric should be calibrated against expert judgment.
The testing framework must accommodate changing workflows and nondeterministic responses, whether it is built internally or supplied by a service. It should retain varying outputs and allow custom metrics in Python or TypeScript. Evaluation also belongs inside the workflow: guardrails at intermediate nodes can check whether each stage produced a correct result before later stages depend on it. Run those checks while prototyping, then revisit them when production supplies real examples. A final answer alone cannot reveal which internal decision failed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploy a system you can observe and change
Once expert review and evaluation justify deployment, the work shifts toward understanding actual behavior. Record LLM calls, their inputs and outputs, and latency so failures can be traced through the system. This becomes especially important when an agent can choose different branches: debugging requires knowing both what it returned and which path it took to get there.
Operational reliability also depends on the services underneath the workflow. Use retries and fallback logic to handle failed API calls and provider outages. Kirkovska cites an OpenAI outage lasting four hours, which she places two months before the talk, as an example of why a workflow might need to switch to another model. The recording does not identify the incident or affected endpoints.
Version prompts and workflow definitions, and test changes in staging before a wider rollout. A prompt improvement in one part of a system can introduce a regression elsewhere. AI-feature deployments should also be separable from the application’s normal release schedule: prompts and workflow logic may need adjustment more frequently than the rest of the product.
Production then becomes a source of new test cases. Capture user interactions and edge cases, add them to evaluation, and check whether revised prompts fix the newly observed behavior. For repeat queries, caching avoids paying for the same expensive model call repeatedly and reduces response time by serving a stored result.
After the product has operated reliably for a period the team is comfortable with, accumulated data may support fine-tuning a custom model. Kirkovska proposes this as a later optimization for more task-specific responses, reduced dependence on external API calls, and potentially lower costs. The sequence matters: collect experience from a working system before using that experience to specialize its model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate the decisions an agent is allowed to make
An agentic workflow may call multiple APIs, use many tools, or coordinate agents executing in parallel. Testing each output is therefore insufficient: evaluation must also ask whether the system made the right decisions and followed the intended logic. Kirkovska organizes that problem around control, reasoning, and autonomy. Her provisional framework runs from L0 through L4; it is a way to discuss behavior, not a fixed definition of an agent.
L0 keeps orchestration fixed. A workflow calls an LLM, retrieves information from a vector database, may run inline evaluations, and returns a response. Reasoning can occur inside the model, but there is no external agent organizing a plan or deciding which actions to take. The distinction is between reasoning within a prompted call and control over the surrounding workflow.
L1 gives the model tool decisions. The system chooses when to call an API, invoke another tool, or retrieve additional information before answering. Memory retains context across conversations and potentially parallel activity. Evaluation must cover the chosen tool, the decision to use it, and the accuracy of the resulting answer. L1 can already be complicated: Kirkovska imagines ten available tools, with an agent choosing whether it needs the first five or the last two. A large branching graph does not, by itself, imply a higher level of autonomy.
L2 adds structured planning and refinement. A trigger can start a process that decomposes a task, retrieves information, calls a tool, evaluates whether the result is useful, and revises the next step. The system is deciding what needs to be done, rather than only whether to call a listed tool. Its reasoning loop is still finite: after completing the planned work and producing an output, it terminates.
L3 stays active. Instead of responding to one request and stopping, the system continuously monitors its environment and acts when needed. With access to email, Slack, Google Drive, or other services, it can plan a next move, execute an action, or ask a person for input. Kirkovska’s example is a marketing assistant that prepares a video or presentation for someone to use later. Persistence and proactive behavior distinguish it from a finite task runner.
L4 invents new workflows. It would create the agents, prompts, function calls, tools, or utilities needed to solve a problem in a new way. Kirkovska treats true L4 as aspirational at the time of the recording, citing overfitting and inductive biases tied to training data as constraints. The proposed change is from operating within an existing process to creating the process itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Useful production systems need not maximize autonomy
Kirkovska places Vellum customers Redfin, Drata, and Headspace in L1. Her emphasis is on orchestration: making models interact correctly with application systems and retrieving context that actually matches the user’s question. A tool-using workflow can be simple or elaborate, but its value depends on the quality of those interactions. Experiments across modalities, tools, and models need continuous testing to determine whether a change improves the system.
Looking ahead from the recording, she expects much of that year’s innovation at L2, with models such as o1, o3, and DeepSeek supporting planning and reasoning over complex tasks. That also creates room for new interfaces: users need ways to work with agents carrying out longer sequences of activity. L3 and L4 remain constrained in her account by both models and surrounding logic, even as experimentation continues.
Before moving into the demonstration, Kirkovska points to a companion resource covering agent architectures, places to test them, and lessons from researchers and practitioners. The practical example then brings those ideas together in an SEO workflow that combines tool use, research, writing, and editorial evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build research and evaluation into the writing process
The SEO agent automates keyword research, content analysis, and content creation. It decides when to use tools and includes an evaluator that acts as an editor. Kirkovska places it between L1 and L2: it combines tool selection with a structured process for developing and refining a draft.
The workflow separates responsibilities so each stage supplies something concrete to the next:
- Analyze competing content. An SEO analyst and researcher take a keyword, use Google search, and examine top-performing articles.
- Identify strengths and gaps. Preserve useful coverage from those articles while identifying missing topics or areas that deserve better treatment.
- Write from the collected context. Give the writer the research and planning material needed to produce a first draft.
- Evaluate the draft. An editor implemented as an LLM judge applies predefined rules from its prompt.
- Revise with memory. Return the editor’s feedback to the writer and retain their conversation history across the loop until a stopping criterion is met.
The intended result is a substantial first draft Kirkovska can develop further. Competitive research and an editorial loop do not establish that the article will outrank its sources; they give the writer better context and a mechanism for revision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow “chain-of-thought prompting” through the workflow
The live run uses the keyword chain-of-thought prompting, along with writing-style and target-audience inputs. The SEO analyst examines the articles Google ranks for that keyword. It identifies useful components to retain and missing opportunities for additional coverage. The researcher then performs another search to fill those gaps, rather than simply repeating the analyst’s original search.
The writer receives the accumulated material as context for its first draft. Kirkovska also describes an optional RAG connection to an internal database of articles and learnings. That would add the organization’s own material to the external research; it is an extension she proposes, not a step she establishes as part of this run.
After the first draft, the editor supplies feedback. That feedback passes through a memory component holding the writer–editor chat history and then through a node that structures the writer’s next input. The loop has two ways to stop: an excellent assessment from the editor, which Kirkovska says is rare, or the demonstration’s limit of one revision cycle. In the run, the cycle limit ends the process even though the editor still has feedback. Stopping and passing evaluation are different outcomes.
The same stopping policy can be expressed directly in Python. Here, revision_count counts completed writer–editor revision cycles after the initial draft. The keyword is unchanged from the demonstration, and stop_reason distinguishes an accepted draft from one returned because the revision budget is exhausted:
python
from dataclasses import dataclass
from typing import Literal
@dataclass(frozen=True)
class LoopDecision:
keyword: str
revision_count: int
stop_reason: Literal["excellent", "revision_limit"] | None
def decide_next_step(
keyword: str,
revision_count: int,
editor_says_excellent: bool,
) -> LoopDecision:
if revision_count < 0:
raise ValueError("revision_count must be nonnegative")
if editor_says_excellent:
reason = "excellent"
elif revision_count >= 1:
reason = "revision_limit"
else:
reason = None
return LoopDecision(keyword, revision_count, reason)
keyword = "chain-of-thought prompting"
# Initial editorial feedback: another writing pass is allowed.
initial = decide_next_step(keyword, 0, False)
assert initial.stop_reason is None
# Feedback remains after one revision, but the demo budget is spent.
revised = decide_next_step(keyword, 1, False)
assert revised.stop_reason == "revision_limit"
This isolates a deterministic part of an otherwise model-driven workflow: whether another iteration is permitted. The editorial judgment and the loop-control decision can be tested separately.
The resulting article is titled “mastering chain-of-thought prompting in AI, a comprehensive guide for developers”. Kirkovska considers changing the title, but sees the article’s coverage as a useful foundation because it reflects the material gathered during competitive research. She reports a demo latency of around 118, without specifying a unit. Kirkovska reports that the SEO workflow usually takes around 300 seconds when it runs more evaluation loops. The practical result is a draft she can build on, with the additional editorial work traded against a longer run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep workflow definitions in the codebase
The demonstration runs in Vellum Workflows, which Kirkovska describes as a way for product and engineering teams to collaborate on tested AI development. The closing announcement addresses a complementary need: developers want control, flexibility, and ownership of workflow definitions in their codebase. The Workflows SDK supplies customizable building blocks and syntax intended to make an agent’s structure and behavior legible directly in code.
Kirkovska presents synchronization between the UI and code as the bridge between those modes of work: teams can define, debug, and improve the same workflow through either representation. She announces the SDK as free and open source. The open-source workflow library is distinct from hosted UI integration and certain nodes, which require a Vellum account. The substantive endpoint is shared ownership of an inspectable workflow—domain experts can shape its behavior while developers retain its definition, control flow, and tests in code.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Python SDK for defining AI workflows in code, with examples and integration with Vellum's visual editor. Some features require a Vellum account.
Original paper introducing a difficult benchmark spanning academic subjects.
Introduces a method that interleaves language-model reasoning with actions and feedback from the environment.
Documentation for Microsoft's graph-based approach to retrieving and summarizing information from a text corpus.
Further reading
- Vellum's January 2025 SDK beta notesDocumentation
Historical release notes describing SDK-enabled workflows and pulling workflow definitions into code, editing them, and pushing them back to the UI.
Explains the distinct training approaches behind DeepSeek-R1-Zero and DeepSeek-R1.
OpenAI's original explanation of reinforcement learning and inference-time reasoning in o1.
Research on model-based evaluation, agreement with human judgments, and recurring judge biases.
Read the complete timestamped transcript
- 0:00
Hi, everybody. My name is Anita, and I'm currently leading GenAI Growth and Education here at Vellum. And over the last few years, we've worked with hundreds of companies who have successfully deployed reliable AI solutions in production.
- 0:13
From simple to more advanced agentic workflows, one thing became very clear: those companies who have adopted a test-driven development approach were able to build reliable and stronger systems for production.
- 0:26
Today, I'm excited to share how you can apply that same approach to build your own effective agentic workflow that actually works. But before we jump in, let's take a step back and truly understand how we got here in the first place.
- 0:40
I'm so excited to get started. Let's do it. So let's go back to 2023. Everyone was building AI wrappers, and most people argued that there is no defensibility strategy around them.
- 0:52
And fast-forward to today, we have Cursor AI, which is the most popular and widely used AI-powered IDE that just hit one hundred million ARR in just 12 months. This is the fastest-growing SaaS in the history of SaaS.
- 1:08
So why and how did this happen? Because models got better at coding? Sure. Because AI adoption skyrocketed? That's absolutely correct. Because coding was an obvious first target that was supposed to be disrupted by these AI models?
- 1:24
There is no doubt about that. But more importantly, we built new techniques and patterns on how we can orchestrate these models to work better, sync better with our data, and then work effectively in production.
- 1:37
We rely on these techniques because there are clear limits to model performance. Hallucinations is still a thing, overfitting is still a problem, and developers needed more structured outputs. And while model providers started to ship better tooling to solve for all of this, we didn't see another leap similar to the leap between GPT-3.5 and GPT-4.
- 1:58
These big jumps started to slow down. And for years, making models bigger and feeding them more data kept making them smarter. But then we hit a wall. No matter how much more data we added, these improvements started to slow down, and models started to reach their limits on existing tasks.
- 2:18
But is this true? Did we really hit that wall? Seems like there were some other avenues and new training methods that we still haven't explored. And so let's see what happened next.
- 2:31
So I don't really think that there is an issue here because since then, and this happened in the last two to three months, we've seen some new training methods that push the field forward.
- 2:41
For example, we got the DeepSeek-R1 model, which is the first model that was trained without using any label data. We call this method real reinforcement learning, and this means that this model was able to learn on its own.
- 2:57
Reportedly, this is what OpenAI used to train their reasoning models like o1 and o3. And all these reasoning models today, they use chain-of-thought thinking at inference time or at response time to generate their answers.
- 3:12
In turn, allowing these models to think before they give an answer to our questions, it enables them to really solve more complex reasoning problems. On top of this, we're seeing all of these model providers to provide more capabilities to their models, like use of tools, more capabilities for research, um, near perfect OCR accuracy when it comes to
- 3:36
the Gemini 2.0 Flash, and really expand the field forward. However, traditional benchmarks are so saturated, so people are starting to introduce new ones that will really capture the performance of these new reasoning models.
- 3:50
For example, the benchmark that you're currently seeing on the slide here, the Humanities Last Exam, it measures performance on truly difficult tasks. So if you check the table on the slide, you can clearly see that even the latest very smart models struggle with these challenges.
- 4:07
So yeah, models are getting better. The field is moving forward. But for an AI product that actually works in production, success isn't just about the models anymore. It's about how you build around it, and that's exactly what's been evolving in parallel to model training.
- 4:24
So we were learning how to prompt all of these models better, and we developed more advanced techniques like chain of thought. Then we thought that we should be able to ground all of these models' responses using our own data, so RAG became an important part of our workflows.
- 4:41
Then we learned that for multi-threaded conversations, memory is gonna be the most important thing that we've had. Long context from the latest models enabled new use cases. Then we started to think about hierarchy of our responses, so we started to experiment with GraphRAG.
- 4:58
And then just lately, we're thinking about using all these reasoning models that, in fact, will take a lot more time to think in real time. However, it also develops new areas and use cases that we can develop.
- 5:11
And lately, we're thinking about Agentic RAG, making our workflows even more powerful so that all this can work on its own. And the field is still evolving. But even using these techniques isn't enough.
- 5:24
You need to understand your problem deeply and take a test-driven development approach to find the right mix of techniques, models, and logic that will actually work for your use case.
- 5:36
And this actually brings me to the main first topic of this presentation: test-driven development for building reliable AI products. Because the best AI teams that I've seen follow this structured approach.
- 5:52
They start to experiment, then they evaluate at scale, then finally, when they deploy in production, they never stop working on their workflow. They capture all of those responses to then continuously monitor, observe, and improve their product for their customers.
- 6:08
Let's look at what you can do at every stage of this process. Before you build anything production grade, you need to experiment a lot. You need to prove whether these AI models can actually solve for your use case.
- 6:22
So you should try different prompting techniques, for example, few-shot or chain of thought. Some of these will work great for simple tasks, and other will help with a bit more complex reasoning.
- 6:32
You should test various techniques. Prompt chaining is usually very, uh, well-received because it's gonna work better if you split your instructions in multiple prompts, or you can a- adopt a more agentic workflows like ReAct that will have a stage to plan and then reason and refine before it actually gives you an answer.
- 6:51
What is really an important part in this stage is that you need to involve your domain experts, um, because engineers shouldn't be the ones who are tweaking prompts. B- uh, and bringing all these experts will actually save a lot of your engineering time because once you do this phase right, then you will actually have a proof that
- 7:09
this works and that engineering time needs to be involved. At this stage, you should also stay model agnostic. Uh, you should incorporate and test different models, and especially when it comes to your use case, you need to think about which models can do the job better.
- 7:23
So in such case, you can, um, maybe use some, uh, different, uh, models like Gemini 2.0 Flash, which is actually really well at OCR, uh, and something that we've seen work really well lately.
- 7:35
So let's say that at this stage you know that, um, these AI models can actually work. You have a few examples that these models have, like, really good performance on.
- 7:44
But how can you test whether this will actually work in production when you will potentially have hundreds, if not thousands or millions of requests per minute? And so this is where evaluation comes in.
- 7:56
In this stage, you actually create a data set of hundreds of examples that you're gonna test your models and workflows against. And so at this stage you need to, uh, try to balance quality and cost and latency and privacy, and you're definitely gonna make a lot of trade-offs because no AI system is gonna get all of this
- 8:14
perfectly. But for example, if you need high quality, maybe you can sacrifice speed. If cost is critical, you might need some lighter and cheaper model. And this is the stage where you, where you need to define your priorities because it's always better if you define your priorities earlier in the process.
- 8:32
You should use ground truth data where possible. If you want to evaluate all these workflows, having your subject matter experts design these databases and test these models and workflows against is gonna be very, very useful.
- 8:46
Synthetic benchmarks help, however, they will not really evaluate these models for your own use case. So it's usually very, very powerful if you can use your ground truth data.
- 8:57
But don't worry, even if you do not have ground truth data, you can use an LLM to evaluate another model's response. This is actually a very standard and reliable way when it comes to evaluating your models.
- 9:12
Very importantly at this stage, you should make sure that you're using a flexible testing framework. No matter if you're building this in-house or if you're using any external service, your AI isn't static, so your workflow should also be dynamic.
- 9:27
It should be able to capture all of these different non-deterministic responses. You need to be able to define custom metrics. You need to be able to write those metrics, metrics using Python or TypeScript.
- 9:39
So you shouldn't be looking at a very strict framework. Customizability is a very big thing here. And then finally, you should run evaluations at every stage. You should have guardrails that will check internal nodes and whether these models are actually producing responses at every step in your, uh, workflow, actually producing responses that are correct at every step
- 10:02
in your workflow. And then you should also test while you're prototyping, but then you should also utilize this evaluation phase to come back once you have some real, uh, data.
- 10:13
But how are you gonna get some real data? So let's say that you evaluate your workflows extensively with your subject matter experts, with your data that they've created, and let's say that you're now satisfied with the product that you have, so you're ready to deploy it in production.
- 10:28
So once that happens, what do you need to do? Is your job done here? When it comes to AI development, you need to monitor more things than deterministic outputs.
- 10:39
You need to log all LLM calls. You need to track all of those inputs and outputs and the latency because AI models, they really-- they're really, really unpredictable. So you need to be able to debug issues and understand how your AI behaves at every step of the way.
- 10:56
And this is becoming extremely more important with agentic workflows because agentic workflows are more complex workflows that can take different paths in your workflow and make decisions on their own.
- 11:09
You should also handle API reliability. You need to maintain, uh, stability in your API calls. You need to have retries. You need to have fallback logic to prevent outages.
- 11:19
For example, two months ago, OpenAI had four hours of downtime. So if you had a fallback logic in your productionized solution, then your, um, AI will know to go back to another model and use another model instead.
- 11:34
You should definitely have version control and staging, and you should always deploy in control environments before you roll out to the wider, uh, public. Uh, because with A- when it comes to AI, you need to be care- careful that once you update a prompt, you're not introducing a regression to another prompt or part of your workflow.
- 11:54
So you need to ensure that all these new updates, they won't break whatever you have in production. And the most important part here is that make sure to decouple your deployments from your scheduled app deployment schedule because the chances are That, um, you will need to update your AI features more frequently than you will need to update
- 12:16
your app as a whole. So make sure to do that. And so let's say that now you have deployed, you're starting to capture all of your responses from your users and create a feedback loop to identify edge cases that you capture in production to then continuously improve and make your workflow better.
- 12:37
You can capture all of these, then run evaluations again and test whether, uh, new prompts that you develop will solve for these new cases. You should also think about building a caching layer because if your system is handling some repeat queries, caching can drastically reduce costs and improve latency.
- 12:55
So for example, instead of calling an expensive LLM for the same request multiple times, you can store and serve frequent responses instantly. And this is something that is a standard these days when it comes to building with AI.
- 13:08
And finally, let's say that your product has been running reliably in production for, uh, a longer period of time, a time that you feel comfortable to then go back to that data and use it to fine-tune a custom model that will, um, basically, uh, create better responses for your specific use case, uh, can reduce reliance on API
- 13:28
calls, and in fact can work with lower costs. And so this process is becoming even more important than ever when it comes with agentic workflows because these workflows are gonna use a wide range of tools.
- 13:43
They will, um, call different APIs. They will have multi-agent structures that will execute a lot of things in parallel. So when it comes to evaluation with agentic workflows and with this test-driven approach, it's not just about measuring performance at every step in your workflow because you also need to assess the behavior of these agents to...
- 14:06
so that you can make sure that they're making the right decisions and following the intended logic. And this year, more than ever, everyone is talking about agentic workflows. But what does that actually mean?
- 14:19
Uh, I would love to talk more about how you can build all these agentic workflows, but I'm not here to give you the perfect definition of what an AI agent is, and instead, I'm gonna try to define different agentic behaviors and some different levels on how, um, they can be built.
- 14:36
So if you think about it, every AI workflow has some level of an agentic behavior in it. It's just a question of how much control, reasoning, and autonomy it has.
- 14:48
So we've looked at the past, the present, and where we're headed, and from that, we put together this framework where we define four or five different levels of agentic behavior.
- 15:00
I'll go into more details on each level, but keep in mind that this is not a final framework. It's not set in stone. As models evolve, this can expand, uh, things can blur, and, um, a lot of things can shift.
- 15:13
But for now, this will give us a way to define where we are today and what we expect to see next. At this stage, you have an LLM call, you retrieve some data from your vector database, and then you might have some inline evals, and finally you're gonna, uh, get some response from this workflow.
- 15:30
So you can notice that in this workflow, there's no reasoning, planning, or decision-making beyond what's baked into the prompt and the model behavior. So the model is doing all the reasoning here within the prompt itself, and so there is no external agentic organizing, uh, the decisions or planning some actions.
- 15:48
However, there is some reasoning and some agentic behavior at the model's level. And so if we move from L0 to L1, we can see that in this stage, our workflows can now use a lot of tools.
- 16:01
And so this AI system is no longer just calling APIs, it kn- now knows when to call them and when to make those actions. And so this is where we start to see more agentic behavior because the model can decide whether, uh, it will call a specific tool or whether it will call our, uh, vector database to
- 16:20
retrieve more data before it actually, uh, generates an output. Memory here starts to play a key role because we're gonna have multi-threaded, uh, conversations and then, uh, all of this will potentially happen in parallel, so we need to capture all context throughout the whole workflow.
- 16:37
Evaluation is also needed at every sta- uh, step of the way here because we need to ensure that these models are making the right decisions, using the right tools, and returning, uh, accurate responses.
- 16:49
But these workflows can be as simple as on the slide right here, or even more complicated where you're gonna have more different branching happen, uh, happens at every stage in this workflow where you can have 10 different tools and the agent needs to reason whether it's gonna call the first five or the, or the last two.
- 17:08
And so this is, uh, where again, we see a lot more agentic behavior, but L2 is where we actually see that, um, these workflows now move from simple tool use, which is not, in many cases, it's not a simple tool use, like the previous workflows can be very complex, but now we see some structured reasoning.
- 17:28
This workflow will notice triggers, it can plan actions, and it can execute tasks in a structured sequence. So this means that it can break down a task into multiple steps, it can retrieve some information, it can decide to call another tool, it can evaluate its usefulness if it thinks that it needs to be refined at that stage.
- 17:48
And once it does this in a continuous loop, it can generate the final output. But, um, you can notice that agentic behavior here starts to look more intentional Because the system isn't just calling the tools that are, uh, listed, uh, for their use, it's also actively deciding what needs to be done and spending more time to think
- 18:08
what needs to be done instead of just deciding whether a tool should be called. And so at this stage, um, o- one part is that the process is still finite, so once this workflow completes the steps, um, as it plans to complete them, it will terminate rather than it will run continuously.
- 18:27
But it's a leap forward, um, from just calling, uh, tools. And so, um, L3 however, is where we see more autonomy, where we see more, uh, decision-making that are not, um, defined by us as the creators of this workflow.
- 18:44
So the L4 system can proactively take actions without waiting for direct input. So instead of responsing, responding to a single request and then terminating, this one will stay alive and will continuously monitor its environment and it will react as needed.
- 19:01
So for example, um, this means that it can, uh, look at your email, Slack, Google Drive, or any other tool, uh, external services actually, that you can give access to, and it can plan its next moves, whether it will execute actions in real time or asks, uh, the human for more input.
- 19:18
And so this is where, uh, our AI workflows become less of a tool and more of an independent system that we can use to truly make our work easier.
- 19:28
So for example, this one can be like a marketer that will prepare this video or a presentation that you can just take and use whenever you want. However, the final stage is where we're gonna have a fully creative workflow.
- 19:41
And so at L4, the AI moves betw- uh, beyond automation and reasoning, and it becomes an inventor. So instead of just executing predefined tasks or just like reasoning within some bounds, um, it can create its own new workflows.
- 19:58
So it can create its own utilities, whether it's agents, prompts, function calls, tools that, uh, it needs to be designed. Uh, it will pro- it will solve problems in novel way.
- 20:09
So while true L4 right now is definitely out of reach because there are some constraints with models like overfitting, because models, they really love their training data, and there is some issues with, uh, inductive bias where models will make assumptions, again, based on their training data.
- 20:26
This makes to be like a very hard task, uh, today, but that's the goal. AI that doesn't just follow instructions, but will invent, it will improve, and it will solve problems in ways we didn't even think of before.
- 20:40
So I would say that L1 is where we're seeing a lot of production-grade solutions. So at Vellum, we've worked with companies like Redfin, Drata, and Headspace, all of which have deployed production-grade AI solutions that fall within the L1 segment.
- 20:54
And again, like just using tools, it can be very simple or it can be very complex workflow. Uh, the focus is though on orchestrations. How do we train our models to interact with our system better?
- 21:07
How do we make our models to work with our data better? How do we make sure that whatever we retrieve from our vector databases is the right and correct context for the, uh, question that the user is asking?
- 21:20
And so like we're experimenting with different modalities and all of those techniques that we mentioned before, and test-driven development truly makes its case here because like you need to have different tools and models, and you need to be able to continuously improve on them to build not only a more efficient system, but a system that will work
- 21:37
continuously better and better. Um, however, L2 is where I think we're gonna see most innovation, uh, happen this year, and this is where we're gonna have a lot of AI agents that are being developed to plan and reason using models like o1 or o3 or DeepSeek.
- 21:54
Uh, we might see a bunch of different use cases. We might see a lot of innovations when it comes to the UI and the UX part of the system where we will definitely create some new experiement-- uh, experiences for our users and, um, essentially this will be, uh, a way for us to make true reasoners that will,
- 22:14
uh, handle complex tasks. So you're gonna have bunch of these agents just working for you doing, uh, different things. However, L3 and L4, they're still both limited by the models today, as well as the surrounding logic.
- 22:28
However, that doesn't mean that, um, there's a lot of innovation happening within those two as well. So if you want to learn more about how to build your own, uh, AI agent, I've included everything that I've shared in this presentation and more.
- 22:41
For example, uh, architectures that you can build, what are the stages that you can test, and, uh, similar things like that. We also feature top researchers and professionals who have shared all of their learnings on how to build these for production.
- 22:54
So feel free to scan this QR code on the screen to download this resource. So now I think it's time to get more practical. I want to show you how I built my own SEO agent.
- 23:06
This specific agent automates my whole SEO process, from keyword research to content analysis, and finally for content creation. It decides whether to use tools and has an embedded evaluator that works on an editor to tell the agent if it's doing a good job.
- 23:20
Let's see a quick sketch of how this agent works. So in a minute, I'm gonna show you a real demo on how this agent actually works. However, I wanted to give you a high-level overview on what are the steps that this workflow will take.
- 23:34
And so when you look at the sketch on the screen right now, you're gonna notice that this workflow lies between L1 and L2 type of ag- agentic workflow. You have the SEO analyst and the researcher who will take a keyword and it will call Google search, and it will analyze the top performing articles for that keyword.
- 23:52
One, it will identify some of the good parts, uh, within these articles that we also need to amplify in our own article, but it will also identify some missing segments or areas of improvement that we should definitely write about to make sure that our Article is actually performing better than the ones that we're competing against.
- 24:12
And then after the research and planning is done, the writer has everything it needs to start writing the first draft. What then the first draft is passed to the editor, which is an LLM-based judge that will evaluate whether the first draft is good enough based on predefined rules that we've set in its prompt.
- 24:30
Then that feedback is passed back to the writer, and this will loop, uh, continuously until some, uh, criteria is met. Uh, within this loop, we also have a memory component that will capture all previous conversations between the writer and the editor.
- 24:47
And finally, we're gonna get a final article that's actually a very useful, um, piece of content that it's not AI-generated and not useful, but truly using all of this context in a smart way, enabling me to have a pretty impressive first draft to work with.
- 25:04
So now let's see the demo. For the sake of time, I'm gonna start running this workflow as I explain what this a- agent does at every step in the workflow.
- 25:13
So we ran this workflow with the keyword chain-of-thought prompting. And so the SEO analyst currently is taking that keyword, is taking some other parameters like my writing style, like the audience that we're trying to cater to, and it analyzes the top articles that Google is ranking for that specific keywords.
- 25:31
It tries to identify some good components from those articles that we need to reinforce in our article, but it also identifies some missing opportunities where the researcher is gonna utilize those to then make another search and capture more data to make our article be better than the articles that we just analyzed.
- 25:51
So now that the SEO analyst, uh, is done with its job, the researcher tries to capture more information about the things that, um, were previously identified as missing pieces to the puzzle.
- 26:03
And then the writer will take a lot of this information in its input, and it will try to create a great first draft using that data as context. So the content here that will be generated by the writer, it's not gonna be like a slop type of article.
- 26:18
It's not gonna be something that it's really not useful. It's gonna actually use all the context that we're sending from different articles that we just analyzed. You can also connect your RAG here that w- it will look into your database of articles and learnings, and it can really create something that's extremely useful.
- 26:36
Now, the editor says, "Okay, this is a good enough article, but here's some feedback." And so it passes the feedback through the memory component here, which is a chat history between these two, and then, um, this node that, uh, basically structures that input.
- 26:50
For the sake of this demo, the conditional here for the loop is that this loop will break if the evaluator actually tells us that this is an excellent post, which actually rarely happens.
- 27:02
So, um, we also said that if the loop runs for at least one time, this loop will break. And so it already ran for one time. We got still more feedback from the editor.
- 27:14
But in this case, for this demo, let's look at the output that we got. So mastering chain-of-thought prompting in AI, a comprehensive guide for developers. I think it's pretty okay, pretty nice.
- 27:25
I might change the title, but I know that the components in this article are the actual components that other articles are writing about. And so, uh, this was great.
- 27:35
The latency was around 118. This usually takes around 300 seconds to run when we have more evaluation loops, but it's pretty great. It gives me some foundations on how I can continue to build on this content, and it saves me a lot of my time.
- 27:49
So the product that I just used is called Vellum Workflows, and it was designed to bridge the gap between the product and engineering teams so they can speed up AI development while still following this test-driven approach that we talked so much about in this presentation.
- 28:04
However, one thing became clear. Developers want more code, developers want more control and flexibility, and they want to own their definitions in their code base. So today, I'm excited to introduce our Workflows SDK.
- 28:16
It provides all the building blocks you need. It's infinitely customizable, and it has a self-documenting syntax where you can actually spot how this agent is working right in your code.
- 28:27
It's also expressive enough so that you can understand what's happening at every stage in your code. The best part is that the UI and the code stay in sync, so whether you're defining, debugging, or improving your workflows, everyone on your team can stay aligned.
- 28:42
I hope that you like it. It's open source and free, and you can check it out on GitHub. Feel free to run, uh, to scan this QR code, uh, to check out the repo.
- 28:52
And that's a wrap. Thank you so much for listening, and I hope that today you learned something new. If you want to talk more about AI, feel free to scan this QR code on the screen to connect on LinkedIn, or if you have any questions, feel free to, um, send me a text message on my email or
- 29:07
on Twitter. I'm gonna follow up for sure.