# AI Agents, Meet Test Driven Development

Anita Kirkovska

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/U3MVU6JpocU-ai-agents-meet-test-driven-development) · [Video](https://www.youtube.com/watch?v=U3MVU6JpocU)

## AI Agents, Meet Test-Driven Development

Reliable agents need more than capable models: representative tests, observable decisions, and bounded feedback loops turn promising examples into workflows that can improve in production.

Familiarity with LLM prompts, API calls, and basic software testing is helpful; the article introduces retrieval, model judges, and agentic workflow levels as they appear.

### What makes an AI wrapper work in production?

[0:00](https://www.youtube.com/watch?v=U3MVU6JpocU&t=0s)

What turns an application wrapped around an AI model into a durable product? Anita Kirkovska approaches that question through her work leading GenAI Growth and Education at Vellum. She reports that, across hundreds of companies the team worked with, those adopting test-driven development built stronger, more reliable production systems. The contrast starts with the skepticism of 2023, when AI wrappers were widely dismissed as lacking defensibility. Kirkovska reports that Cursor subsequently reached $100 million in annual recurring revenue in 12 months, describing it as the fastest-growing SaaS business. That is her framing of the market’s reversal, rather than an independently established growth comparison.

![Opening slide with Cursor annual recurring revenue and a comparison of years to reach $100 million ARR.](https://ai.engineer/api/talk-essay-media/cfd309aaa2e463aa17e0ceb0183575c411d787fbaed9784774516c5042b5cf7d)

Kirkovska opens with Cursor’s reported revenue growth before turning to what makes an AI application useful.

Better coding models helped. So did broader AI adoption and the suitability of programming as an early application. But another change happened around the models: developers learned to orchestrate calls, connect models to application data, and make the resulting systems operate effectively in production. **Model capability and application engineering advance together.** A stronger model does not remove the need to organize its work.

That surrounding engineering matters because hallucinations, overfitting, and inconsistent output structure remain practical problems. Kirkovska describes the apparent slowdown after the leap from **GPT-3.5** to **GPT-4**, then questions whether it represents a real ceiling. Scaling training data and model size was no longer the only avenue worth considering.

### Better reasoning still needs a surrounding system

[2:31](https://www.youtube.com/watch?v=U3MVU6JpocU&t=151s)

Reinforcement learning and additional computation before answering opened another path. The talk uses **DeepSeek-R1** to illustrate this shift, but its description of training without labeled data needs a distinction: the [original DeepSeek paper](https://arxiv.org/abs/2501.12948v1) describes **DeepSeek-R1-Zero** as using reinforcement learning without preliminary supervised fine-tuning, whereas R1 incorporates cold-start data and multiple training stages. Neither description means learning without pretraining data.

Kirkovska connects that development to OpenAI’s **o1** and **o3** reasoning models. OpenAI’s [account of training o1](https://openai.com/index/learning-to-reason-with-llms/) supports the general connection between reinforcement learning, chain-of-thought reasoning, and additional inference computation; it does not establish an identical training recipe for all these models. The application-level consequence is straightforward: spending more computation before answering can help a model tackle more complex reasoning tasks.

Tool use, research capabilities, and document understanding also expand what workflows can do. Kirkovska characterizes **Gemini 2.0 Flash** as having near-perfect OCR accuracy, without specifying a dataset or scoring conditions. That makes it a candidate to test on the documents a product actually handles, not a universal OCR guarantee. Meanwhile, [Humanity’s Last Exam](https://arxiv.org/abs/2501.14249v1) illustrates why harder benchmarks emerged as older ones became saturated: even capable contemporary models struggled with its difficult questions. Better models expand the available building blocks, but production success still depends on how those blocks are assembled.

The application toolkit developed in parallel. Chain-of-thought prompting encouraged more explicit reasoning; **retrieval-augmented generation**, or RAG, grounded answers in an organization’s data. Memory retained context across conversations, while longer context windows opened additional use cases. [GraphRAG](https://microsoft.github.io/graphrag/) brought graph structure and hierarchy into retrieval. Reasoning models added more time spent thinking at response time, and agentic RAG introduced decisions about retrieval into the workflow itself. The engineering task is to choose the combination of techniques, models, and logic that fits a well-understood problem—not to accumulate every available technique.

### Establish feasibility before committing engineering time

[5:36](https://www.youtube.com/watch?v=U3MVU6JpocU&t=336s)

The development cycle starts with experimentation, moves into evaluation at scale, and continues after deployment through monitoring and improvement. In this setting, **test-driven development is a continuing feedback process**: demonstrate that a workflow can solve the problem, test it against a broader range of cases, then use production behavior to improve the next version. A successful prototype is the beginning of that process.

During experimentation, compare approaches that change how the work is divided:

- **Few-shot prompting:** Supply examples that show the model the desired task and response pattern.
- **Chain-of-thought prompting:** Explore whether a reasoning-oriented prompt helps on more complex tasks.
- **Prompt chaining:** Split a large instruction set across multiple prompts, giving each stage a narrower job.
- **ReAct:** Interleave reasoning and actions so feedback from the environment can inform the next step. The [ReAct paper](https://arxiv.org/abs/2210.03629) describes this interaction more specifically than a single planning stage followed by an answer.

These are alternatives to evaluate against the task, rather than a progression every workflow must follow.

Domain experts belong in this phase because they know what a useful answer looks like and which omissions matter. Kirkovska argues against making engineers solely responsible for prompt tuning: expert participation can establish feasibility before the team commits substantial implementation time. Remain model-agnostic, too. Compare models on the actual job they must perform; Vellum’s favorable experience with Gemini 2.0 Flash for OCR is her example of matching a model to a particular task.

### Move beyond a handful of successful examples

[7:35](https://www.youtube.com/watch?v=U3MVU6JpocU&t=455s)

A few good outputs establish possibility, not production coverage. Kirkovska asks what happens when a system faces hundreds, thousands, or even millions of requests per minute. Those are hypothetical demand levels, not measured throughput. Her next step is to assemble an evaluation dataset containing hundreds of examples and use it to expose the compromises that a small demonstration can hide.

![Evaluation slide listing constraints, ground-truth data, custom metrics, a flexible test bank and offline, inline and online evaluations.](https://ai.engineer/api/talk-essay-media/94b12705c7631163edaf376217254b63ab8aa94f046e4b16cfd1cb747f8c4613)

The evaluation checklist connects quality, cost, latency and privacy to ground truth, custom metrics and offline, inline and online tests.

Define priorities before selecting the final model and workflow:

| Priority | Decision it influences |
| --- | --- |
| Quality | Whether slower, more deliberative responses are acceptable |
| Cost | Whether a lighter, cheaper model is sufficient |
| Latency | How much processing the response can afford |
| Privacy | Which data-handling choices are acceptable |

No configuration optimizes every dimension simultaneously. A high-quality result may justify a longer wait; a cost-sensitive task may require a smaller model. Making those priorities explicit lets the evaluation answer a product question rather than merely produce a score.

Use ground-truth examples where possible, with subject-matter experts helping build the dataset and assess the expected behavior. Generic or synthetic benchmarks can be useful, but they do not necessarily represent the product’s inputs or acceptance criteria. Without ground truth, an **LLM judge** can evaluate another model’s response. Kirkovska presents this as a standard, reliable option; in practice, the judge itself needs scrutiny. [Research on LLM judges](https://arxiv.org/abs/2306.05685) documents position, verbosity, self-enhancement, and reasoning biases, so an editorial or correctness rubric should be calibrated against expert judgment.

The testing framework must accommodate changing workflows and nondeterministic responses, whether it is built internally or supplied by a service. It should retain varying outputs and allow custom metrics in Python or TypeScript. Evaluation also belongs inside the workflow: guardrails at intermediate nodes can check whether each stage produced a correct result before later stages depend on it. Run those checks while prototyping, then revisit them when production supplies real examples. **A final answer alone cannot reveal which internal decision failed.**

### Deploy a system you can observe and change

[10:13](https://www.youtube.com/watch?v=U3MVU6JpocU&t=613s)

Once expert review and evaluation justify deployment, the work shifts toward understanding actual behavior. Record LLM calls, their inputs and outputs, and latency so failures can be traced through the system. This becomes especially important when an agent can choose different branches: debugging requires knowing both what it returned and which path it took to get there.

Operational reliability also depends on the services underneath the workflow. Use retries and fallback logic to handle failed API calls and provider outages. Kirkovska cites an OpenAI outage lasting four hours, which she places two months before the talk, as an example of why a workflow might need to switch to another model. The recording does not identify the incident or affected endpoints.

Version prompts and workflow definitions, and test changes in staging before a wider rollout. A prompt improvement in one part of a system can introduce a regression elsewhere. AI-feature deployments should also be separable from the application’s normal release schedule: prompts and workflow logic may need adjustment more frequently than the rest of the product.

Production then becomes a source of new test cases. Capture user interactions and edge cases, add them to evaluation, and check whether revised prompts fix the newly observed behavior. For repeat queries, caching avoids paying for the same expensive model call repeatedly and reduces response time by serving a stored result.

After the product has operated reliably for a period the team is comfortable with, accumulated data may support fine-tuning a custom model. Kirkovska proposes this as a later optimization for more task-specific responses, reduced dependence on external API calls, and potentially lower costs. The sequence matters: collect experience from a working system before using that experience to specialize its model.

### Evaluate the decisions an agent is allowed to make

[13:43](https://www.youtube.com/watch?v=U3MVU6JpocU&t=823s)

An agentic workflow may call multiple APIs, use many tools, or coordinate agents executing in parallel. Testing each output is therefore insufficient: evaluation must also ask whether the system made the right decisions and followed the intended logic. Kirkovska organizes that problem around control, reasoning, and autonomy. Her provisional framework runs from **L0 through L4**; it is a way to discuss behavior, not a fixed definition of an agent.

**L0 keeps orchestration fixed.** A workflow calls an LLM, retrieves information from a vector database, may run inline evaluations, and returns a response. Reasoning can occur inside the model, but there is no external agent organizing a plan or deciding which actions to take. The distinction is between reasoning within a prompted call and control over the surrounding workflow.

**L1 gives the model tool decisions.** The system chooses when to call an API, invoke another tool, or retrieve additional information before answering. Memory retains context across conversations and potentially parallel activity. Evaluation must cover the chosen tool, the decision to use it, and the accuracy of the resulting answer. L1 can already be complicated: Kirkovska imagines ten available tools, with an agent choosing whether it needs the first five or the last two. A large branching graph does not, by itself, imply a higher level of autonomy.

**L2 adds structured planning and refinement.** A trigger can start a process that decomposes a task, retrieves information, calls a tool, evaluates whether the result is useful, and revises the next step. The system is deciding what needs to be done, rather than only whether to call a listed tool. Its reasoning loop is still finite: after completing the planned work and producing an output, it terminates.

**L3 stays active.** Instead of responding to one request and stopping, the system continuously monitors its environment and acts when needed. With access to email, Slack, Google Drive, or other services, it can plan a next move, execute an action, or ask a person for input. Kirkovska’s example is a marketing assistant that prepares a video or presentation for someone to use later. Persistence and proactive behavior distinguish it from a finite task runner.

**L4 invents new workflows.** It would create the agents, prompts, function calls, tools, or utilities needed to solve a problem in a new way. Kirkovska treats true L4 as aspirational at the time of the recording, citing overfitting and inductive biases tied to training data as constraints. The proposed change is from operating within an existing process to creating the process itself.

### Useful production systems need not maximize autonomy

[20:40](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1240s)

Kirkovska places Vellum customers Redfin, Drata, and Headspace in L1. Her emphasis is on orchestration: making models interact correctly with application systems and retrieving context that actually matches the user’s question. A tool-using workflow can be simple or elaborate, but its value depends on the quality of those interactions. Experiments across modalities, tools, and models need continuous testing to determine whether a change improves the system.

Looking ahead from the recording, she expects much of that year’s innovation at L2, with models such as o1, o3, and DeepSeek supporting planning and reasoning over complex tasks. That also creates room for new interfaces: users need ways to work with agents carrying out longer sequences of activity. L3 and L4 remain constrained in her account by both models and surrounding logic, even as experimentation continues.

Before moving into the demonstration, Kirkovska points to a companion resource covering agent architectures, places to test them, and lessons from researchers and practitioners. The practical example then brings those ideas together in an SEO workflow that combines tool use, research, writing, and editorial evaluation.

### Build research and evaluation into the writing process

[23:06](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1386s)

The SEO agent automates keyword research, content analysis, and content creation. It decides when to use tools and includes an evaluator that acts as an editor. Kirkovska places it between L1 and L2: it combines tool selection with a structured process for developing and refining a draft.

The workflow separates responsibilities so each stage supplies something concrete to the next:

1. **Analyze competing content.** An SEO analyst and researcher take a keyword, use Google search, and examine top-performing articles.
2. **Identify strengths and gaps.** Preserve useful coverage from those articles while identifying missing topics or areas that deserve better treatment.
3. **Write from the collected context.** Give the writer the research and planning material needed to produce a first draft.
4. **Evaluate the draft.** An editor implemented as an LLM judge applies predefined rules from its prompt.
5. **Revise with memory.** Return the editor’s feedback to the writer and retain their conversation history across the loop until a stopping criterion is met.

The intended result is a substantial first draft Kirkovska can develop further. Competitive research and an editorial loop do not establish that the article will outrank its sources; they give the writer better context and a mechanism for revision.

### Follow “chain-of-thought prompting” through the workflow

[25:04](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1504s)

The live run uses the keyword `chain-of-thought prompting`, along with writing-style and target-audience inputs. The SEO analyst examines the articles Google ranks for that keyword. It identifies useful components to retain and missing opportunities for additional coverage. The researcher then performs another search to fill those gaps, rather than simply repeating the analyst’s original search.

The writer receives the accumulated material as context for its first draft. Kirkovska also describes an optional RAG connection to an internal database of articles and learnings. That would add the organization’s own material to the external research; it is an extension she proposes, not a step she establishes as part of this run.

After the first draft, the editor supplies feedback. That feedback passes through a memory component holding the writer–editor chat history and then through a node that structures the writer’s next input. The loop has two ways to stop: an excellent assessment from the editor, which Kirkovska says is rare, or the demonstration’s limit of one revision cycle. In the run, the cycle limit ends the process even though the editor still has feedback. **Stopping and passing evaluation are different outcomes.**

The same stopping policy can be expressed directly in Python. Here, `revision_count` counts completed writer–editor revision cycles after the initial draft. The keyword is unchanged from the demonstration, and `stop_reason` distinguishes an accepted draft from one returned because the revision budget is exhausted:

```python
from dataclasses import dataclass
from typing import Literal


@dataclass(frozen=True)
class LoopDecision:
    keyword: str
    revision_count: int
    stop_reason: Literal["excellent", "revision_limit"] | None


def decide_next_step(
    keyword: str,
    revision_count: int,
    editor_says_excellent: bool,
) -> LoopDecision:
    if revision_count < 0:
        raise ValueError("revision_count must be nonnegative")

    if editor_says_excellent:
        reason = "excellent"
    elif revision_count >= 1:
        reason = "revision_limit"
    else:
        reason = None

    return LoopDecision(keyword, revision_count, reason)


keyword = "chain-of-thought prompting"

# Initial editorial feedback: another writing pass is allowed.
initial = decide_next_step(keyword, 0, False)
assert initial.stop_reason is None

# Feedback remains after one revision, but the demo budget is spent.
revised = decide_next_step(keyword, 1, False)
assert revised.stop_reason == "revision_limit"
```

This isolates a deterministic part of an otherwise model-driven workflow: whether another iteration is permitted. The editorial judgment and the loop-control decision can be tested separately.

The resulting article is titled “mastering chain-of-thought prompting in AI, a comprehensive guide for developers”. Kirkovska considers changing the title, but sees the article’s coverage as a useful foundation because it reflects the material gathered during competitive research. She reports a demo latency of around 118, without specifying a unit. Kirkovska reports that the SEO workflow usually takes around 300 seconds when it runs more evaluation loops. The practical result is a draft she can build on, with the additional editorial work traded against a longer run.

### Keep workflow definitions in the codebase

[27:49](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1669s)

The demonstration runs in Vellum Workflows, which Kirkovska describes as a way for product and engineering teams to collaborate on tested AI development. The closing announcement addresses a complementary need: developers want control, flexibility, and ownership of workflow definitions in their codebase. The [Workflows SDK](https://github.com/vellum-ai/vellum-python-sdks) supplies customizable building blocks and syntax intended to make an agent’s structure and behavior legible directly in code.

Kirkovska presents synchronization between the UI and code as the bridge between those modes of work: teams can define, debug, and improve the same workflow through either representation. She announces the SDK as free and open source. The open-source workflow library is distinct from hosted UI integration and certain nodes, which require a Vellum account. The substantive endpoint is shared ownership of an inspectable workflow—domain experts can shape its behavior while developers retain its definition, control flow, and tests in code.

## Resources

- [Vellum Workflows SDK](https://github.com/vellum-ai/vellum-python-sdks) — Python SDK for defining AI workflows in code, with examples and integration with Vellum's visual editor. Some features require a Vellum account.
- [Vellum's January 2025 SDK beta notes](https://docs.vellum.ai/changelog/2025/2025-01) — Historical release notes describing SDK-enabled workflows and pulling workflow definitions into code, editing them, and pushing them back to the UI.
- [DeepSeek-R1 original paper](https://arxiv.org/abs/2501.12948v1) — Explains the distinct training approaches behind DeepSeek-R1-Zero and DeepSeek-R1.
- [OpenAI on training o1 to reason](https://openai.com/index/learning-to-reason-with-llms/) — OpenAI's original explanation of reinforcement learning and inference-time reasoning in o1.
- [Humanity's Last Exam](https://arxiv.org/abs/2501.14249v1) — Original paper introducing a difficult benchmark spanning academic subjects.
- [Judging LLM-as-a-Judge](https://arxiv.org/abs/2306.05685) — Research on model-based evaluation, agreement with human judgments, and recurring judge biases.
- [ReAct: Reasoning and Acting](https://arxiv.org/abs/2210.03629) — Introduces a method that interleaves language-model reasoning with actions and feedback from the environment.
- [GraphRAG](https://microsoft.github.io/graphrag/) — Documentation for Microsoft's graph-based approach to retrieving and summarizing information from a text corpus.

## Slides

### Cursor revenue growth

The slide charts Cursor’s rise from $1M to $100M ARR in 12 months alongside other companies. The speaker credits better models and adoption, but emphasizes new techniques for orchestrating models in production.

[1:03](https://www.youtube.com/watch?v=U3MVU6JpocU&t=63s) [Slide claims Cursor grew from $1 million to $100 million in annual recurring revenue in 12 months, with comparison charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae8bf875b2f20f68eebbdc1099665f6180151d596ce57576d61a4e9991adf88a.webp)

The fastest growing SaaS in the history of SaaS

1-100 Million in 12 months

SACRA
CURSOR
Annualized recurring revenue (ARR)
Legend: ARR ($M); Growth
ARR axis ticks: $0M, $25M, $50M, $75M, $100M
Growth axis ticks: 0%, 5000%, 10000%, 15000%, 20000%
2023: $1M
2024: $100M; 9900%

SACRA
CURSOR
Years from $1M to $100M ARR
Companies: CURSOR, WIZ, deel., together.ai, CoreWeave, OpenAI, docusign
Y-axis ticks: $0M, $25M, $50M, $75M, $100M
X-axis: Years
X-axis ticks: 2, 4, 6, 8, 10

### Neural scaling laws

Model quality rises across labs over time, while the speaker questions whether gains from larger models and more data are slowing after the GPT-3.5-to-GPT-4 leap.

[1:41](https://www.youtube.com/watch?v=U3MVU6JpocU&t=101s) [Slide questions whether increasing compute, model size and dataset size is enough, alongside a chart of language model performance over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/9282e00a837724bccac244d50ba8272c635fa323f5749bf0b43afe955a664cb8.webp)

Neural scaling laws

• Signs of diminishing results
• Increasing compute, model size, and dataset size is not enough?

Frontier Language Model Intelligence, Over Time¹
Artificial Analysis Quality Index: MMLU, GPQA Diamond, MATH-500, HumanEval. Top 5 Labs by Market Share.

Legend: OpenAI, Anthropic, Google, Mistral, Meta
Y-axis: Artificial Analysis Quality Index
Y-axis ticks: 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95
X-axis: Model Release Date
Timeline: 4Q22, 1Q23, 2Q23, 3Q23, 4Q23, 1Q24, 2Q24, 3Q24, 4Q24, 1Q25, 2Q25

Model labels: GPT-3.5 Turbo; GPT-4; GPT-4 Turbo; GPT-4o; o1-preview; o1²; Claude 1.2; Claude 2.1; Claude 3 Opus; Claude 3.5 Sonnet (Jun '24); Claude 3.5 Sonnet (Oct '2[illegible]); PaLM 2-L²; Gemini 1.0 Ultra; Gemini 1.5 Pro (May '24); Gemini 1.5 Pro (Sep '24); Gemini 2.0 Flash; Mistral Instruct 7B; Mistral Medium; Mistral Large; Mistral Large 2; Llama 65B²; Llama 2 Chat 70B; Llama 3 Instruct 70B; Llama 3.1 405B; Llama 3.3 70B

Interface text:
But then we hit a wall.

### AI workflow techniques

RAG grounds model responses in a user’s own data; memory supports multi-threaded conversations. The progression continues through reasoning models and agents to agentic RAG, which aims to make workflows run on their own.

[4:26](https://www.youtube.com/watch?v=U3MVU6JpocU&t=266s) [Diagram connects prompts, chain of thought, RAG, memory, long context, Graph RAG, reasoning models, agents and agentic RAG.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c16372d3f25da97adf3aa80ceb14635292ec8517f661d635e6319804e5b1433.webp)

Prompt → CoT → RAG
→ MEMORY   LONG CONTEXT →
GRAPH RAG → REASONING MODELS
→ AGENTS → AGENTIC RAG →

### The Ideal AI Product Development Process

AI product development runs as a cycle: experiment, evaluate against defined criteria, deploy, and monitor responses to improve the workflow.

[5:39](https://www.youtube.com/watch?v=U3MVU6JpocU&t=339s) [A cycle connects experiment, evaluate, deploy, and monitor around the Vellum logo.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d97c12efe18c7eebb68dbcd27292316a7ee1bb0a76ae89885f9f639f09029f3.webp)

The Ideal AI Product Development Process

EXPERIMENT
Experiment across models, prompts, and architectures

EVALUATE
Test driven development to meet defined metrics and criteria

DEPLOY
Make requests in production, and version control changes

MONITOR
Measure performance and ensure edge cases are covered

vellum

### Experiment

Before building for production, test whether models can solve the use case. Vary prompts, architectures, providers and modalities, and involve subject matter experts in the experiments.

[6:13](https://www.youtube.com/watch?v=U3MVU6JpocU&t=373s) [Experiment with prompts, architectures, models, providers, and modalities, with subject matter expert participation.](https://assets.ai.engineer/sites/aiecode2025/sha256/54f54bc421209b96bb26175f509d5c36177baf004e9a26631b4f931784baa439.webp)

Experiment
Models, prompts, architectures.

- Experiment with prompting techniques (e.g., few-shot, Chain of Thought, Tree of Thought).
- Test various architectures (e.g., single prompt, chat, multi-step chain, ReACT, RAG).
- Enable subject matter experts to participate to save engineering time.
- Stay model and inference provider agnostic to leverage the best models reasoning and pre-trained models
- Experiment with different modalities (image, audio, text))

Interface text:
something that we've seen work really well lately.

### Evaluate

Build a test set for models and workflows, using ground truth where possible. Set priorities for quality, cost, latency and privacy, then run evaluations across development and production.

[7:54](https://www.youtube.com/watch?v=U3MVU6JpocU&t=474s) [Evaluation guidance covers quality, cost, latency and privacy, ground truth, custom metrics, flexible testing, and evaluations at multiple stages.](https://assets.ai.engineer/sites/aiecode2025/sha256/d7033b10b260fe68b24846650cb92d183b5f94764d8655255386334c57664908.webp)

Evaluate
Test driven development to meet quality, cost, latency criteria

- Solve for desired quality, cost, latency, and privacy constraints
- Use ground truth data where possible
- Create custom metrics via LLM as a judge or libraries like RAGAS
- Ensure your testing framework is flexible with various metrics and a large test case bank for high accuracy needs.
- Run evals offline, inline and online

Interface text:
when it comes to evaluating your models.

### Deploy

Log calls and workflow steps to debug agent behavior; use retries and fallbacks for API outages, and test updates in staging to catch regressions before rollout.

[10:34](https://www.youtube.com/watch?v=U3MVU6JpocU&t=634s) [Deployment guidance covers logging, alerts and feedback, API resilience, version control, staging, and regression testing.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f8f703634dee89c4d4b0c49c25220d8281ffae745a41194b4a2c8950fd0bbf3.webp)

Deploy

Make requests in production, version control changes.

- Log all LLM calls, track inputs/outputs/latency, and the exact steps your AI system took.
- Set up alerts and capture user feedback (implicit or explicit)
- Maintain stateful API calls, retry, and fallback logic to handle outages and rate limits.
- Ensure good version control, use staging environments, and perform regression testing to maintain stability during updates.

Interface text:
from your users and and create a feedback loop to

### Monitor

Feed production edge cases back into offline tests, cache repeat queries to reduce cost and latency, and consider fine-tuning once enough good completions are available.

[12:37](https://www.youtube.com/watch?v=U3MVU6JpocU&t=757s) [Monitoring guidance covers caching, feedback loops for edge cases, and fine-tuning with good completions.](https://assets.ai.engineer/sites/aiecode2025/sha256/2bf1cec8ebb3f0708129e08fe27902ad23059a4e713c4f07a4c5b6e78a5d5247.webp)

Monitor
Measure quality, identify edge cases.

- Build a caching layer if you’d like to save cost or lower latency.
- Embed feedback loops to identify edge cases and add them to offline test cases.
- Fine-tune a custom model when you have enough good completions to enhance performance.

Interface text:
And so this

### TDD is more important than ever with Agents

Agents can use many tools and run tasks in parallel, so evaluation must check their decisions and intended logic alongside output accuracy.

[13:34](https://www.youtube.com/watch?v=U3MVU6JpocU&t=814s) [Slide lists tool use, parallel multi-agent execution, and evaluation of behavior alongside output accuracy.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f687dba99fc73caa42b5d466451b28f4edfa41e780c58c82862060329958d6a.webp)

TDD is more important than ever with Agents

• Using a wide range of tools
• Multi-agents executing things in parallel
• Need for evaluation of ‘behavior’ on top of functionality/accuracy of output

Interface text:
right decisions and following the intended logic.

### Five levels of agentic behavior

The proposed levels progress from a basic workflow to tool use, reasoning, autonomy, and creativity. The diagram shows an LLM exchanging information with retrieval, tools, and memory. The speaker calls the framework provisional.

[14:36](https://www.youtube.com/watch?v=U3MVU6JpocU&t=876s) [Slide lists levels L0 through L4 and diagrams an LLM connected to retrieval, tools, and memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e27eba52eb082ed51a93020aa84b136762045ed70778b23a096b6c18d8cf7bf.webp)

Five levels of agentic behavior

• L0: Basic Workflow (Caller)
• L1: Use of Tools (Actor)
• L2: Plan, Observe, Act (Reasoner)
• L3: Fully Autonomous (Operator)
• L4: Fully Creative (Inventor)

In → LLM → Out
LLM ↔ Retrieval: Query/Results
LLM ↔ Tools: Call/Response
LLM ↔ Memory: Read/Write

Interface text:
but keep in mind that this is not a final

### L0: Basic Workflow (Executor)

An LLM call passes through retrieval and inline evaluations to produce an output. Reasoning happens within the model and prompt; no external agent plans or organizes decisions.

[15:22](https://www.youtube.com/watch?v=U3MVU6JpocU&t=922s) [Slide describes invoking a model, prompting and retrieval, and API calls; a diagram leads from an LLM call through retrieval and evaluations to output.](https://assets.ai.engineer/sites/aiecode2025/sha256/e57978ee5f3cd8153c63973084f69f09bd14866f3e53baa03e79cb1a04e7f17a.webp)

L0: Basic Workflow (Executor)

• Invoke a model (reasoner model)
• Advanced prompting, RAG, Long context
• Can make API calls

LLM Call → Retrieval → In-line Evals → Final Output

Legend: LLM Call, Agent, Logic operator, Tools, Evals, Memory

Caption: and then you might have some inline evals.

### L1: Use of Tools (Actor)

A router directs the model to web search or retrieval. Memory preserves context, and in-line evaluations help check decisions before an LLM call produces the final output.

[15:55](https://www.youtube.com/watch?v=U3MVU6JpocU&t=955s) [Slide describes model tool use and looping, with a diagram connecting routing, search, retrieval, memory, evaluations, and output.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d454a5325aa207cc8eee273300f89c6a7fe481a958696df1f059ab86221cbb0.webp)

L1: Use of Tools (Actor)

• Invoke a model
• It can invoke tools (memory, crawl, search, connect)
• Loop back around and exit

Diagram:
LLM Call
Router
Web Search
Retrieval
Memory
In-line Evals
LLM Call
Final Output

Legend: LLM Call, Agent, Logic operator, Tools, Evals, Memory

Interface text:
generates an output.

### L2: Plan, Observe, Act (Operator)

A planner and coordinator feed a router that branches to search and retrieval. The workflow can evaluate and repeat steps before producing a final output and terminating.

[17:10](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1030s) [Slide describes planning, memory, a series of actions, and termination; its diagram includes a planner, coordinator, tools, evaluations, and an actor.](https://assets.ai.engineer/sites/aiecode2025/sha256/15320a34ef60ea045ee8165b30e2040290a87bc8b48914986e95ce2e8dd468e9.webp)

L2: Plan, Observe, Act (Operator)

• The model notices the trigger
• Planning, memory, series of actions (reasoning models)
• Can take long time to execute
• Terminates

Diagram:
Planner → Coordinator → Router
Router → Web Search → In-line Evals → Actor
Router → Retrieval → Actor
Actor → Router → Final Output
Memory

Legend: LLM Call; Agent; Logic operator; Tools; Evals; Memory

### L3: Fully Autonomous (Explorer)

An L3 agent continuously monitors services such as email, Slack and Google Drive, plans its next moves, and can act without waiting for a direct request. The diagram routes work through tools and evaluations, with memory feeding the loop.

[18:32](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1112s) [A slide describes an autonomous agent that acts proactively and continuously observes its environment, alongside a workflow diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/aaed82a73b7407c5aefc03dc98dfe023401b6d87bb39e998c25e796b4c0e1e4d.webp)

L3: Fully Autonomous (Explorer)

• Agent is invoked whenever, can do actions proactively
• It stays alive, never terminates
• It is constantly observing and reacting to environment

Diagram:
Email
Slack
Google Drive
...
Planner
Coordinator
Router
Web Search
Retrieval
In-line Evals
Actor
Memory
Router
Action A
Action B

Legend: LLM Call; Agent; Logic operator; Internal Tools; External Tools; Evals; Memory

### L4: Fully Creative (Inventor)

An L4 AI inventor creates its own workflows and utilities to find new solutions. The speaker says true L4 remains out of reach, citing overfitting and inductive bias.

[19:37](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1177s) [A slide describes an inventive AI that creates function calls, agents and prompts, and finds new solutions to old problems.](https://assets.ai.engineer/sites/aiecode2025/sha256/f3ea632f411fa14af0de98e590ce7f3ea0b6039010db9b475d57a4a93e1067a3.webp)

L4: Fully Creative (Inventor)

• Creates its own function calls
• Creates its own agents
• Creates its own prompts
• Finds new solutions to old problems

AI

Subtitle: it will solve problems in novel way.

### Where are we now?

The speaker places many production-grade solutions at L1, where agents use tools, and expects most innovation this year at L2, where agents react to their environment and plan actions.

[20:41](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1241s) [A matrix compares five agent roles across levels L0 through L4, with annotations for production-grade systems and innovation.](https://assets.ai.engineer/sites/aiecode2025/sha256/cca87d07c566509954c266b2d372b72b850d80f9908b00cd2151f1c738125f2d.webp)

Where are we now?

Production-grade
Most innovation

L0: Responding
Generates outputs for specific tasks using AI.

L1: Using Tools
Selects tools, loops, and runs actions in parallel.

L2: Observing/Reacting
Reacts to triggers in the environment, and plans actions.

L3: Fully Autonomous
Runs proactively, and never terminates.

L4: Fully Creative
Can create its own workflow, and invent new things

Matrix columns: L0, L1, L2, L3, L4
CALLER: ✓, ✗, ✗, ✗, ✗
ACTOR: ✓, ✓, ✗, ✗, ✗
OPERATOR: ✓, ✓, ✓, ✗, ✗
EXPLORER: ✓, ✓, ✓, ✓, ✗
INVENTOR: ✓, ✓, ✓, ✓, ✓

Video caption: all of which have deployed production grade AI

### Learn more about agents

Scan the QR code for a PDF on building agents, including architectures, testing stages, and lessons from researchers and professionals using agents in production.

[22:33](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1353s) [A slide offers a PDF about building agents and displays a QR code for the resource.](https://assets.ai.engineer/sites/aiecode2025/sha256/1321e3876aa0c6962b4d22bd68244473f147c9214a57a86491bd3364f32331b6.webp)

Want to learn more about agents?

Download our PDF with best practices of building agents

Featuring top researches & professionals using agents in production

Interface text:
download this resource.

### SEO agent example

The analyst and researcher feed a writer. An editor evaluates the draft: good work goes to the user for review, while feedback and memory feed revisions.

[23:24](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1404s) [A workflow connects an SEO analyst and researcher to a writer and editor, with feedback and memory feeding revisions.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf022f23db271096f979c04209f292a7a148fb58c331c02c15bf83396aa6f917.webp)

Let’s look at an example: SEO agent

SEO Analyst
Analyze Results
Extract Articles
Extract Intent

Router

Researcher
Identify Good Content
Identify Bad Content
Find New Info To Improve

Writer
Use Good Examples
Writing Style/Format
Generate Article

Editor
Aanalyze Article
Good
Bad
Generate Feedback
Memory

User
Human Review

Interface text:
are the steps that this workflow will take.

### SEO agent workflow demo

The running workflow sends the keyword, writing style and audience to an SEO analyst. The speaker says it examines Google-ranked articles for strong components and missing opportunities.

[25:06](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1506s) [A workflow editor shows an SEO analyst node connected to an optimization router while the workflow runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ca70f75a7f81f6c919552dddc712f583bdb6402c6e91a334e017469faad808a.webp)

Entrypoint

Linked Deployment
1 mock configured (Disabled)
SEO analyst
Node ID: [illegible]
keyword: keyword
styling: style
audience: audience
Open Deployment Details
Inputs
Outputs

optimization_router
Node ID: [illegible]
If
1 rule
Else

Subtitles: I'm gonna start running this workflow as I explain

Interface text:
Workflows > Anita > Trinity > SEO Agent > SEO-Agent
Run workflow
⌘ + return
Saved
History
Cancel
Deploy
Clear
Nodes
Inputs
Chat History
State
Time to First Byte: 0.889s
Latency: 1.215s

[25:23](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1523s) [Workflow canvas showing the SEO analyst connected to an optimization router.](https://assets.ai.engineer/sites/aiecode2025/sha256/314e82096b5105e057e2085ff6d9425645ed58dbf1fa274abe17d17584375e01.webp)

Entrypoint

Linked Deployment
1 mock configured (Disabled)
SEO analyst
Node ID: [illegible]
keyword: keyword
styling: style
audience: audience
Open Deployment Details
Inputs
Outputs
View Subworkflow Execution

optimization_router
Node ID: [illegible]
If
1 rule
Else

Subtitles: It tries to identify some good components from

Interface text:
Workflows > Anita > Trinity > SEO Agent > SEO-Agent
Saved
History
Cancel
Deploy
Clear
Nodes
Inputs
Chat History
State
Time to First Byte: 0.889s
Latency: 23.216s

### Researcher in the workflow

The demo routes the SEO analyst’s output toward the Researcher, which will search for more data on missing opportunities.

[25:31](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1531s) [Workflow canvas showing the entry point, SEO analyst, router, and researcher.](https://assets.ai.engineer/sites/aiecode2025/sha256/b7f0b24a5a639fe766d76c1c6ab689f4f87200302c0f6fd93fbc8b5f8af7c824.webp)

Entrypoint

Linked Deployment
1 mock configured (Disabled)
SEO analyst
Node ID: [illegible]
keyword: keyword
styling: style
audience: audience
Open Deployment Details
Inputs
Outputs
View Subworkflow Execution

optimization_router
Node ID: [illegible]
If
1 rule
Else

Linked Deployment
1 mock configured (Disabled)
Researcher
Node ID: [illegible]
bad_sections: SEO analyst (bad_sections)
Open Deployment Details

Subtitle: where the researcher is going to utilize those to

Interface text:
Workflows > Anita > Trinity > SEO Agent > SEO-Agent
Saved
History
Cancel
Deploy
Clear
Nodes
Inputs
Chat History
State
Time to First Byte: 0.889s
Latency: 32.215s

### Analyst output and researcher

The SEO analyst summarizes search intent and passes flagged sections to the researcher, which searches for information to address gaps in competing articles.

[25:40](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1540s) [The SEO analyst displays a search-intent summary while the researcher processes its input.](https://assets.ai.engineer/sites/aiecode2025/sha256/44e101247f16f2c67aa4fc87338d97d0396f024f50ff133f9070696aacd59ac7.webp)

SEO analyst
Linked Deployment
1 mock configured (Disabled)
Node ID: [illegible]
keyword: keyword
styling: style
audience: audience
Open Deployment Details
Inputs | Outputs
The search intent of the blog post is to explore and explain the concept of chain-of-thought (CoT) prompting in AI, particularly focusing on its application in reasoning tasks. It discusses the benefits of CoT prompting, including few-shot and zero-shot CoT, and introduces Auto-CoT as a method to automate the generation of reasoning chains, highlighting its potential to improve AI performance in complex reasoning tasks.
View Subworkflow Execution
Latency: 35.732s

optimization_router
Node ID: [illegible]
If: 1 rule
Else

Researcher
Linked Deployment
1 mock configured (Disabled)
Node ID: [illegible]
bad_sections: SEO analyst (bad_sections)
Open Deployment Details
Inputs | Outputs
View Subworkflow Execution

Interface text:
Workflows > Anita > Trinity > SEO Agent > SEO-Agent
Saved
History
Cancel
Deploy
Clear
Nodes
Inputs
Chat History
State
Time to First Byte: 0.889s
Latency: 41.216s

Caption: make our article be better than the articles that

### Researcher and writer inputs

The Researcher takes the analyst’s bad sections; the Writer combines research and analyst inputs with keyword, audience and feedback to draft an article.

[26:03](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1563s) [Workflow canvas showing the researcher connected to a writer with inputs from the analyst and researcher.](https://assets.ai.engineer/sites/aiecode2025/sha256/2abc7f862430f2d27fe696288205ae843166f2100b203ad156a986699746b56c.webp)

Researcher
Linked Deployment
1 mock configured (Disabled)
Node ID: 3c94b513-81d5-40f9...
bad_sections: SEO analyst (bad_sections)
Open Deployment Details
Inputs
Outputs
View Subworkflow Execution

Writer
Linked Deployment
1 mock configured (Disabled)
Node ID: 5e3b2939-c57f-4edc...
good_sections: SEO analyst (good_sections)
keyword: keyword
intent: SEO analyst (intent)
audience: audience
feedback: Pass Feedback
Fallback value: feedback
og_content: Researcher (good_sections)
Fallback value: research
messages: passPreviousIteration
Fallback value: messages
Open Deployment Details

Subtitle: a great first draft using that data,

Interface text:
Workflows > Anita > Trinity > SEO Agent > SEO-Agent
Saved
History
Cancel
Deploy
Clear
Nodes
Inputs
Chat History
State
Time to First Byte: 0.889s
Latency: 62.217s

[26:10](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1570s) [A workflow connects a researcher node to a writer node with article context and feedback inputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/21a374e5d02a322d25147c87f547831f4e90d58d6a0dce811f4fd0b75a801b96.webp)

Researcher
Linked Deployment
1 mock configured (Disabled)
Node ID: 3c94b513-81d5-40f9...
bad_sections: SEO analyst (bad_sections)
Open Deployment Details
Inputs
Outputs
View Subworkflow Execution

Writer
Linked Deployment
1 mock configured (Disabled)
Node ID: 5e3b2939-c57f-4edc...
good_sections: SEO analyst (good_sections)
keyword: keyword
intent: SEO analyst (intent)
audience: audience
feedback: Pass Feedback
Fallback value: feedback
og_content: Researcher (good_sections)
Fallback value: research
messages: passPreviousIteration
Fallback value: messages
Open Deployment Details

Subtitle: data as context.

Interface text:
Workflows > Anita > Trinity > SEO Agent > SEO-Agent
Saved
History
Cancel
Deploy
Clear
Nodes
Inputs
Chat History
State
Time to First Byte: 0.889s
Latency: 63.216s

### Research and draft outputs

The workflow shows research excerpts alongside the writer’s draft on chain-of-thought prompting. The speaker explains that the writer uses context from the analyzed articles to produce its first draft.

[26:15](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1575s) [Research output and a generated article draft appear beneath workflow nodes.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b6044738a78b8235f89770f2d9c3f08a59e861d101bf06143ff57f345866461.webp)

Research output
Outputs

{"tool_calls": [{"id": "[illegible]", "type": "function", "function": {"name": "section_rationale", "arguments": "{\"content\": \"Introduced in Wei et al. (2022) (opens in a new tab), chain-of-thought (CoT) prompting enables complex reasoning capabilities through intermediate reasoning steps. You can combine it with few-shot prompting to get better results on more complex tasks that require reasoning before responding.\", \"section\": \"Introduction to Chain-of-Thought Prompting\", \"rationale\": \"This section provides a clear and concise introduction to the concept of chain-of-thought prompting, explaining its purpose and how it can be combined with few-shot prompting to enhance reasoning capabilities. It sets the stage for understanding the rest of the article.\"}"}}]}

{"tool_calls": [{"id": "[illegible]", "type": "function", "function": {"name": "section_rationale", "arguments": "{\"content\": \"One recent idea that came out more recently is the idea of zero-shot CoT (opens in a new tab) (Kojima et al. 2022) that essentially involves adding \\\"Let's think step by step\\\" to the original prompt. Let's try a simple problem and see how the model performs: I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many [illegible]

Writer output
Outputs

# Mastering Chain-of-Thought Prompting in AI: A Comprehensive Guide for Developers

In the rapidly evolving landscape of artificial intelligence, the ability to perform complex reasoning is becoming increasingly important. As AI systems are tasked with more intricate problems, the need for methodologies that enable and enhance reasoning capabilities is paramount. One such methodology is Chain-of-Thought (CoT) prompting. This article delves into the intricacies of CoT prompting, its applications in AI, and the innovative approaches like zero-shot CoT and Auto-CoT that are revolutionizing the field.

## Introduction to Chain-of-Thought Prompting

Chain-of-Thought prompting, introduced by Wei et al. in 2022, represents a paradigm shift in how AI systems approach reasoning tasks. By enabling intermediate reasoning steps, CoT prompting facilitates a more structured and logical process, akin to human problem-solving methods. Imagine solving a complex puzzle; instead of attempting to piece it together in one go, you break it down into smaller, manageable parts.

Interface text:
Workflows > Anita > Trinity > SEO Agent > SEO-Agent
Saved
History
Cancel
Deploy
Clear
Nodes
Inputs
Chat History
State
Time to First Byte: 0.889s
Latency: 71.216s
audience: audience
feedback: Pass Feedback
Fallback value: feedback
og_content: Researcher (good_sections)
Fallback value: research
messages: passPreviousIteration
Fallback value: messages
Open Deployment Details
View Subworkflow Execution
Latency: 27.679s

Caption: It's not going to be something that it's really

## Full transcript

[0:00](https://www.youtube.com/watch?v=U3MVU6JpocU&t=0s) Hi, everybody. My name is Anita, and I'm currently leading GenAI Growth and Education here at Vellum. And over the last few years, we've worked with hundreds of companies who have successfully deployed reliable AI solutions in production.

[0:13](https://www.youtube.com/watch?v=U3MVU6JpocU&t=13s) From simple to more advanced agentic workflows, one thing became very clear: those companies who have adopted a test-driven development approach were able to build reliable and stronger systems for production.

[0:26](https://www.youtube.com/watch?v=U3MVU6JpocU&t=26s) Today, I'm excited to share how you can apply that same approach to build your own effective agentic workflow that actually works. But before we jump in, let's take a step back and truly understand how we got here in the first place.

[0:40](https://www.youtube.com/watch?v=U3MVU6JpocU&t=40s) I'm so excited to get started. Let's do it. So let's go back to 2023. Everyone was building AI wrappers, and most people argued that there is no defensibility strategy around them.

[0:52](https://www.youtube.com/watch?v=U3MVU6JpocU&t=52s) And fast-forward to today, we have Cursor AI, which is the most popular and widely used AI-powered IDE that just hit one hundred million ARR in just 12 months. This is the fastest-growing SaaS in the history of SaaS.

[1:08](https://www.youtube.com/watch?v=U3MVU6JpocU&t=68s) So why and how did this happen? Because models got better at coding? Sure. Because AI adoption skyrocketed? That's absolutely correct. Because coding was an obvious first target that was supposed to be disrupted by these AI models?

[1:24](https://www.youtube.com/watch?v=U3MVU6JpocU&t=84s) There is no doubt about that. But more importantly, we built new techniques and patterns on how we can orchestrate these models to work better, sync better with our data, and then work effectively in production.

[1:37](https://www.youtube.com/watch?v=U3MVU6JpocU&t=97s) We rely on these techniques because there are clear limits to model performance. Hallucinations is still a thing, overfitting is still a problem, and developers needed more structured outputs. And while model providers started to ship better tooling to solve for all of this, we didn't see another leap similar to the leap between GPT-3.5 and GPT-4.

[1:58](https://www.youtube.com/watch?v=U3MVU6JpocU&t=118s) These big jumps started to slow down. And for years, making models bigger and feeding them more data kept making them smarter. But then we hit a wall. No matter how much more data we added, these improvements started to slow down, and models started to reach their limits on existing tasks.

[2:18](https://www.youtube.com/watch?v=U3MVU6JpocU&t=138s) But is this true? Did we really hit that wall? Seems like there were some other avenues and new training methods that we still haven't explored. And so let's see what happened next.

[2:31](https://www.youtube.com/watch?v=U3MVU6JpocU&t=151s) So I don't really think that there is an issue here because since then, and this happened in the last two to three months, we've seen some new training methods that push the field forward.

[2:41](https://www.youtube.com/watch?v=U3MVU6JpocU&t=161s) For example, we got the DeepSeek-R1 model, which is the first model that was trained without using any labeled data. We call this method real reinforcement learning, and this means that this model was able to learn on its own.

[2:57](https://www.youtube.com/watch?v=U3MVU6JpocU&t=177s) Reportedly, this is what OpenAI used to train their reasoning models like o1 and o3. And all these reasoning models today, they use chain-of-thought thinking at inference time or at response time to generate their answers.

[3:12](https://www.youtube.com/watch?v=U3MVU6JpocU&t=192s) In turn, allowing these models to think before they give an answer to our questions, it enables them to really solve more complex reasoning problems. On top of this, we're seeing all of these model providers to provide more capabilities to their models, like use of tools, more capabilities for research, um, near perfect OCR accuracy when it comes to

[3:36](https://www.youtube.com/watch?v=U3MVU6JpocU&t=216s) the Gemini 2.0 Flash, and really expand the field forward. However, traditional benchmarks are so saturated, so people are starting to introduce new ones that will really capture the performance of these new reasoning models.

[3:50](https://www.youtube.com/watch?v=U3MVU6JpocU&t=230s) For example, the benchmark that you're currently seeing on the slide here, the Humanity's Last Exam, it measures performance on truly difficult tasks. So if you check the table on the slide, you can clearly see that even the latest very smart models struggle with these challenges.

[4:07](https://www.youtube.com/watch?v=U3MVU6JpocU&t=247s) So yeah, models are getting better. The field is moving forward. But for an AI product that actually works in production, success isn't just about the models anymore. It's about how you build around it, and that's exactly what's been evolving in parallel to model training.

[4:24](https://www.youtube.com/watch?v=U3MVU6JpocU&t=264s) So we were learning how to prompt all of these models better, and we developed more advanced techniques like chain of thought. Then we thought that we should be able to ground all of these models' responses using our own data, so RAG became an important part of our workflows.

[4:41](https://www.youtube.com/watch?v=U3MVU6JpocU&t=281s) Then we learned that for multi-threaded conversations, memory is gonna be the most important thing that we've had. Long context from the latest models enabled new use cases. Then we started to think about hierarchy of our responses, so we started to experiment with GraphRAG.

[4:58](https://www.youtube.com/watch?v=U3MVU6JpocU&t=298s) And then just lately, we're thinking about using all these reasoning models that, in fact, will take a lot more time to think in real time. However, it also develops new areas and use cases that we can develop.

[5:11](https://www.youtube.com/watch?v=U3MVU6JpocU&t=311s) And lately, we're thinking about Agentic RAG, making our workflows even more powerful so that all this can work on its own. And the field is still evolving. But even using these techniques isn't enough.

[5:24](https://www.youtube.com/watch?v=U3MVU6JpocU&t=324s) You need to understand your problem deeply and take a test-driven development approach to find the right mix of techniques, models, and logic that will actually work for your use case.

[5:36](https://www.youtube.com/watch?v=U3MVU6JpocU&t=336s) And this actually brings me to the main first topic of this presentation: test-driven development for building reliable AI products. Because the best AI teams that I've seen follow this structured approach.

[5:52](https://www.youtube.com/watch?v=U3MVU6JpocU&t=352s) They start to experiment, then they evaluate at scale, then finally, when they deploy in production, they never stop working on their workflow. They capture all of those responses to then continuously monitor, observe, and improve their product for their customers.

[6:08](https://www.youtube.com/watch?v=U3MVU6JpocU&t=368s) Let's look at what you can do at every stage of this process. Before you build anything production grade, you need to experiment a lot. You need to prove whether these AI models can actually solve for your use case.

[6:22](https://www.youtube.com/watch?v=U3MVU6JpocU&t=382s) So you should try different prompting techniques, for example, few-shot or chain of thought. Some of these will work great for simple tasks, and other will help with a bit more complex reasoning.

[6:32](https://www.youtube.com/watch?v=U3MVU6JpocU&t=392s) You should test various techniques. Prompt chaining is usually very, uh, well-received because it's gonna work better if you split your instructions in multiple prompts, or you can a- adopt a more agentic workflows like ReAct that will have a stage to plan and then reason and refine before it actually gives you an answer.

[6:51](https://www.youtube.com/watch?v=U3MVU6JpocU&t=411s) What is really an important part in this stage is that you need to involve your domain experts, um, because engineers shouldn't be the ones who are tweaking prompts. B- uh, and bringing all these experts will actually save a lot of your engineering time because once you do this phase right, then you will actually have a proof that

[7:09](https://www.youtube.com/watch?v=U3MVU6JpocU&t=429s) this works and that engineering time needs to be involved. At this stage, you should also stay model agnostic. Uh, you should incorporate and test different models, and especially when it comes to your use case, you need to think about which models can do the job better.

[7:23](https://www.youtube.com/watch?v=U3MVU6JpocU&t=443s) So in such case, you can, um, maybe use some, uh, different, uh, models like Gemini 2.0 Flash, which is actually really well at OCR, uh, and something that we've seen work really well lately.

[7:35](https://www.youtube.com/watch?v=U3MVU6JpocU&t=455s) So let's say that at this stage you know that, um, these AI models can actually work. You have a few examples that these models have, like, really good performance on.

[7:44](https://www.youtube.com/watch?v=U3MVU6JpocU&t=464s) But how can you test whether this will actually work in production when you will potentially have hundreds, if not thousands or millions of requests per minute? And so this is where evaluation comes in.

[7:56](https://www.youtube.com/watch?v=U3MVU6JpocU&t=476s) In this stage, you actually create a data set of hundreds of examples that you're gonna test your models and workflows against. And so at this stage you need to, uh, try to balance quality and cost and latency and privacy, and you're definitely gonna make a lot of trade-offs because no AI system is gonna get all of this

[8:14](https://www.youtube.com/watch?v=U3MVU6JpocU&t=494s) perfectly. But for example, if you need high quality, maybe you can sacrifice speed. If cost is critical, you might need some lighter and cheaper model. And this is the stage where you, where you need to define your priorities because it's always better if you define your priorities earlier in the process.

[8:32](https://www.youtube.com/watch?v=U3MVU6JpocU&t=512s) You should use ground truth data where possible. If you want to evaluate all these workflows, having your subject matter experts design these datasets and test these models and workflows against is gonna be very, very useful.

[8:46](https://www.youtube.com/watch?v=U3MVU6JpocU&t=526s) Synthetic benchmarks help, however, they will not really evaluate these models for your own use case. So it's usually very, very powerful if you can use your ground truth data.

[8:57](https://www.youtube.com/watch?v=U3MVU6JpocU&t=537s) But don't worry, even if you do not have ground truth data, you can use an LLM to evaluate another model's response. This is actually a very standard and reliable way when it comes to evaluating your models.

[9:12](https://www.youtube.com/watch?v=U3MVU6JpocU&t=552s) Very importantly at this stage, you should make sure that you're using a flexible testing framework. No matter if you're building this in-house or if you're using any external service, your AI isn't static, so your workflow should also be dynamic.

[9:27](https://www.youtube.com/watch?v=U3MVU6JpocU&t=567s) It should be able to capture all of these different non-deterministic responses. You need to be able to define custom metrics. You need to be able to write those metrics, metrics using Python or TypeScript.

[9:39](https://www.youtube.com/watch?v=U3MVU6JpocU&t=579s) So you shouldn't be looking at a very strict framework. Customizability is a very big thing here. And then finally, you should run evaluations at every stage. You should have guardrails that will check internal nodes and whether these models are actually producing responses at every step in your, uh, workflow, actually producing responses that are correct at every step

[10:02](https://www.youtube.com/watch?v=U3MVU6JpocU&t=602s) in your workflow. And then you should also test while you're prototyping, but then you should also utilize this evaluation phase to come back once you have some real, uh, data.

[10:13](https://www.youtube.com/watch?v=U3MVU6JpocU&t=613s) But how are you gonna get some real data? So let's say that you evaluate your workflows extensively with your subject matter experts, with your data that they've created, and let's say that you're now satisfied with the product that you have, so you're ready to deploy it in production.

[10:28](https://www.youtube.com/watch?v=U3MVU6JpocU&t=628s) So once that happens, what do you need to do? Is your job done here? When it comes to AI development, you need to monitor more things than deterministic outputs.

[10:39](https://www.youtube.com/watch?v=U3MVU6JpocU&t=639s) You need to log all LLM calls. You need to track all of those inputs and outputs and the latency because AI models, they really-- they're really, really unpredictable. So you need to be able to debug issues and understand how your AI behaves at every step of the way.

[10:56](https://www.youtube.com/watch?v=U3MVU6JpocU&t=656s) And this is becoming extremely more important with agentic workflows because agentic workflows are more complex workflows that can take different paths in your workflow and make decisions on their own.

[11:09](https://www.youtube.com/watch?v=U3MVU6JpocU&t=669s) You should also handle API reliability. You need to maintain, uh, stability in your API calls. You need to have retries. You need to have fallback logic to prevent outages.

[11:19](https://www.youtube.com/watch?v=U3MVU6JpocU&t=679s) For example, two months ago, OpenAI had four hours of downtime. So if you had a fallback logic in your productionized solution, then your, um, AI will know to go back to another model and use another model instead.

[11:34](https://www.youtube.com/watch?v=U3MVU6JpocU&t=694s) You should definitely have version control and staging, and you should always deploy in control environments before you roll out to the wider, uh, public. Uh, because with A- when it comes to AI, you need to be care- careful that once you update a prompt, you're not introducing a regression to another prompt or part of your workflow.

[11:54](https://www.youtube.com/watch?v=U3MVU6JpocU&t=714s) So you need to ensure that all these new updates, they won't break whatever you have in production. And the most important part here is that make sure to decouple your deployments from your scheduled app deployment schedule because the chances are That, um, you will need to update your AI features more frequently than you will need to update

[12:16](https://www.youtube.com/watch?v=U3MVU6JpocU&t=736s) your app as a whole. So make sure to do that. And so let's say that now you have deployed, you're starting to capture all of your responses from your users and create a feedback loop to identify edge cases that you capture in production to then continuously improve and make your workflow better.

[12:37](https://www.youtube.com/watch?v=U3MVU6JpocU&t=757s) You can capture all of these, then run evaluations again and test whether, uh, new prompts that you develop will solve for these new cases. You should also think about building a caching layer because if your system is handling some repeat queries, caching can drastically reduce costs and improve latency.

[12:55](https://www.youtube.com/watch?v=U3MVU6JpocU&t=775s) So for example, instead of calling an expensive LLM for the same request multiple times, you can store and serve frequent responses instantly. And this is something that is a standard these days when it comes to building with AI.

[13:08](https://www.youtube.com/watch?v=U3MVU6JpocU&t=788s) And finally, let's say that your product has been running reliably in production for, uh, a longer period of time, a time that you feel comfortable to then go back to that data and use it to fine-tune a custom model that will, um, basically, uh, create better responses for your specific use case, uh, can reduce reliance on API

[13:28](https://www.youtube.com/watch?v=U3MVU6JpocU&t=808s) calls, and in fact can work with lower costs. And so this process is becoming even more important than ever when it comes with agentic workflows because these workflows are gonna use a wide range of tools.

[13:43](https://www.youtube.com/watch?v=U3MVU6JpocU&t=823s) They will, um, call different APIs. They will have multi-agent structures that will execute a lot of things in parallel. So when it comes to evaluation with agentic workflows and with this test-driven approach, it's not just about measuring performance at every step in your workflow because you also need to assess the behavior of these agents to...

[14:06](https://www.youtube.com/watch?v=U3MVU6JpocU&t=846s) so that you can make sure that they're making the right decisions and following the intended logic. And this year, more than ever, everyone is talking about agentic workflows. But what does that actually mean?

[14:19](https://www.youtube.com/watch?v=U3MVU6JpocU&t=859s) Uh, I would love to talk more about how you can build all these agentic workflows, but I'm not here to give you the perfect definition of what an AI agent is, and instead, I'm gonna try to define different agentic behaviors and some different levels on how, um, they can be built.

[14:36](https://www.youtube.com/watch?v=U3MVU6JpocU&t=876s) So if you think about it, every AI workflow has some level of an agentic behavior in it. It's just a question of how much control, reasoning, and autonomy it has.

[14:48](https://www.youtube.com/watch?v=U3MVU6JpocU&t=888s) So we've looked at the past, the present, and where we're headed, and from that, we put together this framework where we define four or five different levels of agentic behavior.

[15:00](https://www.youtube.com/watch?v=U3MVU6JpocU&t=900s) I'll go into more details on each level, but keep in mind that this is not a final framework. It's not set in stone. As models evolve, this can expand, uh, things can blur, and, um, a lot of things can shift.

[15:13](https://www.youtube.com/watch?v=U3MVU6JpocU&t=913s) But for now, this will give us a way to define where we are today and what we expect to see next. At this stage, you have an LLM call, you retrieve some data from your vector database, and then you might have some inline evals, and finally you're gonna, uh, get some response from this workflow.

[15:30](https://www.youtube.com/watch?v=U3MVU6JpocU&t=930s) So you can notice that in this workflow, there's no reasoning, planning, or decision-making beyond what's baked into the prompt and the model behavior. So the model is doing all the reasoning here within the prompt itself, and so there is no external agent organizing, uh, the decisions or planning some actions.

[15:48](https://www.youtube.com/watch?v=U3MVU6JpocU&t=948s) However, there is some reasoning and some agentic behavior at the model's level. And so if we move from L0 to L1, we can see that in this stage, our workflows can now use a lot of tools.

[16:01](https://www.youtube.com/watch?v=U3MVU6JpocU&t=961s) And so this AI system is no longer just calling APIs, it kn- now knows when to call them and when to make those actions. And so this is where we start to see more agentic behavior because the model can decide whether, uh, it will call a specific tool or whether it will call our, uh, vector database to

[16:20](https://www.youtube.com/watch?v=U3MVU6JpocU&t=980s) retrieve more data before it actually, uh, generates an output. Memory here starts to play a key role because we're gonna have multi-threaded, uh, conversations and then, uh, all of this will potentially happen in parallel, so we need to capture all context throughout the whole workflow.

[16:37](https://www.youtube.com/watch?v=U3MVU6JpocU&t=997s) Evaluation is also needed at every sta- uh, step of the way here because we need to ensure that these models are making the right decisions, using the right tools, and returning, uh, accurate responses.

[16:49](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1009s) But these workflows can be as simple as on the slide right here, or even more complicated where you're gonna have more different branching happen, uh, happens at every stage in this workflow where you can have 10 different tools and the agent needs to reason whether it's gonna call the first five or the, or the last two.

[17:08](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1028s) And so this is, uh, where again, we see a lot more agentic behavior, but L2 is where we actually see that, um, these workflows now move from simple tool use, which is not, in many cases, it's not a simple tool use, like the previous workflows can be very complex, but now we see some structured reasoning.

[17:28](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1048s) This workflow will notice triggers, it can plan actions, and it can execute tasks in a structured sequence. So this means that it can break down a task into multiple steps, it can retrieve some information, it can decide to call another tool, it can evaluate its usefulness if it thinks that it needs to be refined at that stage.

[17:48](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1068s) And once it does this in a continuous loop, it can generate the final output. But, um, you can notice that agentic behavior here starts to look more intentional Because the system isn't just calling the tools that are, uh, listed, uh, for their use, it's also actively deciding what needs to be done and spending more time to think

[18:08](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1088s) what needs to be done instead of just deciding whether a tool should be called. And so at this stage, um, o- one part is that the process is still finite, so once this workflow completes the steps, um, as it plans to complete them, it will terminate rather than it will run continuously.

[18:27](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1107s) But it's a leap forward, um, from just calling, uh, tools. And so, um, L3 however, is where we see more autonomy, where we see more, uh, decision-making that are not, um, defined by us as the creators of this workflow.

[18:44](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1124s) So the L3 system can proactively take actions without waiting for direct input. So instead of responsing, responding to a single request and then terminating, this one will stay alive and will continuously monitor its environment and it will react as needed.

[19:01](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1141s) So for example, um, this means that it can, uh, look at your email, Slack, Google Drive, or any other tool, uh, external services actually, that you can give access to, and it can plan its next moves, whether it will execute actions in real time or asks, uh, the human for more input.

[19:18](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1158s) And so this is where, uh, our AI workflows become less of a tool and more of an independent system that we can use to truly make our work easier.

[19:28](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1168s) So for example, this one can be like a marketer that will prepare this video or a presentation that you can just take and use whenever you want. However, the final stage is where we're gonna have a fully creative workflow.

[19:41](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1181s) And so at L4, the AI moves betw- uh, beyond automation and reasoning, and it becomes an inventor. So instead of just executing predefined tasks or just like reasoning within some bounds, um, it can create its own new workflows.

[19:58](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1198s) So it can create its own utilities, whether it's agents, prompts, function calls, tools that, uh, it needs to be designed. Uh, it will pro- it will solve problems in novel way.

[20:09](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1209s) So while true L4 right now is definitely out of reach because there are some constraints with models like overfitting, because models, they really love their training data, and there is some issues with, uh, inductive bias where models will make assumptions, again, based on their training data.

[20:26](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1226s) This makes to be like a very hard task, uh, today, but that's the goal. AI that doesn't just follow instructions, but will invent, it will improve, and it will solve problems in ways we didn't even think of before.

[20:40](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1240s) So I would say that L1 is where we're seeing a lot of production-grade solutions. So at Vellum, we've worked with companies like Redfin, Drata, and Headspace, all of which have deployed production-grade AI solutions that fall within the L1 segment.

[20:54](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1254s) And again, like just using tools, it can be very simple or it can be very complex workflow. Uh, the focus is though on orchestrations. How do we train our models to interact with our system better?

[21:07](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1267s) How do we make our models to work with our data better? How do we make sure that whatever we retrieve from our vector databases is the right and correct context for the, uh, question that the user is asking?

[21:20](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1280s) And so like we're experimenting with different modalities and all of those techniques that we mentioned before, and test-driven development truly makes its case here because like you need to have different tools and models, and you need to be able to continuously improve on them to build not only a more efficient system, but a system that will work

[21:37](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1297s) continuously better and better. Um, however, L2 is where I think we're gonna see most innovation, uh, happen this year, and this is where we're gonna have a lot of AI agents that are being developed to plan and reason using models like o1 or o3 or DeepSeek.

[21:54](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1314s) Uh, we might see a bunch of different use cases. We might see a lot of innovations when it comes to the UI and the UX part of the system where we will definitely create some new experiement-- uh, experiences for our users and, um, essentially this will be, uh, a way for us to make true reasoners that will,

[22:14](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1334s) uh, handle complex tasks. So you're gonna have bunch of these agents just working for you doing, uh, different things. However, L3 and L4, they're still both limited by the models today, as well as the surrounding logic.

[22:28](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1348s) However, that doesn't mean that, um, there's not a lot of innovation happening within those two as well. So if you want to learn more about how to build your own, uh, AI agent, I've included everything that I've shared in this presentation and more.

[22:41](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1361s) For example, uh, architectures that you can build, what are the stages that you can test, and, uh, similar things like that. We also feature top researchers and professionals who have shared all of their learnings on how to build these for production.

[22:54](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1374s) So feel free to scan this QR code on the screen to download this resource. So now I think it's time to get more practical. I want to show you how I built my own SEO agent.

[23:06](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1386s) This specific agent automates my whole SEO process, from keyword research to content analysis, and finally for content creation. It decides whether to use tools and has an embedded evaluator that works as an editor to tell the agent if it's doing a good job.

[23:20](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1400s) Let's see a quick sketch of how this agent works. So in a minute, I'm gonna show you a real demo on how this agent actually works. However, I wanted to give you a high-level overview on what are the steps that this workflow will take.

[23:34](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1414s) And so when you look at the sketch on the screen right now, you're gonna notice that this workflow lies between L1 and L2 type of ag- agentic workflow. You have the SEO analyst and the researcher who will take a keyword and it will call Google search, and it will analyze the top performing articles for that keyword.

[23:52](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1432s) One, it will identify some of the good parts, uh, within these articles that we also need to amplify in our own article, but it will also identify some missing segments or areas of improvement that we should definitely write about to make sure that our Article is actually performing better than the ones that we're competing against.

[24:12](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1452s) And then after the research and planning is done, the writer has everything it needs to start writing the first draft. But then the first draft is passed to the editor, which is an LLM-based judge that will evaluate whether the first draft is good enough based on predefined rules that we've set in its prompt.

[24:30](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1470s) Then that feedback is passed back to the writer, and this will loop, uh, continuously until some, uh, criteria is met. Uh, within this loop, we also have a memory component that will capture all previous conversations between the writer and the editor.

[24:47](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1487s) And finally, we're gonna get a final article that's actually a very useful, um, piece of content that it's not AI-generated and not useful, but truly using all of this context in a smart way, enabling me to have a pretty impressive first draft to work with.

[25:04](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1504s) So now let's see the demo. For the sake of time, I'm gonna start running this workflow as I explain what this a- agent does at every step in the workflow.

[25:13](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1513s) So we ran this workflow with the keyword chain-of-thought prompting. And so the SEO analyst currently is taking that keyword, is taking some other parameters like my writing style, like the audience that we're trying to cater to, and it analyzes the top articles that Google is ranking for that specific keywords.

[25:31](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1531s) It tries to identify some good components from those articles that we need to reinforce in our article, but it also identifies some missing opportunities where the researcher is gonna utilize those to then make another search and capture more data to make our article be better than the articles that we just analyzed.

[25:51](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1551s) So now that the SEO analyst, uh, is done with its job, the researcher tries to capture more information about the things that, um, were previously identified as missing pieces to the puzzle.

[26:03](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1563s) And then the writer will take a lot of this information in its input, and it will try to create a great first draft using that data as context. So the content here that will be generated by the writer, it's not gonna be like a slop type of article.

[26:18](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1578s) It's not gonna be something that it's really not useful. It's gonna actually use all the context that we're sending from different articles that we just analyzed. You can also connect your RAG here that w- it will look into your database of articles and learnings, and it can really create something that's extremely useful.

[26:36](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1596s) Now, the editor says, "Okay, this is a good enough article, but here's some feedback." And so it passes the feedback through the memory component here, which is a chat history between these two, and then, um, this node that, uh, basically structures that input.

[26:50](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1610s) For the sake of this demo, the conditional here for the loop is that this loop will break if the evaluator actually tells us that this is an excellent post, which actually rarely happens.

[27:02](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1622s) So, um, we also said that if the loop runs for at least one time, this loop will break. And so it already ran for one time. We got still more feedback from the editor.

[27:14](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1634s) But in this case, for this demo, let's look at the output that we got. So mastering chain-of-thought prompting in AI, a comprehensive guide for developers. I think it's pretty okay, pretty nice.

[27:25](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1645s) I might change the title, but I know that the components in this article are the actual components that other articles are writing about. And so, uh, this was great.

[27:35](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1655s) The latency was around 118. This usually takes around 300 seconds to run when we have more evaluation loops, but it's pretty great. It gives me some foundations on how I can continue to build on this content, and it saves me a lot of my time.

[27:49](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1669s) So the product that I just used is called Vellum Workflows, and it was designed to bridge the gap between the product and engineering teams so they can speed up AI development while still following this test-driven approach that we talked so much about in this presentation.

[28:04](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1684s) However, one thing became clear. Developers want more code, developers want more control and flexibility, and they want to own their definitions in their code base. So today, I'm excited to introduce our Workflows SDK.

[28:16](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1696s) It provides all the building blocks you need. It's infinitely customizable, and it has a self-documenting syntax where you can actually spot how this agent is working right in your code.

[28:27](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1707s) It's also expressive enough so that you can understand what's happening at every stage in your code. The best part is that the UI and the code stay in sync, so whether you're defining, debugging, or improving your workflows, everyone on your team can stay aligned.

[28:42](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1722s) I hope that you like it. It's open source and free, and you can check it out on GitHub. Feel free to run, uh, to scan this QR code, uh, to check out the repo.

[28:52](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1732s) And that's a wrap. Thank you so much for listening, and I hope that today you learned something new. If you want to talk more about AI, feel free to scan this QR code on the screen to connect on LinkedIn, or if you have any questions, feel free to, um, send me a text message on my email or

[29:07](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1747s) on Twitter. I'm gonna follow up for sure.
