Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust

A few successful prompts can make an AI feature look ready to ship. They cannot tell you how often it fails, which users it serves poorly, or what an improvement costs elsewhere. Jess Wang, a DevRel engineer at Braintrust, opens with teams making release decisions on exactly this kind of impression: engineering says it is ready, or a product manager tries a couple of prompts and likes the answers.

An eval turns that impression into a testable statement. Wang’s illustrative alternatives are concrete: 200 test cases with a 94% pass rate, or a change that improves accuracy by 5% while worsening tone by 5%. These are examples of the decisions an eval can support, rather than results from the coding experiment that follows. The useful starting point is a question about the system.

Those questions can concern model choice, performance across languages or programming languages, cost, company voice, or regressions. A system that works well for English inputs may struggle with Japanese; success on Python does not establish success on TypeScript. Wang also recalls OpenAI’s April 2025 update that became overly agreeable and less truthful. That example gives regression testing a specific target: an apparent improvement in helpfulness can damage another behavior users depend on.

An eval needs an input collection, a defined behavior, a quality standard, and a configuration you can compare with another configuration:

• Dataset. Inputs should include strong reference cases, edge cases, and known failure modes.
• Task. Define how the system turns an input into an output, usually including the system prompt and chosen model.
• Scorer. Specify what makes the output good or bad. Deterministic checks, an LLM judge, and human review are different ways to apply that standard.
• Experiment. A particular dataset, task, and scoring system form one experiment. Changing one of them creates another configuration to evaluate.

Comparison should reach below the aggregate score. Individual rows show which cases improved and which regressed. Braintrust’s interface also supports natural-language queries across experiments and traces—for example, asking for recurring problems or the best-performing experiment. That can help navigate long, multi-turn conversations where hallucination or drift is difficult to spot manually. The talk describes UI, CLI, and MCP access to this analysis.

Observability supplies the inputs for the next round of evaluation. In Wang’s example, a newly shipped documentation chatbot starts receiving real customer questions. Wrapping application code with the Braintrust SDK brings production logs into the platform, where the team can monitor behavior and build dashboards.

How does a customer conversation become an application change? Sample perhaps 10–20 log rows into a dataset, define the task and scorer, and compare candidate changes. A promising experiment informs a code or system change, which is deployed back into the chatbot. The cycle below makes the feedback path visible: deployment changes the behavior that produces the next set of logs.

This work crosses job titles. Engineers bring in data and change code; product managers formulate hypotheses about success and help adjust prompts; subject matter experts label examples and define quality in fields such as insurance, medicine, and law; analysts examine the results. “Evals are a team sport” is a practical observation about who can supply the inputs and judgments the loop requires.

The concrete experiment begins with a request from Wang’s CEO after seeing a Cursor post about search improving coding-agent performance. Online discussion provided a hypothesis worth testing: how would vector search and agentic search compare on real coding tasks?

Vector search converts code or text chunks into embeddings that represent semantic meaning. Wang illustrates the idea with animals and fruit: wolf, dog, and cat sit near one another, while banana and apple form another nearby group. A vector database, such as Qdrant or Pinecone, stores these representations. A query asking where database logic lives retrieves chunks with similar semantic content.

Agentic search gives an LLM tools such as grep, find, ls, cat, and shell commands. The agent can find a function name, open the file, read the function, follow a call into another file, and continue investigating. Each observation helps determine the next action. This resembles a developer navigating a repository: the route through the code develops as the agent learns what connects to what.

Microsoft’s TypeScript Go repository supplied the dataset. The team selected merged pull requests with “fix” in the title and checked out each parent commit, returning the repository to a state where the bug still existed. Claude then generated a bug-task description from the difference between buggy and fixed code. The resulting task asked the agent to locate the faulty logic using either retrieval or agentic search. The dataset contained approximately 20 PRs.

Both conditions used Claude Code as the harness. Agentic search required little extra implementation because it was the default behavior. The vector-search condition required active restrictions: otherwise the agent could fall back to ordinary repository exploration and blur the comparison.

Two controls enforced that restriction:

• Prompt instruction. Explicitly tell the agent not to use agentic search.
• Tool restriction. Use the disallowed-tools flag to block tools it would normally use for that exploration.

The second control matters because an instruction alone does not prevent a tool call. Holding the harness steady while restricting the available search behavior made the experiment’s intended difference concrete.

The Claude Code runs executed as subprocesses, and their traces initially became orphaned. The parent trace showed a “Run Claude Agent” step, but it did not expose what happened inside that process. That left a crucial gap: a final score could show success or failure without explaining the search that produced it.

Passing parent span IDs through environment variables connected the subprocess activity to the parent trace. After that change, the trace showed individual turns, LLM calls, and terminal commands inside the run. The observable change was from a single opaque subprocess step to an inspectable sequence of actions—the information needed to debug the search behavior later.

Scoring used the repository’s own test suite. A run that passed received 100%; a failure received 0%. This binary rule gave both search conditions the same executable success criterion. It also separated two questions: whether the run passed, and what work the agent performed along the way.

The two conditions reached the same reported accuracy, while vector search cost about four times as much. This is a result for the tested implementation and dataset: Wang later describes the vector implementation as relatively weak, the sample as approximately 20 rows, and repeated trials and additional repositories as important improvements. The comparison does not establish a general ranking of the two search methods.

The traces gave the cost difference a plausible mechanism. Retrieved chunks often omitted imports, tool calls, or surrounding calling code. A chunk could be relevant to the bug while still leaving the agent unable to understand how that code participated in the larger behavior.

One dataset row makes the distinction concrete. The vector agent performed 26 searches and still could not piece together how three functions interacted across the file. Repeated retrieval brought it near the code without resolving the relationship among those functions. Agentic search could grep a function name, read the full function, identify a flaw in its logic, and follow the calling logic into another file. Reading the surrounding code supplied the next investigative step.

Where did the extra work accumulate? The comparison below follows the search behavior described in the traces. The vector path repeatedly returned another chunk while the missing relationship remained unresolved. The agentic path used a discovered call as a route into further context. Wang’s phrase “connective tissue” captures the useful difference: proximity found relevant pieces; navigation helped connect their behavior.

That unresolved context drove more searching, which drove more LLM calls. Those calls accumulated cost even though aggregate accuracy ended up the same. The traces therefore changed the interpretation of the score: equal success rates concealed different amounts of work, and the expensive condition spent much of that work trying to reconstruct context.

The ending gives three practical improvements to the experiment:

• Repeat each task. LLM behavior varies between runs; multiple trials make the resulting score more trustworthy.
• Improve vector retrieval. A stronger implementation would test whether the observed context problem persists beyond this particular setup.
• Broaden the dataset. More cases and different repositories would extend the comparison beyond approximately 20 tasks in one codebase.

The useful achievement is an end-to-end evaluation that connects a real question to real bugs, controlled tool access, executable scoring, and inspectable behavior. Its most actionable finding comes from combining the score with the trace: when an agent keeps searching, examine whether each result gives it enough context to choose a productive next step.