How to look at your data; what to look for, how to measure

Which chunking strategy should you use? Is your embedding model the best choice for your data? These are application questions: answering them requires a way to measure what changes when you change the system. Jeff Huber starts with retrieval inputs; Jason Liu then turns to conversations and agent outputs. Together, those two views connect engineering experiments to the problems users actually encounter. Measurement makes improvement systematic.

To assess retrieval, you could guess, use an LLM judge, or consult a public benchmark such as MTEB. Each offers a different kind of evidence. A public embedding score does not directly tell you whether your application retrieves the documents it needs. Huber’s example of an expensive judge-based evaluation costs $600 and takes three hours; it is an illustration, not a benchmark with a specified workload. His proposed starting point is a fast eval that checks retrieval directly.

A golden dataset contains queries paired with documents that should be returned. The evaluation asks a concrete question: when this query goes in, does the expected document come out?

1. Associate each evaluation query with its relevant document IDs.
2. Run the query through the retrieval system.
3. Inspect the first k results—perhaps five, ten, or twenty, depending on the application.
4. Check which expected documents appear in those results.

A small Python function expresses the per-query recall check:

def recall_at_k(
    expected_ids: set[str],
    ranked_ids: list[str],
    k: int = 10,
) -> float:
    if not expected_ids:
        raise ValueError("A query needs at least one relevant document.")
    if k <= 0:
        raise ValueError("k must be positive.")

    retrieved_ids = set(ranked_ids[:k])
    return len(expected_ids & retrieved_ids) / len(expected_ids)

With one expected document, the result is either zero or one. With several, it measures the fraction of relevant documents recovered at the chosen depth.

The short feedback loop matters as much as the metric. Huber describes fast retrieval checks as running quickly for pennies, without specifying a workload. Waiting hours between experiments drains the time and attention available for trying alternatives. A cheap check makes it practical to test a change, inspect failures, and try again.

You may already have documents, chunks, and a retrieval system without having user queries. An LLM can help create the missing questions, but simply asking it to write a question about each document is insufficient. The generator needs guidance about how users ask questions, especially how specific those questions should be.

The pergola example makes the problem visible. The query asks, “What is a pergola used for in a garden?” The answer passage begins by naming a pergola in a garden. That close correspondence makes a tidy benchmark pair, but it can also make retrieval easier than it will be in practice. A question written with the answer directly in view may contain precisely the vocabulary and detail needed to find that answer. Chroma’s Generative Benchmarking research investigates how to produce queries that better represent real usage.

Synthetic queries must reflect user specificity, not just document content. Huber describes aligning the specificity of generated queries with real queries. Otherwise, an evaluation can reward the system for answering artificially convenient questions and leave you with an inflated impression of retrieval quality.

When a new embedding model attracts attention, a golden dataset gives you a better question than whether its public ranking is impressive: does it retrieve more of the expected documents for the queries you care about? An improvement still has to justify the cost of switching. You must re-embed the corpus, and the new service may cost more, respond more slowly, or have a less reliable API. Retrieval success supplies a concrete quality measure to weigh against those operational constraints.

For the Weights & Biases chatbot, Huber compares four embedding models using Recall@10. The ground-truth queries came from interactions logged in Weave; the other queries were generated synthetically. A useful synthetic evaluation should produce reasonably similar scores and preserve the ordering of models, so that it supports the same engineering decision.

The Generative Benchmarking report supplies these exact results. Its ground truth uses weighted, manually labeled production queries from 2023, and the evaluation excludes queries without relevant documents.

The real and generated evaluations preserve the same model ordering for Recall@10. The application’s original model, text-embedding-3-small, finishes last. Huber contrasts what he describes as Jina’s strong English MTEB standing with its weaker result on this application; voyage-3-large leads this comparison.

Editor’s note: This agreement is specific to Recall@10. The report’s NDCG@10 results reverse the order of text-embedding-3-small and jina-embeddings-v3 between ground-truth and generated queries. Synthetic evaluation agreement should therefore be checked for the metric used to make the decision.

Huber points to the full report, a companion video, and open-source notebooks for applying the method to other datasets. The useful result is a repeatable comparison on your own data, with enough detail to inspect why one configuration succeeds where another fails.

Retrieval tells you what information enters the system. Conversations and agent executions show what happens next: what users attempt, which tools they need, and where interactions go wrong. Liu recommends starting with careful manual inspection when the collection is still manageable—even a few hundred conversations can merit individual attention. Sending everything to o3 should not be an automatic substitute for understanding the interactions yourself.

As usage grows to thousands of queries or tens of thousands of conversations, manual review becomes harder. Volume is only part of the problem. Long interactions contain tool calls, chains, and reasoning steps that are difficult to scan, and interpreting them may require expertise the reviewer does not have. The analysis needs a way to reduce detail without losing the signals that matter.

Those signals often exist without a separate feedback widget. A user asks for another attempt, corrects the assistant, or says, “This is not really what I meant.” Frustration and retry patterns are evidence about the interaction. Extracting them from the conversation can reveal failures that a thumbs-up or thumbs-down control never captured.

An evaluation score of 0.5 or a factuality score of 0.6 does not, by itself, identify what to change; Liu uses both as illustrative numbers. His marketing analogy shows what is missing. In the hypothetical campaign, 80% of users belong to the younger cohort and 20% to the older cohort, with stronger performance in the younger cohort. The audience split makes the aggregate result actionable.

Now there are two concrete directions:

• Expand the successful segment: Invest further in reaching the younger audience.
• Investigate the weaker segment: Find out why the campaign performs poorly with older users and change the approach for them.

Even channel choices—such as podcasts or a Super Bowl advertisement—become easier to reason about once the intended audience is clear. The same principle applies to AI applications: a segment connects a performance problem to a population and a possible intervention.

The bridge from conversation text to segmentation is structured extraction. Instead of asking for an unconstrained interpretation of an entire conversation collection, define the information each record should contain:

• Summary: What happened in the interaction.
• Tools: Which tools the system used.
• Errors: What failed or appeared incorrect.
• Interaction details: Events needed to understand the conversation.
• User response: Signals of satisfaction and frustration.

This creates a portfolio of metadata. You can embed the extracted information, find clusters, identify segments, and begin testing hypotheses about them.

An LLM helps produce those structured records; the next step is conventional product analysis. Work such as Anthropic’s Clio provides a reference point for finding patterns in real AI usage. Liu illustrates the investment opportunity with a roughly 40× comparison between coding in Claude usage and GDP value creation. The practical connection is that understanding what people use a product for helps identify where to invest.

Editor’s note: The published Clio account does not establish that GDP comparison; the ratio remains Liu’s illustration, not a verified Clio result.

Kura organizes this workflow around conversation summaries, clusters, and hierarchies of clusters. Those groups let you compare evaluation metrics across different kinds of requests. A global factuality score might conceal poor results for queries requiring time filters and strong results for contract searches. Separating those cases identifies where the system works and where further investigation belongs.

The pipeline has three stages: summarization, clustering, and aggregation. Liu demonstrates it with an explicitly synthetic dataset of conversations generated with Gemini.

1. Summarize each conversation. Extract topics, frustrations, errors, and other useful attributes into a consistent record.
2. Find cohesive groups. The demonstration produces groups such as data visualization requests, SEO content requests, and authentication errors.
3. Aggregate groups into a hierarchy. Broader themes, such as technical support, make it possible to inspect related needs together while retaining more specific categories underneath.

The resulting cluster summaries expose practical questions. Does the agent have tools for database debugging? Can it diagnose authentication problems? Can it create the visualizations people request? At a broad level, the collection may reveal SEO content or data analysis; at a finer level, it may distinguish blog posts and marketing tasks. Moving between those levels helps generate hypotheses about tools, prompts, and even how to market the product.

Segmentation becomes useful when it leads to a specific investment. In Liu’s hypothetical example, 80% of conversations concern SEO optimization. That concentration would justify investigating SEO integrations, stronger workflows, or prompt changes for that use case. The objective is to give the agent the tools, metadata filters, and data sources needed for the work users bring to it.

Sometimes the missing capability is ordinary infrastructure. Queries that depend on time may perform poorly because the system has no time filter. Changing the model does not add that filter. Liu also describes a contract-search case where extracting signature information during OCR made it possible to filter signed contracts at scale. In both examples, analyzing the requests points to a specific missing input or operation that engineering can supply.

Defining evaluations is only the beginning. Comparing KPIs across clusters adds the information needed to decide what to build, fix, or leave alone. Two dimensions are especially useful: how much of the population uses a capability, and how well the system performs for that category.

The low-usage cases are consequential. A strong capability may be hidden by the interface, while a weak capability with little demand may not deserve a large engineering investment. Conversation share and evaluation performance together make those choices more explicit.

Once useful categories emerge, they can become operational parts of the application. Build classifiers to recognize intents, routers to direct requests, and tools for the needs you have identified. Monitoring can then group results by query type and track performance within each category over time. A change inside a recognizable use case is more informative than an isolated aggregate number.

The categories also expose changes in demand. Liu describes newly onboarded customers using applications differently from historical customers. Looking at those differences can reveal that the next investment should serve a new pattern of usage. Research into actual interactions becomes an input to the product roadmap.

Clearer hypotheses and faster experiments reinforce each other. Segmentation narrows a question to a particular population or failure mode. Fast evaluations let you test more proposed changes, and monitoring shows what happens as the product continues to be used. Progress depends on both the quality of the hypotheses and the ability to test them repeatedly.

For retrieval applications, Liu puts evidence quality first: evaluate on application data and establish that the needed documents are being retrieved before spending heavily on changes to the LLM. His rationale is that an improved model does not itself repair missing retrieval evidence. Before users arrive, synthetic queries can help establish an evaluation set; once real usage exists, inspect it and use it to improve that understanding.

Conversation counts, tool misuse, errors, and frustration let you connect a failure pattern to the population experiencing it. Comparing KPIs across similar conversations turns a broad request for more tools into a claim about where those tools could matter.

Liu’s closing hypothetical has 40% of conversations involving data visualization, while the code engine handles that work well only about 10% of the time. A proposal to build two plotting tools now has a defined audience, a specific weakness, and a change to evaluate. The experiment can ask whether the tools improve performance for that segment. Starting with a small amount of structure is enough to begin making these choices about what to fix, build, or ignore.

The closing resources point to Chroma research and three Jupyter notebooks that load Weights & Biases data, perform cluster analysis, and use the results to inform product decisions. These return to application data after the earlier synthetic Gemini demonstration.

Editor’s note: The first notebook in the related Kura tutorial series represents individual query/document pairs as single-message conversations. Its inputs should not be mistaken for complete multi-turn chatbot histories.

Asked for a provocative take in the closing Q&A, Liu shifts to the business model: more agent businesses should price services according to work done rather than tokens consumed. The exchange ends with success and value as the basis for charging customers.