Evals in AI: A Deep Dive — Tejas Kumar, IBM

An agent fetching a conference schedule encounters an HTML comment telling it to ignore its previous instructions. In Tejas Kumar’s account, Claude Code returns the requested schedule information and reports that it treated the attempted injection as data. The protection matters at the moment the agent reads the page: external content must not become instructions merely because a tool returned it.

Kumar, an AI engineer on IBM’s watsonx.data team, uses this example to distinguish two kinds of reliability. A harness—the software that runs the model, tools, and surrounding controls—protects an agent during execution. Evals exercise behavior ahead of deployment, when a failure can still stop a release. His analogy is “two pedals on a bicycle”: both help the system move safely, and neither replaces the other.

The security stakes rise when an agent can change accounts rather than merely answer questions. Kumar describes an alleged Instagram support-bot incident involving unauthorized secondary-email changes. He does not establish the incident’s details or internal cause, so the useful lesson is the failure mode: an account-changing tool needs identity checks at runtime, and adversarial evaluation cases should test whether persuasive requests can bypass them before deployment.

A unit test for addition has an easy target: add(1, 2) should return 3. A customer-support answer can express the same correct decision in many ways. Kumar’s deliberately plain description, “fuzzy unit tests,” changes the question from whether a function returned one exact value to whether an agent behaved acceptably across scenarios. The fuzziness belongs to judging meaning; it does not remove the need to specify acceptable behavior.

The evaluation has four cooperating parts:

• Test cases: customer requests and the responses or actions produced for them.
• Expected behavior: the outcome the system should reach, which need not be one prescribed sentence.
• A judge: a mechanism that decides whether the behavior meets the standard.
• Aggregation: a way to combine individual results into a score that supports a decision.

Those parts can answer different questions: did the agent follow policy, use the appropriate tools, and resolve the issue? Some answers remain ordinary software checks. If tool calls are stored as JSON, code can check the tool name and arguments directly. Schema validation can check a message’s structure. Neither task automatically needs another model.

For judgments about response quality, pairwise evaluation asks which of two answers is preferable. Pointwise evaluation asks whether one answer meets a standard. Kumar favors pointwise judgments for the support-policy exercise because they make the criterion explicit and avoid choosing merely the better of two options. His cited numerical comparison of attack vulnerability lacks a defined study and metric here; it does not establish that pointwise evaluation is universally more reliable.

An LLM judge can interpret meaning, but its preferences can drift away from the task. Kumar identifies four distortions worth testing:

• Position bias: a judge favors the first answer. Swap the options and check whether the preference follows position rather than content.
• Sycophancy: an agreeable answer wins even when it violates policy. Approving a return may sound helpful while being operationally wrong.
• Self-preference: a judge favors text from its own model family. Kumar recommends considering a different family for judging generated answers.
• Verbosity bias: a long, polished answer receives credit that its actual decision does not deserve.

Calibration starts with examples containing a request, an answer, and a human verdict. Kumar suggests roughly 30 examples as a starting collection. Hide the human verdict from the judge, ask it to score the request–answer pair independently, then compare the two verdicts. “Training” here means refining the judge’s instructions, context, or model selection; it does not mean updating model weights.

The workshop targets 80–85% agreement and proposes blocking a release below 80%. These are chosen operating thresholds, not a demonstrated universal standard. Kumar also treats 100% agreement as a warning about overfitting or sycophancy. Perfect agreement on a small collection alone cannot diagnose either problem; the substantive concern is whether the judge has learned a useful standard that carries to unfamiliar cases.

Deployment introduces two moving targets. A model alias may begin pointing to a different version, so pinning the judge’s version helps keep changes interpretable. Meanwhile, real customer requests expose scenarios the original dataset missed. Sample those requests and responses, have humans label them, and add them to the evaluation collection. Policy moves too: OpenRAG enters the design as a way to retrieve current organizational rules rather than continually copying them into prompts.

The live exercise begins with a TypeScript addition function and a passing Vitest test, then replaces arithmetic with a customer scenario: headphones were purchased 20 days ago, and the customer wants to return them. The intended policy allows returns within 14 days. A correct answer should therefore refuse the return.

The first approximation checks whether an answer contains cannot:

expect(answer).toContain("cannot");

Initially, an approval fails and a refusal passes. Then the approval’s shipping sentence changes to say the customer cannot pay for shipping. The test turns green even though the answer still approves an out-of-policy return. The assertion recognizes a word, but cannot tell which action that word negates.

A judge using GPT-3.5 Turbo replaces the substring check. Its prompt receives the scenario and two numbered answers, then requests only the number of the correct answer. It initially selects answer two, the refusal. Repeating the call matters because one successful response says little about consistency. The run also encounters a network timeout, prompting a 30-second test timeout; an incomplete API call and a wrong verdict are different failure modes.

Next, the approving answer is regenerated with the same model family. The judge starts choosing that wrong answer consistently. Kumar interprets the change as self-preference, though the replacement also changes the answer’s wording, so this experiment does not isolate authorship as the cause. The repair is concrete: add the 14-day return policy to the judge’s prompt. The judge returns to the intended refusal. At this stage the policy is interpolated text; retrieval comes later.

The exercise grows into 30 cases with human verdicts. A defective coffee grinder within the return window receives a passing verdict for offering a replacement; an answer claiming returns are only for unused items receives a failing verdict. The distinction matters: a defect and a change of mind can have different rules. Each case now asks whether one response follows policy, rather than which of two responses sounds more helpful.

The judge receives the scenario and answer but not humanVerdict. Its requested output is just pass or fail. Initial agreement is poor, and some mismatches come from capitalization rather than judgment. Lowercasing the output removes that accidental difference. Agreement still falls short, so the next decision is whether to supply better context or buy a more capable model. Kumar chooses context first to control cost.

Adding a 14-day window and a defective-item rule improves the results but leaves disagreements. A yoga-mat case exposes another rule: change-of-mind returns require an unused item in its original packaging. The discussion of that case does not fully establish the candidate answer or why its human verdict is correct, so it cannot settle the mat’s eligibility. It does show why a time window alone is an incomplete judging standard.

When further context still does not produce sufficient agreement, the judge changes to GPT-4o mini. Output capitalization becomes more consistent, but a policy disagreement remains. A loyal customer asks for an exception after a speaker stops charging at six weeks. The candidate answer acknowledges the relationship and refuses because the defect window is 14 days. The human labels that answer a pass; the judge labels it a fail.

The next prompt edit states that loyalty and polite appeals do not change the return policy. That makes an implicit human assumption available to the judge: empathy can change the tone, but cannot authorize an exception. The subsequent run records 27 agreements and three disagreements, or 90%. This is the work behind calibration—inspect a mismatch, identify the missing criterion, express it, and run the collection again.

Where does the human verdict enter this process without giving away the answer? The diagram separates the judge’s inputs from the later comparison. Disagreements return to inspection, where policy context or the model can change; the reference verdict stays outside the judging prompt.

Individual assertions still leave the test suite red when three verdicts disagree. The intended gate needs a different assertion: count matching verdicts, divide by the number of cases, and fail only below the selected threshold.

At an 80% threshold, 24 of 30 cases must agree, so six disagreements are allowed. The earlier live discussion uses five as its failure budget, which is stricter than the stated threshold.

After the test is rewritten, another run passes at exactly 80%. The earlier run reached 90%; the aggregate assertion changes how results block CI, not the judge’s underlying decisions. An audience question then prompts a proposed upper bound to flag 100% agreement, but that upper-bound check is not demonstrated. The gate measures agreement with these human labels, while a runtime harness remains necessary for failures that escape the collection.

A browser demo returns to the same headphones: 20 days since purchase, a 14-day return window, and two possible replies. Five naive judgments choose incorrectly; including policy makes the judgments correct in the demonstrated rerun. OpenRAG then takes the next step: instead of writing the rule directly into the prompt, make the policy available through a knowledge base.

The described ingestion pipeline uses Docling to convert documents and other source material into model-ready content. OpenSearch stores the resulting searchable material, while the system handles chunking and embeddings. This moves policy maintenance into the document collection: a judge can retrieve relevant rules rather than depend on whichever abbreviated version was last copied into its prompt.

The observable change is particularly clear. The first headphone query finds no relevant supporting sources. Kumar uploads a file called Refund Policy; Docling parses it, and the system stores and indexes it. With nothing else changed, the repeated query retrieves the rule and refuses the return because 20 days exceeds 14. The new document supplies the missing basis for the decision.

How does a newly uploaded file reach the decision? The diagram shows ingestion feeding retrieval, with the unchanged customer question entering separately. Retrieval supplies policy context; the answering or judging model must still apply it. The demonstration establishes this successful path, rather than showing that every relevant rule will always be retrieved.

The closing advice is to buy semantic judgment only when simpler checks stop answering the question. Start with deterministic tests, then use pattern matching where it captures the requirement. Inspect recorded inputs, outputs, and tool calls for structural and operational checks: which tool ran, how often, with what arguments, and for how long. When meaning requires an LLM judge, begin with a low-cost model and improve context before escalating capability. Vitest was enough to host this exercise; a dedicated evaluation platform is optional.

Waiting for customers is not the only way to expand coverage. Synthetic cases can use policy, previous requests, and known failures to generate plausible new requests. Adversarial generation asks for cases likely to break the system; red-team prompts belong in the evaluation dataset too. Kumar’s compact instruction is “Here’s everything that failed. Now generate more things that could fail.” This accelerates case discovery, while human labeling remains the source of the reference standard used in calibration.

A question about coding-agent skills sharpens the responsibility discussion. Kumar says he does not write evals for his own skills, because he relies on the coding-harness provider to evaluate protections against skills poisoning a conversation. That is his division of responsibility and an assumption about the provider’s coverage. A skill’s written text can be fixed while the agent’s behavior under that text remains variable, so the answer should not be read as a general guarantee that skill-driven behavior needs no evaluation.

The final design decision is policy parity. The production agent and judge should retrieve the same rules from the same source, rather than maintain separate copies that can drift. Retrieval can be explicitly invoked or autonomous; Kumar prefers matching the production agent’s retrieval tools so the judge operates with comparable context. This complements his earlier concern about model-family self-preference: sharing policy and retrieval capabilities does not require using the same model family.

The intended signal is practical: if an evaluation fails under the policy and retrieval conditions the deployed agent will face, there is a reason to investigate before shipping. Fresh cases keep that signal relevant. Runtime protections cover the next request that the dataset did not anticipate.