Your agents lack context: Here's how to fix "You're absolutely right!" — Brandon Waselnuk, Unblocked
Read the talk
Your Agents Lack Context: How to Fix “You’re Absolutely Right!”
Brandon Waselnuk of Unblocked explains why capable coding agents still produce locally plausible, unmergeable work—and how identity-aware retrieval, conflict resolution, permissions, and structured queries can give them the organizational context they lack.
From a talk by Brandon Waselnuk
At a glance
Ideas worth remembering
Agent errors become more expensive as teams move from autocomplete toward parallel and background execution, because misunderstandings survive longer before a human catches them.
Static Markdown and MCP connections are useful, but maintenance, tool selection, and satisfaction-of-search failures keep them from supplying organizational understanding by themselves.
A context engine must combine cross-system retrieval with identity-aware relevance, conflict resolution, permission enforcement, and token-efficient delivery.
Semantic retrieval and structured querying solve different problems: RAG finds related material, while deterministic queries handle identities, relationships, statuses, and time ranges.
The reported 21-million-versus-10.8-million-token comparison illustrates the potential cost of repeated discovery, but it is a single speaker-reported task rather than a general benchmark.
The agent starts as a brilliant new hire
AI-generated code should feel like it came from someone who has worked on your team for years. That standard is demanding because experienced engineers have absorbed far more than source code. Rejected pull requests taught them local conventions. Meetings supplied product history. An overnight production incident connected a rollout procedure to its consequences. In effect, the engineer’s brain became a context engine through repeated exposure to how this company actually works. 1:13
A new terminal session has none of that history. The model may be intelligent, but it does not know how this organization ships, which architectural decisions still apply, or what happened the last time a similar change reached production. If that missing context sends the agent in the wrong direction, every later action builds on a bad starting point. 1:51
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bad context gets more expensive as autonomy increases
Autocomplete kept the cost of a context failure small. A suggestion appeared, an engineer judged it using local knowledge, and either pressed Tab or rejected it. Moving toward agentic IDEs, parallel workers, and background agents removes that immediate filter. The agent must recognize when it has reached a wall, ask the right questions, and produce code that fits a revenue-bearing brownfield system—not merely a greenfield demo. 2:17
What changes as the human moves farther out of the loop? The diagram follows the accumulating cost. A wrong autocomplete suggestion is discarded immediately; a longer agent run spends search tokens and rework on repeated corrections; parallel agents create more output to review; and a background agent needs machine-accessible context if it is expected to continue without asking a person. The relationship to notice is compounding: greater autonomy magnifies an early context error rather than merely repeating it. 3:17
The familiar failure is the correction loop: the agent announces that it finished, the engineer rejects the result, and several rounds of repair follow. Those rounds consume search tokens and human time. Parallel agents add a review tax because reviewers also need business logic and operating context to distinguish a plausible patch from a mergeable one. Fully unattended work raises the requirement again: the context source must remain queryable while the agent operates. 3:17
The compounding cost of missing context
Read the diagram as text
- Autocomplete. human-in-loop — A human immediately accepts or rejects one suggestion.
- Agent correction loop. supervised-agent — Wrong assumptions generate repeated search and rework.
- Parallel agents. scaled-agents — More generated work creates a review tax.
- Background agents. autonomous — Unattended workers need context they can query themselves.
- Autocomplete → Agent correction loop: Less immediate filtering.
- Agent correction loop → Parallel agents: More work reaches review.
- Parallel agents → Background agents: Human leaves the loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two useful approaches that plateau
Static project instructions can improve an agent, but they create what Waselnuk calls the curated context trap. A team writes Markdown files, gives agents a filesystem to search, and sees better results. Then the operational problems begin: the files must be distributed, they age like other documentation, and someone must decide which version of the organization’s practices deserves to be authoritative for everyone. 4:11
MCP access solves a different problem: it gives an agent a path to another system. It does not ensure that the agent calls the tool, because tool and server descriptions influence selection. Even when the call happens, the agent may stop at the first plausible result. Waselnuk describes this as satisfaction of search: an architecture record says to use approach B, so the agent proceeds without finding last night’s Slack conversation saying the team has switched to A. 5:11
Access is therefore not understanding. A patch can compile while remaining operationally wrong because the agent cannot see what lies “below the waterline”: rollout procedures, feature-flag sequencing, incident history, and current decisions that have not reached an architecture document. The concrete consequence is code that passes a local technical threshold and still causes a production incident. 5:43
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a context engine must do before returning an answer
Consider the request, “I want to get auth stood up.” The words alone do not identify the right repositories, conventions, or experts. A context engine can begin with the requester: where they commit, which code they touch, and who reviews their work. That identity narrows the initial search, and the resulting relationships provide pivots into the rest of the organization’s information. 6:12
The engine then has to perform several distinct jobs:
- Unified system context: Search across the relevant systems instead of treating one repository or document store as the whole organization.
- Targeted retrieval: Return a linked document quickly when the task is narrow, while allowing deeper research when discovery is necessary.
- Conflict resolution: Detect that an old architecture diagram and a recent executive Slack thread disagree, then determine which evidence should guide the task rather than silently choosing the first result.
- Personalized relevance: Use who is asking, where they work, and what they are working on to focus retrieval.
- Token optimization: Compress the research into useful context instead of sending every discovered artifact to the model.
- Permission enforcement: Preserve OAuth, SSO, and source permissions so information from a restricted project does not leak into another user’s answer. 6:42
Permission enforcement must happen during retrieval, not after an unrestricted answer has already been assembled. If the requester cannot access “secret project A,” material from that project must not enter the response. Human-facing answers and machine-to-machine responses also need different packaging: a person in Slack may benefit from explanation, while an agent needs a compact packet that avoids spending tokens on presentation. 6:42
What data movement turns scattered workplace records into task-specific context? The engine ingests engineering sources, including real-time incident-management signals, reasons across them using the six capabilities, and emits an answer shaped for the destination workflow. The diagram makes the critical middle layer visible: connecting sources directly to an agent skips identity, conflict, permission, and compression decisions—the very work that turns reachable information into usable context. 7:19
From organizational data to workflow-specific context
Read the diagram as text
- Organizational sources. input — Engineering data and real-time operational signals enter from workplace systems.
- Identity and relevance. engine — The requester’s work, commits, and relationships focus the search.
- Retrieval and conflict resolution. engine — The engine supports fast lookup or deeper research and reconciles contradictory evidence.
- Permission filtering. engine — The requester’s access limits what may enter the response.
- Token-optimized context. engine — Research is compressed and formatted for the destination.
- Human or agent workflow. output — The answer reaches Slack, a coding agent, or another task-specific interface.
- Organizational sources → Identity and relevance: Ingest and focus.
- Identity and relevance → Retrieval and conflict resolution: Narrow the territory.
- Retrieval and conflict resolution → Permission filtering: Filter allowable evidence.
- Permission filtering → Token-optimized context: Shape a safe response.
- Token-optimized context → Human or agent workflow: Deliver in the required form.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The same prompt, with less repeated discovery
Unblocked ran the same prompt against the same model twice, once with context and once without. The run without the engine used about 21 million tokens; the context-assisted run used 10.8 million and finished about two hours sooner. Waselnuk attributes much of the difference to removing repeated exploratory greps at the start of agent sessions: the assisted run began “hydrated” with relevant organizational context instead of rediscovering it. 9:19
This is a speaker-reported comparison of one sizable task; the recording does not supply enough task, harness, or evaluation detail to treat the numbers as a general benchmark. Its useful mechanism is narrower: if multiple sessions repeatedly search for the same architecture, ownership, and business rules, supplying that context up front can remove duplicated search work and leave more of the token budget for the requested change. 9:19
The reported outcome was roughly 50% fewer tokens, faster triage, and better answers because the model had business context. The token result is easiest to explain; the recording asserts the answer-quality improvement but does not provide an independent quality measure. 9:49
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three open-source components expose the underlying techniques
The first tool maps engineering relationships from GitHub activity. It identifies what a person commits, where they commit, and who reviews their work, then distills that activity into an experts graph. Team labels can optionally be added with an OpenAI or Anthropic API key, but the underlying relationship construction is deterministic. This graph gives the earlier auth request a concrete starting point: locate the requester, follow their collaboration and expertise links, and focus research on the parts of the organization where the request is likely to belong. 10:19
The second tool, the repo rules agent, discovers rules files across a repository, checks the extracted instructions and severities, and flags duplicates or other problems. The resulting index is queryable, so an agent can retrieve the rules relevant to its current work instead of loading every instruction file or searching several overlapping conventions independently. 10:49
The third tool demonstrates why semantic retrieval is only part of a context engine. Ask, “What are the open PRs that I worked on in the last week with authentication?” Similarity search alone cannot reliably apply the identity, status, time, and topic constraints. The workshop builds a relational query engine in six stacked pull requests: the agent discovers a schema, writes a query, and runs that query deterministically to retrieve structured records. RAG remains useful for finding semantically related documents; structured queries answer questions whose meaning depends on filters and relationships. 11:19
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Builds a developer collaboration and expertise graph from GitHub pull-request and Git history. Useful for grounding personalized retrieval in reviewers, teams, code areas, and recent working relationships.
Discovers coding-instruction files, extracts and indexes their rules, merges near-duplicates, flags potential conflicts, and supports task-scoped queries instead of injecting every rule into every prompt.
A six-step workshop for turning natural-language questions into validated MongoDB aggregation pipelines over GitHub PR and issue data, including schema discovery, identity resolution, validation, and retry.
Related talks
- Stop babysitting your agents: building a context engine for mergeable code
Waselnuk develops the same context-engine argument with a detailed mergeability comparison, permission model, and MCP-assisted planning demonstration.
- No Vibes Allowed: Solving Hard Problems in Complex Codebases
Dex Horthy presents a complementary workflow for researching and planning changes in established codebases while deliberately controlling agent context.
- Building agents is trivial now, context is the next frontier
Jeff Ng uses an incident-related example to show how unattended agents can miss organizational history even when connected to relevant systems.
Transcript & slides
Loading transcript and slides…