AI Engineer Europe 2026
Why More Context Makes Your Agent Dumber and What to Do About It
Read the talk
Why More Context Makes Your Agent Dumber—and What to Do About It
Larger context windows do not guarantee better reviews. Reliable agents need selected evidence, bounded exploration, specialist tasks, and a judge that distinguishes learned preferences from mandatory rules.
From a talk by Nupur Sharma
Before you start: Familiarity with LLM prompts, tool-calling agents, and pull-request reviews will help you follow the architecture.
When more inputs create more failure modes
When a pipeline crashes, you can inspect the failure and fix it. What do you fix when an agent keeps searching, changes its interpretation, or produces an answer that misses the task? Nupur Sharma approaches that question from DevSecOps and her work on agentic code reviews at Qodo: moving from deterministic pipelines to agents changes both the failure modes and the engineering needed to contain them.
Early static prompts made the input-selection problem obvious. With a 4K context window, developers had to decide what mattered before asking the model to act. A poor selection meant a poor answer. Larger windows made more inputs possible, and tool-equipped agents could search documents, act, then search again. But that introduced a different problem: without a stopping criterion, the agent could keep requesting more information instead of completing the task.
Adding agents expands the available expertise but also the coordination burden. A security agent looks for vulnerabilities, a review agent evaluates code, and a coding agent proposes fixes. Their interpretations can clash. More context, tools, and agents increase capability without automatically producing a coherent result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Capacity is not the same as usable context
A model may accept a large input without using every part equally well. Sharma describes a U-shaped pattern: the initial instructions and final inputs remain prominent, while information in the middle receives less reliable attention. The practical concern is neglected evidence, not literal deletion of tokens from the context window.
An audience member asks how she knows. Sharma points to internal code-review benchmarking: task-specific agents receive context, including experiments with an entire codebase, and the team checks whether that context affects the result. She reports that the initial goal and final input remain salient while intermediate material—such as Jira information and context available through MCPs—can be overlooked. The talk provides no named models, evaluation protocol, or quantitative results for that comparison.
Context selection remains an engineering responsibility even when context capacity stops being the immediate constraint. Instead of giving the model everything and expecting it to discover what matters, build a process that selects useful evidence for the task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a context strategy you can maintain
A context engine acts as a gatekeeper. For a large, messy codebase, it builds search and ranking logic so that a task retrieves material judged relevant before the model starts working. Sharma characterizes the initial indexing effort as moderate but reports mapping and indexing slowdowns when dealing with 600–700 repositories. She gives no repository sizes or latency measurements, so this is an operational warning rather than a fixed scaling limit. Maintaining that infrastructure may be disproportionate for a team whose main product is not the context engine itself.
Two alternatives make the codebase navigable through different structures:
- Hierarchical summarization: Create summaries for files and folders, then let an agent use those summaries to decide where to inspect the underlying code. The expense is both initial and ongoing: new or changed files require fresh LLM processing and updated mappings.
- Knowledge graphs: Represent dependency chains, such as one file affecting another that affects a third. A graph database is especially useful for complex logic and dependencies spanning repositories, but requires substantial developer setup and hosting.
The distinction is what the agent needs to navigate: descriptions of content, or explicit relationships between pieces of the system.
For teams building agents for their own processes, Sharma favors iterative retrieval as a lower-setup starting point. Give the agent a topic index—a library card—rather than a full summary of everything. It checks whether a topic is relevant, then retrieves the associated code for deeper inspection. Retrieval still costs money, but developers do not have to construct as much infrastructure upfront.
A critic node addresses a different failure: the agent has produced something, but it has drifted from the original goal. The critic compares the output with that goal and can request another attempt. This requires relatively little initial developer input, but repeated runs add latency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stop researching the method and finish the task
Context selection does not solve the orchestration paradox: an increasingly capable model can spend its effort deciding how to solve a problem instead of solving it. It selects a tool, questions that choice, researches another method, and repeats. Tokens go into method selection while the task remains unfinished. Sharma uses the then-current Opus, without specifying a version, as an example of this repeated self-challenging behavior.
Her team's response is an 80/20 hybrid approach: allow broad exploration for the research portion, then constrain final validation and summarization. The percentages express a design heuristic, not a measured optimum. The final stage has prescribed behavior: given a particular result, produce the required output rather than reopening the entire search for a method.
The exploratory portion still needs a boundary. Sharma gives two organizational examples:
- Iteration limit: After four or five cycles, proceed with the latest result.
- Time limit: After five minutes, proceed with the latest tool choice or decision, then revisit it if the resulting output is poor.
These limits force a transition from exploration to execution; they do not establish that the latest answer is correct.
| Stage | Work | Model role |
|---|---|---|
| Exploration | Discovery, planning, tool choice | High-reasoning model |
| Finalization | Validation, goal checks, summarization | More constrained model task |
The research model decides what to investigate. The finalization stage turns the collected material into the requested result. Sharma argues that this second stage need not use the most capable reasoning model because its instructions are narrower. The workflow can enforce hard gates, but a lighter model's judgment is not thereby guaranteed to be deterministic.
The same division applies between individual actions. Deciding what to investigate next belongs to the exploratory side. Once the results are available, deciding how to assemble the user's requested output belongs to the constrained side. Separating those responsibilities prevents every synthesis step from becoming another open-ended research task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specialists need a judge
A large context window also tempts teams to give one agent every responsibility. Testing and reviewing code may use overlapping material, but they remain distinct tasks. Sharma illustrates the risk with an agent assigned four tasks that delivers strong results for two while losing the other two along the way. A mixture of agents instead assigns small issue experts a specific job.
Specialization creates a reconciliation problem. Imagine separate agents finding the best hotel, destination, and flights for a vacation. One returns a hotel in Greece; another returns a flight from Amsterdam to Portugal. Each recommendation may look plausible in isolation, but together they do not describe a usable trip.
A judge agent receives the specialist outputs and checks whether they make sense together. Its job goes beyond collecting answers: it must turn local recommendations into one coherent result. The specialists supply depth; the judge supplies a shared decision about what belongs in the final answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From pull-request context to a filtered review
Qodo applies this pattern to pull-request reviews. The pipeline separates gathering evidence from producing findings:
- Collect context. A context collector gathers material from the PR, the context engine, and available tools.
- Partition the evidence. Rather than immediately generating a review, it distributes selected material to specialist agents.
- Produce candidate findings. Specialists investigate concerns such as security flaws, code differences, and Jira issues.
- Judge relevance. The judge examines the returned findings and can revisit the context engine and the PR before deciding which belong in the review.
In Sharma's example, ten candidate findings are inputs to this refinement process, not ten comments that must all reach the developer.
The first implementation question is how those agents communicate. Does the system exchange files, or use proprietary machinery? Sharma identifies LangChain as underlying infrastructure. The handoff she describes is straightforward: collect an agent's results and use them to construct the next agent's prompt. When several outputs need combining, another agent can collect and refine them before the next handoff. The linked documentation describes current subagent patterns; the talk does not identify a particular LangChain API or version.
A small Python prompt builder makes that handoff concrete. It retains the review goal, associates each finding with its specialist, and asks the next agent to evaluate the combined candidates. It constructs a prompt; it does not itself decide which findings are valid.
python
import json
def build_judge_prompt(goal: str, results: dict[str, list[str]]) -> str:
candidates = [
{"specialist": specialist, "finding": finding}
for specialist, findings in results.items()
for finding in findings
]
return (
f"Review goal: {goal}\n\n"
"Evaluate the candidate findings together. "
"Keep findings relevant to the goal and identify conflicts.\n\n"
f"Candidate findings:\n{json.dumps(candidates, indent=2)}"
)
prompt = build_judge_prompt(
"Review the pull request for relevant security and code issues.",
{
"security": ["A changed file contains a hardcoded API key."],
"code-review": ["The change follows an existing repository pattern."],
},
)
The meaningful boundary is the conversion from separate outputs into an explicit next task. Merely concatenating results does not perform the judge's reconciliation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Teach each agent what matters locally
A second question concerns calibration: how does an agent learn what counts as a good or bad review finding? The same Java framework can serve healthcare, retail, and finance organizations with different priorities. Framework knowledge alone does not tell the agent what matters to a particular team.
Sharma describes indexing PR history and retrieving earlier changes similar to the current submission. That history enters the workflow twice: first when specialists receive their context, and again when the judge evaluates their recommendations. If specialists return fifteen recommendations, the judge can consult previous reviewer and developer comments to decide which are worth presenting.
This calibration applies to every agent, but it does not mean every agent receives the same complete history. The context engine selects the subset relevant to each specialist's task. Shared organizational knowledge and identical agent prompts are different things.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Selective context still needs a system-wide perspective
The audience pushes on the cost of that separation. A code-quality agent and a framework-specific reviewer may each operate correctly on their own inputs without seeing the full architectural picture. Checking linting or the presence of tests is relatively local. Architectural decisions require weighing concerns, including security, against one another. Where does that broader judgment happen?
Sharma answers through the structure of human reviews. A senior engineer knows the codebase and its packages. A security expert checks for issues such as hardcoded API keys and SQL injection. An auditor concerned with ISO or SOC 2 compliance may ask whether changes are logged. These reviewers contribute specialized knowledge rather than duplicating one another's entire perspective.
The agent equivalent needs explicit organizational inputs. Sharma describes a web portal where architects and compliance staff supply guidelines that agents can check. The context collector holds the broader collected picture, then routes relevant portions to specialists. This makes the selection step consequential: an agent cannot validate a requirement that never reaches its context.
Supplying those documents is not universally required. Organizations can use the system out of the box, but without guidance they should not expect findings specific to their working practices unless relevant PR history is available. When an audience member questions whether history is the best source, Sharma qualifies it as one source among several. Past reviews can supply evidence about a team; they are not a complete specification of how that team should work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn preferences without learning away the rules
The remaining question is how to balance new engineering principles against historical behavior. Sharma describes several weighting signals: the importance assigned in the compliance portal, whether an item is an error or a recommendation, and whether developers accept the resulting suggestions. Accepted suggestions gain weight; unaccepted suggestions lose weight. The talk identifies these signals but does not provide a numerical formula or a complete procedure for resolving conflicts between them.
Two sources of behavioral evidence feed that process: acceptance of current recommendations, and whether similar findings in past PRs were actually implemented. But those signals can encode bad habits. Sharma recalls arguing with a developer accustomed to hardcoding API keys: an established practice does not make the practice safe.
The audience makes the objection explicit: historical behavior does not establish correctness. Sharma's final distinction is between explicit rules and feedback-sensitive bug findings. If an organization supplies a rule, violations remain highlighted regardless of whether developers want that feedback. For a bug finding, repeated reviewer disagreement and non-implementation may reduce its weight; her example of ten rejections illustrates that behavior rather than specifying a product threshold.
That boundary determines what adaptation is allowed to change. Developer feedback can influence which bug findings receive attention, while explicit rules remain obligations. A review system needs both forms of knowledge: evidence about what the team tends to accept, and requirements that acceptance history must not erase.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current implementation guidance for coordinating subagents and controlling their input and output context.
Further reading
Original research on how evidence position affects multi-document question answering and key-value retrieval.
Qodo's release explanation of specialist review agents, judge filtering, and pull-request history as context.
Updates since the talk
A May 2026 account of controlled PR defects, precision and recall, ensemble evaluation, and per-agent debugging.
- Qodo's Context EngineArticle
An August 2026 explanation of bounded context packages and the separate roles of context, rules, reasoning, and memory.
Read the complete timestamped transcript
- 0:00
[upbeat music] I'm Nupur.
- 0:15
I work with Qodo. Uh, at Qodo, we do agentic reviews. Uh, I have a background in DevSecOps, so I'm coming from an industry where everything was deterministic. The pipelines, they run, they crash.
- 0:29
If they crash, we fix them. Uh, to a place where we are doing agents, where nothing is deterministic. So, in my last few years, I have learned where and how agents fail, what are the learnings, and today I'll be sharing some of my learnings with you.
- 0:52
So, um, if you see the evolution of agents, it started with static prompts where it was a four K context window, and we tried to put whatever was important or whatever we deemed important, and the AI models will process it and provide you with the results, right?
- 1:12
When we started with that, that means that it was on us to tell LLMs what do they should, uh, they should look into. That means if we provide wrong inputs, we might not get proper results.
- 1:24
And then we thought maybe if the context window grows, if the context size grows, we can do better, we can have more inputs. And, uh, we started with agentic workflows.
- 1:34
So we created an agent. We get that... get them tools like search tool to go into search into documents and do something, uh, as a command. Then again, look into the search and do something, which again created kind of a loop where the tool does not know where to stop.
- 1:51
It thinks like, "I need more inputs." Again, going back and back. It's a loop.
- 1:58
To improvise on that, uh, nowadays multi-agents is becoming more popular. Create multi-agents, do a lot of stuff together. When we see it like that, we have a lot of agents working for you.
- 2:11
So a security agent trying to figure security concerns, a review agent trying to review the tool, a coding agent trying to fix things. Now, again, the more the tools, the more issues you have.
- 2:24
Not every agent understands, and they have clash in their understandings where you don't get into the results.
- 2:34
So, what do we learn from here? What, uh, we see is context is not a problem. Day by day, the models are coming where you can dump a lot of context, a lot of data.
- 2:47
But does that make sure that the results you are getting is smart enough to give you everything or smart enough to decide what's important? If you see the current LLM models, we see a pattern where it takes the f- initial inputs you provide, it takes the last inputs, but the in-between context is basically removed.
- 3:11
So they don't focus on the in-between context. Agents look at the starting point, end point, and try to provide you the results. This is like a U-curve, where some of the things from the start, some of the things from the end make sense, but whatever you are providing in between that, that is not taken up.
- 3:29
Yeah.
- 3:30
How do you know this?
- 3:32
This is something which we are working on, and we are actually benchmarking things. So when we create agents, we try to see this is the context we provide to the agents.
- 3:40
Does it take this context into effect and also give us the results? So, we are working with multi-agentic architecture, where for each of the tasks we do, do for code reviews, we give the task to an agent and say, "Okay, give us the result."
- 3:56
Now, every time we... for example, code reviews, we try to see can we give all the context? Can we give the whole code base, for example, and see if we can get the results.
- 4:05
But we see that, that whenever we start working with that, the initial prompt or the initial goal which we start with, that is in focus. If we give something at the end as an input, that is in focus.
- 4:16
But all between context, like I have Jira, I have MCPs, can you look into that? The LLMs try to get rid of those things and push them to give...
- 4:26
make sense by themself. So, to s- have this or to make a way out of this, how we deal with is creating strategic solution for context optimization.
- 4:42
Rather than dumping everything down to the models and asking them to be smart enough to find out what is more important, uh, we usually try to see, okay, what we can do to make it a better context for the model.
- 4:55
There are s- lots of solutions in place if you see currently, and context engine is a buzzword. Like, everybody wants to create context engine and everybody wants to, uh, provide that.
- 5:07
But context engine is like a bouncer, right? So your high-speed car is going, and it acts as a bouncer and tells you, "This is more important." Now, if you have a large messy code base, it makes sense to create a context engine because it creates a search pattern, it create a ranking logic, so that whenever you ask
- 5:28
for a task, it looks for those rankings and say, "This is more important for you. Take it and work with it." The problem is the indexing part takes moderate effort, but the scaling is a challenge.
- 5:41
Like, if you start talking about six hundred repositories or seven hundred repositories, the mapping and the indexing starts to slow down, and it becomes, again, unpredictable to find or create a context engine if you are not actually into making context engine only.
- 5:59
There are- Lots of areas where agent can get more context instead of investing highly on context engine. Hierarchical summarization, where instead of creating or going through everything, a summary is created for each file and folder, so that when the agents try to find, they can try to read the summary and see if that is more important to
- 6:22
us or not, can be a good one. The only thing is that you need a lot of LLM processing. So every time a file is created or changed, some of the agents need to go and create a mapping for that.
- 6:34
So it's a high upfront, um, context on, uh, LLM processing that is needed. Another way is knowledge graph. Now, knowledge graph is complex, but it works wonder when you have logical dependencies.
- 6:49
For example, you have one file which impacts another file, which again impacts other file. You can create a graph DB hosting. It is the initial input needed by the developer is very high.
- 7:02
It takes a lot of time to create that. But if you have complex logics or you have mul- dependencies on multiple repos, that works wonder.
- 7:11
For me, I think for most of the task, if you're not a product company, but if you're building agents for yourself or your processes, iterative retrieval works really s- good.
- 7:22
Because instead of even creating a summary, it creates kind of an index. So it's like a library card, which you give to your agents and say, "This is the topic.
- 7:31
If that is relatable to you, you can look deep into the code, uh, and, uh, look for the results." Again, it has quite, uh, cost impacts, but, uh, you do not have to invest a lot of energy.
- 7:45
The, uh, input re- uh, required by the developers to provide to the LLMs is low, and it provides better results. There is also option of self-correction, where you ask, uh, the LLMs to do something, and there is a critic node which looks and say if that is relevant to your initial goal or not.
- 8:06
In that case, if the context is lost, uh, you can again ask the agent to do it again, retry it again, because the critic node said this is not the right way.
- 8:16
It takes a little bit more time because it ta- adds a latency of running the agents and again, but it does not require a lot of input initially from the developers to create something.
- 8:31
Another challenge, uh, which I have seen people getting into when they create, uh, these, um, agents is the Orchestration paradox. Now, what it does is that now LLMs are becoming more and more smart.
- 8:47
So when you give them the task, they think like, "Okay, I should use this tool. Uh, maybe I can do better. I should research more on what should I use."
- 8:54
It goes into a loop that instead of actually s- looking into solve the problem, they look for the method to solve the problem. They hop on from one method to one...
- 9:04
another method, and most of the API tokens are wasted on finding a way to do it rather than doing it. So you will just go on the research mode.
- 9:15
For example, if you use Opus latest and greatest, they will try to see what is the best method to do it and challenging themself again and again, maybe not this, another way, another way, and it just goes into a loop of doing s- trying to do something rather than doing something.
- 9:34
To s- resolve this, uh, we worked with 80/20 hybrid approach. I think this is one of the most interesting, uh, outcomes I have seen or the way to en- resolve this infinite loop.
- 9:49
What our teams are doing is giving the latest and the greatest models or giving the agents power to research 80% of the time. So you give them the goal and say, "Okay, try to do whatever you can."
- 10:02
But the 20% of the task where you need final validation, you want summarization, that are not something which is free-flowing. That is more restricted. Those are more hard gates.
- 10:14
For example, if I get X results, I want Y. It's more deterministic so that the research which is coming from the 80% can be lowered down. Now, when you see, you can always say that the 80% tool can still go on and go into infinite loops.
- 10:32
We have mechanisms to work on that. For some organization do, they c- they do counter mechanism, where after four or five counters, you have to work with whatever was the last results.
- 10:45
For some of them, they have timeout counters, and after five minutes, whatever is the last tool or whatever is the last decision, you work with that and then go back if the results are not good.
- 10:55
But you can restrict that 80%. But in short, if you are using anything like discovery or you're trying to see which tool to use, you're trying to plan, those 80%, uh, research models are really good.
- 11:10
But if you are again trying to create a summarization, you're trying to see, okay, this is the research I have got, now I have to make a result out of it, the 20% works really well.
- 11:21
Now, for 80%, usually you use, uh, high reasoning models, latest and the greatest, but you don't need a, a high reasoning model for the 20% because those 20% things are doing deterministic.
- 11:34
They are-- You are actually telling them what is needed. For example, the critic node which we talked about. They don't need to research. They don't need to find out what is the best thing to do.
- 11:44
They just need to see what was your goal, what was the result you are trying to achieve, and, uh, how to provide or how to summarize that for that.
- 11:52
Also, things like if you think about what would be the next possible action. I have this result. What should I go and look for? That are things can be done by the 80% uh, dynamic models.
- 12:04
Whereas I have all the results from the 80% models, but what is the proper way or what is the proper, uh, results the g- uh, the user is looking into?
- 12:15
Those kind of precisions can be taken by the 20% model.
- 12:22
Finally, uh, this is again an interesting failure which we have seen, where as the context grows, teams think, "Okay, we can do everything with one agent because the context window is quite great, right?
- 12:35
We can put everything. We can ask an agent to, uh, do, uh, the testing part. We can ask the da- agent to do review part. We can ask the agent all the kind of things because the context is same, and they can provide us the results."
- 12:48
That make sure that the... when the agent is going forward, it get overwhelmed with the inputs. And again, it tries to start losing, uh, what was the original task.
- 12:59
So maybe you give four tasks to the agent and somewhere down the line, it focuses on two tasks, so you get great results for the two tasks, but the other two just, just get lost in the middle.
- 13:10
For those particular purpose, we have something called mixture of agents, and that's, that's where you hear a lot of buzz about multiple agents or multi-agent tech architecture, where you create instead of one big agent, we create issue expert agents.
- 13:28
We create small, small agents which are doing great in a, a specific task which they have provided. Now, to build on top of that, each of the agent come up with their own interesting ideas or results.
- 13:42
How to make sure those results combine and make sense together? Because, for example, I am trying to search for a vacation. I give an agent to find the best hotel, another agent to find the best location, another agent try to find best flights.
- 13:59
But all three of them gives me different result. The, uh, hotel is in Greece. Uh, the flight is from Amsterdam to, um, maybe Portugal, and everything just doesn't make sense, right?
- 14:12
So for that particular purpose, there's a concept called a judge agent. What it does it, it tries to get all the results and see if they can make sense together.
- 14:22
So now you are doing all the greatest things from different agent, getting the best results from their part. But a judge agent help us to combine these and make one sense out of it instead of getting so many things which doesn't make sense together.
- 14:41
Something similar is implemented by us. So this is our architecture, uh, Qodo's architecture, where for the code reviews, we are using the same, uh, formula. So as part of a PR review, we have a context collector which actually goes and collect context from the PRs.
- 15:01
It go collect context from the context engine. It, uh, co-collect context from the tools. But then it does not start working and, uh, giving you the reviews. It actually bifurcates all the context it has provided and pass it on to different agents.
- 15:17
Now, what these agents do is basically specializing in what they are supposed to. For example, there will be a security agent trying to find security flaws. There might be a agent try to code.
- 15:29
There might, uh, code differences. There might be an agent trying to find the Jira issues. Once all these ir- agents give us back, a judge agent actually looks for the results and say, "Okay, these are interesting enough, but is it relevant to you?"
- 15:44
We can again go back in the context engine, look into the PRs and see out of the 10 things which is provided by you, how many of them actually make sense for your thing.
- 15:54
So again, refining the results, uh, to make sense to you.
- 16:01
Yeah, I think that was it from my side. Uh. [clapping] [laughs]
- 16:08
Any questions?
- 16:10
Yes. Um, in practice-
- 16:13
Mm-hmm
- 16:13
... how do you let the swarm communicate with each other?
- 16:17
Uh, you are talking about the agents?
- 16:20
So you have agent A and agent B and the judge. I can imagine they write to a file system, or do you have some kind of tool proprietary or?
- 16:28
Uh, we ch- we use, uh, LangChain at the bottom, uh, and that is being used to communicate and build infrastructure for, uh, different agents.
- 16:38
Do you know what LangChain uses for that? Like, just collects the responses and then shoves it back into the prompt of the next agent?
- 16:47
Yes. Yes. Yes. Yes. That- that's what. So what we do is we try to cr- uh, get the results and create a prompt for the next agent. And if it's multiple things, again, there is an agent just to collect the results and create a better prompt, uh, which is refined for the next agent.
- 17:05
Okay.
- 17:06
Have you thought about a calibration step for each agent?
- 17:10
When you say the calibration, can you t- tell me more-
- 17:12
The calibration, right? So you... When you do a code review with an agent, right? You need at least, what I heard today multiple times is doing some kind of calibration, right?
- 17:25
That you actually tell that what is good and what is bad.
- 17:28
Yes. So when you say it like that, uh, and let me know if that makes sense for your question or not. We do calibration in a form. We check what we have as a context.
- 17:40
So for example, uh, when we get the code reviews, LLM does not know what is important for you or how do you work. So for example, an LLM when they gets input, they get input from healthcare industry, they get from retail industry, they get from finance industry.
- 17:57
And all of them can use same Java framework in different ways. Uh, different, uh, things are important for them. The rest of them doesn't make sense. So what we do is we give you two different options to tell the agents how to perform or what to work on.
- 18:15
One, uh, part is we give them the PR history. So we index all your PR and see when was the last time something like this was identified and compare the current sub-version, if that is-
- 18:30
Yes, yes, yes, yes. We do the... So the changes we, uh, you make to the code, we look into if we can see something similar in the past. That again is, uh, transferred to the context twice.
- 18:42
First is when we are actually giving context to the sub-agents to find things for you, and another time to the judge agent, so that when I get fifteen different recommendations for your code review, my judge agent can look into what was there before, how did your reviewer commented, how your developers commented, and based upon that decide if
- 19:03
that is worth providing to your developers or not. And the other part is-
- 19:07
And this happens for every agent?
- 19:09
For every agent, yes.
- 19:10
Okay. And, and if I understood you right, you don't share the context between the agents, right? So you have the, the co- a specific context for every agent.
- 19:22
Yes. We are trying to resolve that new part that instead of dumping everything to the LLM, we, uh, take the part which is more important. We use a context engine for that.
- 19:33
We take the part which is more important and only provide that particular part to that particular agent.
- 19:39
Sorry, one more follow-up.
- 19:40
Yeah, sure.
- 19:41
Um, but I... But then for me it's not clear how you bridge the gap, right? Let's say you have an, uh, code quality agent, and you have a, I don't know, uh, um, what else agent, um, uh, uh, framework-specific coding review agent, right?
- 20:03
And then you basically, you only, as I understood-
- 20:08
Mm-hmm
- 20:09
... um, you only share the specific information to each agent, right? And then basically each agent runs atomic, autonomously-
- 20:18
Yeah
- 20:18
... right? Um, and doesn't have the full picture, right? And at least when, let's say as a human, right, and if you do code reviews, it's always good if you have a full prospect, right?
- 20:33
I, I would say, like, that kind of methodology would... it works for simple things like does it use linting? Does it use, I don't know, i- are tests implemented?
- 20:43
But when you think about, or at least I think when you think about the overall architecture, for example, to make architecture decisions, um, that covers security because everything is a balance and a, and a-
- 20:57
Yeah
- 20:57
... and you have to weigh that somehow. So how, how do you solve that then?
- 21:02
Yes. So I think if you look into the older v- version of code reviews, you should ha- you used to have a senior engineer who knows your code, who knows what kind of packages you are using, and they can, uh, comment on if the developer has done something, uh, similar to what you're used to or did something,
- 21:20
uh, totally weird, right? Then you used to have a, a security person who used to see if you are providing all kinds of security. You are not hard coding your APIs, or you are not, uh, putting any SQL injections.
- 21:33
So all those kind of things the security experts know. On the other hand, if you are working with ISO compliance or SOC 2 compliance, there might be an auditor who might ask the, uh, team lead or the senior engineer, "Is your code, uh, being, um, you know, logged?
- 21:50
I-Is it logging the changes?" and so on. So previously as well, there were lots of people having specialized knowledge looking into those kind of areas specifically. Now, when the context is provided, it's always like, these are my security concerns which I always have to look into.
- 22:05
These are my architectural concerns. An architect might look from the architecture per- perspective. We can do that something similar, uh, with the agents as well because, for example, architecture security concerns, we have a web portal where architects can provide their guidelines for, uh, co- compliance people can provide their guidelines, and an agent can look into all those
- 22:25
guidelines and say, "Is it validated or not?" So if you see the
- 22:31
initial, uh, picture, the context collector knows everything, and then it provides relevant context to the agents.
- 22:39
So you, you enforce basically your customer to upload that, these kind of documents, right? Is that a kind of a requirement then? Or-
- 22:48
It-
- 22:49
Because, I mean, the system will have completely, let's say, different result, right, if you don't share them.
- 22:55
Exactly. So that's something which depends upon organization. There are some people who say, "We don't work with any rules or regulations, so just give me out of the box."
- 23:04
That also means that don't expect the agents to find something very specific to your working until and unless we have certain PR history. Because then again, the PR history kicks in and tells you what is relevant even then when you don't provide.
- 23:19
I'm not sure if the, if the PR history is really the best source, right? It-- I think it-
- 23:26
It can be one. It can be one of... It can-
- 23:29
Yeah.
- 23:29
So that's why there are various, uh, sources, right? So it's PR history, it's your reso- it's your... [phone message alert]
- 23:35
And somebody's cooking food at my home. [laughs] Yeah. But it, it can be, uh, one of the source. And that's why-
- 23:42
Yeah. And my question is like... Yeah, I mean, of course it can, it can be, right?
- 23:45
Yeah.
- 23:46
But at the end you need to decide, let's say also with jury, right, how much you weight, right? So like the new documents for the, let's say, the engineering principles, architecture principles, and compare, compare them to, let's say, your, your merge request-
- 24:04
Yeah, yeah, yeah
- 24:05
... right away, right? But they can be completely out of the balance, right?
- 24:09
Um, it depends. It depends. So again, uh, I think it's, if you look into from one perspective, it's difficult to decide. But if you're getting the context from many angles, for example, PR history, that's one part.
- 24:22
But when you do the compliance and you tell in the compliance portal, this is really important. So we have various, uh, segments of it. It's an error, it's an, uh, a recommendation.
- 24:32
All those kind of things adds weight to a feedback to say if it's good or not. And every time your developer expe- uh, accepts a suggestion, it gets more weighted for the next one.
- 24:44
If it does not, uh, uh, accept the suggestion, it gets a less weight. So it's all about indexing and making sure those weights are managed somewhere.
- 24:53
How do you weight them?
- 24:54
Uh, two ways. One is, uh, by when you, when you give your recommendations, does your developer actually accept it or not? We, uh, index that.
- 25:03
Right.
- 25:03
Another way is, uh, from the past PRs, we try to find out similar issues identified and if your developer actually implemented them. So for example, some people are used to hard coding their, uh, API keys, and I literally had a tough, um, argument with the developer.
- 25:21
But this is how we do it. No, this should, this should not be the way.
- 25:24
And then-
- 25:24
Yeah, but-
- 25:25
That's, uh, and I think [phone message alert] it nicely matches also what I meant, right? So if you look in the, in, in the history-
- 25:32
Yeah
- 25:32
... that happened doesn't mean it's, it's good.
- 25:34
It's g- Yeah. A-and, and that's the way where the system tries to tell you, "This is not good. This is not good." And then it's up to you and-
- 25:41
But only if you provide the guidance, right? Or-
- 25:44
No. It, uh, so there is something called bug fixes, and there is something called rules. So if you provide them as a rule, it will get, uh, highlighted, doesn't matter if you want it or not.
- 25:55
And then there are bugs where agent try to tell you there's something wrong, and if the reviewer also agrees with it and did not implement, uh, 10 times, the reviewer might get it less weighted and give you.
- 26:06
Okay. Yeah. Interesting. Cool. Thank you so much.
- 26:11
Thank you. [outro music]