AI Engineer World's Fair 2026
Enterprise Agents Have a Structure Problem - Ishita Daga, Tesla
Read the talk
Enterprise Agents Have a Structure Problem
Better data-agent answers depend on knowing which source to trust, keeping definitions current, and choosing the metric the requesting team actually means.
From a talk by Ishita Daga
Before you start: Familiarity with SQL queries, business metrics, and agents that retrieve organizational data will help.
A bad answer does not always call for a bigger model
A data agent gives a bad answer. Should you give it a bigger model, a longer context window, or more knowledge? The usual additions—Markdown files, documents, MCP servers, and plugins—can all help. But adding capacity and information does not tell the agent which information deserves authority. Ishita Daga’s starting point is that better answers require structure around the model, not just more material inside its context.
Three different failures can hide behind the same incorrect answer:
- Ambiguity: The agent does not know which table, column, data source, or knowledge base to use. Several sources may contain relevant information without identifying which is authoritative, which is cleanest, and which allows more flexibility.
- Staleness: Decisions, KPI definitions, and processes change, while the context supplied to the agent remains unchanged.
- Preference: An individual or team expects a particular metric definition, query, or set of filters—even when a canonical query already exists.
These failures need different remedies. Source selection needs an ordering; changing knowledge needs a lifecycle; team preferences need a way to connect the requester with the intended calculation. Daga treats that last problem as especially open-ended.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with the most curated source
An agent should not weight every knowledge base equally. Daga proposes a source-of-truth hierarchy that begins with the cleanest, least flexible source and moves toward more dynamic sources as the question requires. This makes authority part of the retrieval decision rather than leaving the model to reconcile an undifferentiated collection of documents and schemas.
The first layer is a semantic layer: a curated collection of queries, KPI definitions, metric calculations, and business meanings. The agent finds the closest relevant KPI and uses its associated definition and query information to answer. This constrains interpretation before query generation begins.
The second layer is a set of canonical, parameterized queries. Daga initially calls this layer canonical tables, then describes reusable query patterns. The agent chooses an appropriate pattern and has more latitude to adjust parameters, filters, or the query itself. That flexibility lets it answer questions beyond the exact metrics already represented in the semantic layer.
The third layer is a database graph. Tables connect to their columns and to the metrics they can answer, with relationships available in both directions. An agent can use these connections to find the data needed for a question and construct a query over it. The expanded reach comes with substantial work: the graph must first be built, then maintained as tables, columns, and definitions change.
| Layer | What the agent uses | Main tradeoff |
|---|---|---|
| Semantic layer | Curated KPI definitions and queries | Strong constraints, narrow flexibility |
| Canonical queries | Parameterized query patterns | More freedom over queries and filters |
| Database graph | Metric–table–column relationships | Broad flexibility, difficult maintenance |
The hierarchy is an order of approach: use the curated definition when it fits, then move toward sources that allow more construction and interpretation.
Daga estimates that the semantic layer and canonical queries can solve about 80% of problems, with the database graph addressing the remaining 20%. This is a prioritization estimate, not a reported accuracy benchmark. Her implementation recommendation follows from the maintenance tradeoff: build the first two layers first, then add the graph for questions they cannot cover.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give context a maintenance loop
Even a well-ordered source hierarchy can give confidently outdated answers. Processes change, definitions are deprecated, and manually maintained .md files or skills become difficult to keep current. Daga’s response is a context lifecycle with two components: maintained knowledge sources and a feedback loop.
A live source, in this framework, is one that people regularly update, review, and curate. Examples include GitHub, CRM tools, Tableau, and the dbt Semantic Layer. The useful property is not simply that the agent has a connector. It is that the connected source has an ongoing process for keeping its information current.
The feedback loop begins with corrections that might otherwise disappear into a conversation. A user may report that the database is wrong, a definition is outdated, a metric has a new calculation, or a query needs a different filter. Each of these is an event worth capturing. Logging makes the correction available for a context update rather than limiting its effect to one answer.
Those events also need evaluation. Daga describes two complementary approaches:
- Human evaluation: Curate questions with human annotations or have people assess the agent’s answers.
- Automated evaluation: Revisit questions asked over the last few days and compare newly generated answers with their actual answers.
The recurring process is to log corrections, evaluate behavior, and update context. Without evaluation, a team cannot see whether its agent is improving or how its performance changes over time. Refreshing context becomes useful operational work when the team can assess what the refresh did to the answers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One milestone question, two valid calculations
Preference creates a harder problem: the source can be authoritative and current, yet still represent the wrong interpretation for the requester. Different teams may calculate the same named metric differently or apply different filters to the same query. Daga illustrates this with two teams asking for the average time taken for a milestone.
Team A measures from completion of the previous milestone to completion of the current milestone. Team B measures from the start of the current milestone to the start of the next milestone. Both are valid in the example, but they measure different intervals.
| Team | Interval | Slide result |
|---|---|---|
| A | Previous completion → current completion | 18.4 days |
| B | Current start → next start | 23.1 days |
The slide’s values illustrate the consequence of choosing different endpoints; they are not an agent-performance comparison.
The distinction can be made explicit in SQL. For an illustrative PostgreSQL table named milestone_instances, with one row per milestone instance and timestamp columns for the relevant endpoints, the two definitions become separate expressions:
sql
SELECT
AVG(
EXTRACT(EPOCH FROM (current_completed_at - previous_completed_at))
/ 86400.0
) AS team_a_average_days,
AVG(
EXTRACT(EPOCH FROM (next_started_at - current_started_at))
/ 86400.0
) AS team_b_average_days
FROM milestone_instances;
Naming both calculations makes their difference inspectable. It does not tell the agent which one a user intends when asking for average milestone time. The missing input is the requester’s intended definition, not another timestamp or a larger context window. Daga sees resolving that intent as an open problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect the requester to the right definition
One approach Daga is trying is to store every alternative calculation in the semantic layer. The user can then explicitly tell the agent which definition to use. For the milestone example, that would mean selecting either the completion-to-completion calculation or the start-to-start calculation. The catalog can represent both meanings, but explicit selection does not itself preserve the user’s preference. On the next underspecified question, the agent faces ambiguity again.
The other approach is agent memory, such as Mem0 or a memory.md file. Daga’s concern is that storing a preference is not the same as selecting the correct metric in context. A memory can retain what a user prefers without establishing the distinction between two business definitions or when each should apply. Her critique concerns that selection step, not the absence of user-specific memory.
The desired behavior is to route the agent to the appropriate metric based on the individual or team using it. In the milestone example, the shared agent would need to connect the requester with the relevant interval definition rather than repeatedly require a fully specified question. Daga presents this as a research direction, not a demonstrated routing implementation.
This extends the engineering task beyond organizing knowledge. An enterprise agent needs a structure for choosing sources, a process for maintaining and evaluating context, and a way to incorporate individual preferences into shared behavior. Daga’s closing aspiration is a shared data agent that can retain those individual meanings without losing the organization’s common knowledge. The difficult part is making the same agent answer appropriately for different people using the same words.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Documentation for defining shared business metrics, handling joins through MetricFlow, and deploying metrics to downstream tools.
Examples for storing preferences and retrieving memories scoped to users, agents, applications, and sessions.
Further reading
Connection instructions and limitations for consuming semantic models in Tableau, including how local overrides affect inherited definitions.
Read the complete timestamped transcript
- 0:01
Hello everyone, and welcome to my talk, Enterprise Agents Have a Structural Problem. A quick introduction about myself. I'm Ishita Daga. I work as a machine learning engineer in Tesla, and I'm building enterprise agents for our organization.
- 0:19
Before we get into the framework and the solution space, I actually want to talk about why these agents fail in the first place. So when an agent gives a bad answer, the first re-reflex that we have is that we need a bigger model.
- 0:34
We need, um, the latest model. We want a model with a lot of context so that it can hold a lot of information, or we need to add a lot of knowledge base in terms of .MD files, documents, or MCP servers, plugins, et cetera.
- 0:52
But while all of these are fair solutions, they're not the answer to actually improving the data agent itself.
- 1:01
Model... The agent actually has these three main problems.
- 1:10
One is ambiguity. So it doesn't know what table is right, what column is right, which data source or which knowledge base to access when, which one holds the source of truth, or like the cleanest information, which one is just a bit more flexible.
- 1:26
So we have a, a lot of these knowledge bases, but it's not defined which one is the best, um, or which one to use when. So ambiguity, I feel, is one big problem.
- 1:41
Second is staleness. We all know context updates so quickly. Um, a lot of decisions change, a lot of definition, KPIs, they change so often. Processes, they get updated so often.
- 1:54
But to keep our data agents accurate, we need to update this context, this context lifecycle, and that's a second part of the problem. The last one is preference, which is more so capturing an individual or a team preference on what metric to use to answer a specific question,
- 2:19
how to use a query, or what filters to use in a query.
- 2:25
Although we have like a canonical query, let's say, each team has a different preference on what filters or what definitions to use. So this, I feel, is a very open-ended problem, still needs a lot of research.
- 2:39
A lot of, um, industries or the frontier labs are actually working on solving this one.
- 2:46
So let's start with the first one, ambiguity. The agent needs to understand what source of truth to use or what knowledge base to use. It can't weight all the knowledge bases equally.
- 2:59
There has to be a bit more structure to the one it chooses to answer a particular question.
- 3:06
In my opinion, the source of truth actually is a hierarchy which goes from the cleanest, least flexible source of truth to the messiest, uh, but most flexible, most dynamic source of truth.
- 3:19
And the way to answer any question is to start from the cleanest to the messiest or the most dynamic one. We have divided in-- the sources of truth in three different buckets, uh, for this framework.
- 3:32
The first one being the semantic layer, which is the best source of truth, a very curated list of all the different queries, KPI definitions, metric, uh, or how to calculate a metric, um, the business definitions, and everything combined into just one semantic layer, which the model can use to answer a query.
- 3:56
So the agent just looks into the semantic layer, finds the closest KPI, and references all of these different data points from that KPI to actually give you an answer.
- 4:08
The second part is canonical tables. These are more so the parametric tables or, um, queries, sorry,
- 4:16
where you give the agent all of a list of different parametric queries, and it sort of tries to understand which ones can it use to actually answer your question.
- 4:28
It gives it a bit more flexibility in choosing or writing its own query or its own filters, et cetera, and
- 4:37
can answer a bit more wider set of questions.
- 4:42
The last one is the database graph, which I feel is the most trickiest because it takes a lot of effort, but it gives you a lot of flexibility in the kind of questions that can be answered.
- 4:54
So the idea here is you connect each table to all the different columns, uh, each table to the metric that can be answered through this table and vice versa, and sort of create this,
- 5:09
uh, enormous database graph that can be served to the agent, and it can answer or query on top of it. While it is tricky, it is also very difficult to maintain or update.
- 5:25
So I feel if an enterprise is approaching or adding sources of truth, they should start adding the first and the second layer first. These are easy to set up, solve, I think, 80% of the problems, and then the 20% can be solved by the database graph.
- 5:43
Moving on to our second problem, which is staleness. I think the word speaks for itself. But the context gets rotten, or it gets deprecated, or processes change so often that it's hard to maintain .MD files or keep on updating your skills with the most latest context.
- 6:05
It's just not easy, and we need a better solution for that.
- 6:11
My answer to this problem is creating a context lifecycle, which basically has two different components. The first one being embedding live
- 6:21
data sources or knowledge sources. By live, I mean something th- or the data sources that are going to be always updated, would be reviewed and well-curated, and always provide the
- 6:36
newest or the most up-to-date data. This can include your GitHub, your, um, CRM tools, or your Tableau or dbt semantic layers. Whatever you want to include can be there, but it has to be a mandatory data source, like a, a data source that is updated most often.
- 6:57
The second part is a feedback loop, which is something that a lot of
- 7:03
enterprise or data agents actually miss. The idea here is you should be able to log each and every, uh, event. And by event, I mean whenever someone says that this database is incorrect, or this definition is, definition is incorrect or outdated, or there's a new way to calculate a metric, or the f- there is a new filter
- 7:24
that you need to use. All of these events need to be captured, logged, and
- 7:30
used to update your data agent context. Once you have captured these events, the second part is to actually evaluate your data agent, and you can do that by either curating an evaluation suite with, um, human-annotated questions or human, uh, evaluations.
- 7:53
Or you can create an automated evaluation where you see all the questions that have been asked in the last couple of days and the actual answers and how close the new answers are to the actually, the, the actual answers.
- 8:07
So I think there are a lot of ways you can do that, but a lot of teams do not focus on evaluation that much, and which is why the agents actually fail so often because you don't know how the agent is progressing or, or how is the performance.
- 8:22
You don't keep a track of that. So I feel creating this feedback loop or context loop where you log, evaluate, update the context every o- every so often is very important.
- 8:37
The last problem is preference, which is a very subjective problem because preferences, they change very often in terms of what kind of metric should be used, what's the right calculation.
- 8:51
Different teams will calculate the same metric differently or, um, use the same query but different filters. So there is a lot of subjectivity which needs to be captured, but it's very hard to do that on individual or team level.
- 9:07
To give an example, um, so let's say there is team A and team B, and they do-- both want to calculate the average time, uh, taken for a particular milestone.
- 9:21
Team A calculates it by looking at the previous milestone, so completion of the previous milestone to completion of the current milestone. While team B actually looks at the start of the current milestone to the start of the next milestone.
- 9:34
So while both of these are correct metrics or the correct way to calculate a metric, they both will give you very different answers, and it's just about preference. We just need to understand what an individual mean when they say:
- 9:51
What's the average milestone time? So while this is a great question, I feel the industry still does not have a correct answer or a correct way to solve the problem itself.
- 10:05
I feel like there are sort of two ways to solve it. One is a semantic layer, which is something that I'm trying, wherein you store all the different ways to calculate a metric as a semantic layer, and then the individual sort of can prompt the model which one do they want to use.
- 10:22
So it-- you can solve it, but still is not storing the preference of the user. And the second thing is you, again, bump into the first challenge, which is ambiguity.
- 10:33
The agent doesn't know which metric is the right one. You still have to prompt it. Second part is agent memory, which is your Memzero or like storing a memory.md file, but it still is not the best way.
- 10:49
Uh, it stores your preference, but it can't understand the distinction between two different metrics, um, or which one to use when. So I, I feel there are ways to solve it, but none of the ways are actually solving the preference problem itself.
- 11:06
What we actually want is a way to route the agent to the right metric based on who
- 11:14
or what, what team or who, which individual is using that agent. So still an open-ended problem, but yeah, I think something we should work on or research on a bit more.
- 11:30
Which brings us to the end of this presentation. So just to conclude, I feel there are these three big problems: ambiguity, staleness, and preference. And what we really need to solve for is a better structure, a better way to manage the context and evaluate the context.
- 11:49
And then preference, which is such an open-ended problem. It requires not just understanding how the agent works, but also embedding an individual preference, like creating a hive mind for your data agent.
- 12:04
That's about it. Thank you so much.