AI Engineer Code 2025

Dashboards Are Dead — Sarah Simionescu, Composio

Loading slides…

Read the talk

Dashboards Are Dead

Sarah Simionescu explains why moving work into an agent requires more than an MCP connection: tools must be discoverable, their dependencies must be clear, and data must move across apps without filling the model’s context.

From a talk by Sarah Simionescu

At a glance

Ideas worth remembering

  • MCP provides communication with a service; tool discovery, prerequisite guidance, and coordination across apps still require interface design.

  • Returning a plan alongside relevant tools helps an agent resolve dependencies, such as finding a Slack channel ID before querying its messages.

  • The cross-app analytics example saves a PostHog cohort outside model context, inspects Metabase’s structure, and generates SQL that uses the saved IDs.

  • Preparing an application for agents means making its operations usable toward a goal, beyond making its website discoverable or adding an AI button.

Six months of Datadog, without learning the dashboard

Sarah Simionescu, whose team is responsible for the Composio dashboard, opens with an awkward discovery. She had used Datadog every day for six months. Yet when a coworker asked to see an alert and she opened its dashboard, she froze: she did not know where anything was. Daily use of the service had stopped requiring familiarity with its visual interface.

Selected presentation frame from Dashboards Are Dead — Sarah Simionescu, Composio at 61 secondsOpen full source frame
A Datadog dashboard is displayed as the speaker describes not knowing where to find an alert.

That experience gives “Dashboards Are Dead” its specific meaning. An agent can become the place where a person asks questions and gets work done, while the applications supplying the data recede from view. The provocation comes with a joke at her own expense: this is great news for the audience, perhaps less comfortable news for someone building a dashboard.

0:120:43

Five interfaces stand between a bug report and a fix

The postmortem begins in 2022, the jokingly named Dark Ages. A Slack message reports an error when searching for prod. Resolving it means reading the complaint in Slack, querying Datadog, checking the session in PostHog, changing code in VS Code, and opening a pull request in GitHub. The task is one investigation; its execution spreads across five tools and five interfaces.

Selected presentation frame from Dashboards Are Dead — Sarah Simionescu, Composio at 91 secondsOpen full source frame
A Slack message reports an error searching for prod.

Each interface also asks the user to translate intent into its own language. Datadog has query syntax, Jira has JQL, and Slack has search modifiers. Even tools that offer SQL can disagree about the dialect. Redesigns add another learning cost, but the deeper burden is expressing the same investigation through several different systems. In this framing, dashboards and query languages are translation devices between a person’s question and the data that could answer it.

In the talk’s 2023 chapter, the “sparkle button” moves some of that translation into an LLM. A user asks a question, and the model writes a query inside the application. Simionescu’s joke about questions requiring more than two database joins captures her skepticism about the reliability of these early features; it is not a demonstrated technical ceiling. The user still visits the dashboard, and the generated query is only sometimes correct.

The next change is where the interaction happens. With Anthropic’s November 2024 announcement of MCP, an agent such as Claude could connect to a service, generate a query, execute it, and return the answer directly. The question no longer had to begin inside that service’s dashboard. But a communication protocol leaves the service responsible for deciding what tools and guidance the agent receives.

1:181:48

MCP opens the doors; the agent still needs a map

Connecting a dozen MCP servers exposes a second interface problem. The agent can reach the applications, but reaching them does not tell it how to complete a task. Simionescu identifies three recurring failures in that setup:

  • Repeated discovery. In the fresh-conversation setup she describes, yesterday’s mistake formatting a Slack link does not teach today’s agent how to do it. Skills can carry instructions forward, but those instructions consume context.
  • Too many tool definitions. Connecting enough servers can load thousands of definitions into the context window. Composio’s GitHub toolkit alone has over 200 tools. The model may select the wrong operation or struggle to determine which prerequisite call must happen first.
  • Disconnected applications. Each server describes its own app. A task spanning multiple apps still needs someone to connect the steps and move the relevant information between them.

The tradeoff is between supplying useful guidance and overwhelming the agent with everything it might need. More tools and more instructions can increase the search burden before any work begins. The talk’s memorable image is an agent standing in thousands of separate rooms: MCP provides entry, while the missing map contains tool selection, prerequisites, and relationships across applications.

3:263:56

Search returns tools and the order in which to use them

The bug-report example returns in a 2026 mock-up. The user pastes a Slack message link into Claude and asks it to use Sentry and Datadog, find the root cause, and create a draft pull request. The mock-up depicts a fix in less than five minutes; that timing illustrates the proposed experience rather than establishing an independently measured debugging result.

Claude first calls Composio Search with three goals: fetch Slack messages, search Sentry issues, and search Datadog logs. Search returns the relevant tools together with a plan for using them. The Slack prerequisite makes the difference concrete: the agent must find the channel ID before querying messages. A tool name alone would leave that dependency for the model to discover.

Once the agent retrieves the complaint, it can investigate Datadog and Sentry in parallel while scanning the codebase. Those steps supply the evidence needed to identify a root cause and propose a code change. The observable change in the workflow is from a human carrying context through several windows to an agent carrying out a coordinated investigation and producing a draft PR. Simionescu says she did not build a custom workflow or write a skill for this task.

The talk then presents early, unreleased comparisons against apps’ native MCP servers listed in the Claude marketplace, using the same tasks and model. Simionescu reports a clear difference, but the supplied explanation gives neither numerical results nor an evaluation protocol, so it supports no quantified advantage. The proposed reason is the interface: Composio translates changing, sparsely documented APIs into operations that agents can discover and use together, building on MCP, CLI, and native tools.

4:545:46

Carry a cohort across apps without carrying every row into context

The analytics example starts with a straightforward question: which verticals do users select during onboarding? Claude uses Composio Search, writes a PostHog query, executes it, and returns the distribution. The follow-up is more interesting: among users who selected e-commerce, which toolkits do they use? Here, “toolkits” means the apps used through Composio. PostHog identifies the cohort, while Metabase supplies the data needed to examine its toolkit usage.

Claude queries PostHog for the e-commerce users’ IDs and saves the results without loading the full list into its context. It then queries Metabase for the database schema and samples data to understand the structure. This separates two needs: the model needs enough information to decide how to query, while the operation needs the actual IDs that define the cohort.

Composio Remote Workbench then dynamically generates SQL containing a regex string that searches for those user IDs, again without placing the entire list in the model’s context. What moves between the systems, and what does the model need to see? The diagram traces the saved cohort separately from the schema and samples used to plan the query.

The result is a view of the most popular toolkits for the e-commerce persona, assembled from both sources in less than a few minutes as described in the demonstration. The important mechanism is the separation of planning from bulk data handling: a saved cohort can participate in a later query without every ID becoming model input. The recording does not explain the regex construction or matching semantics, so it does not establish that this method works for arbitrary identifier formats.

A saved cohort connects PostHog to Metabase

PostHog cohort querycohort-data — Select IDs of users who chose e-commerce during onboarding.
Saved user IDscohort-data — Preserve the query results without loading the full list into model context.
Metabase schema and samplesquery-planning — Inspect how the database is structured before generating the query.
Remote Workbenchquery-planning — Generate SQL with a regex string to search for the saved user IDs.
Toolkit popularity for the cohortCombine the PostHog-selected persona with toolkit data queried through Metabase.
The user IDs remain outside the model’s full context. Schema and sample queries inform the SQL that uses those IDs.
Read the diagram as text
  • PostHog cohort query. cohort-data — Select IDs of users who chose e-commerce during onboarding.
  • Saved user IDs. cohort-data — Preserve the query results without loading the full list into model context.
  • Metabase schema and samples. query-planning — Inspect how the database is structured before generating the query.
  • Remote Workbench. query-planning — Generate SQL with a regex string to search for the saved user IDs.
  • Toolkit popularity for the cohort. Combine the PostHog-selected persona with toolkit data queried through Metabase.
  • PostHog cohort query → Saved user IDs: Save cohort results.
  • Saved user IDs → Remote Workbench: Use IDs without loading the full list into context.
  • Metabase schema and samples → Remote Workbench: Inform query construction.
  • Remote Workbench → Toolkit popularity for the cohort: Query toolkit usage for the selected users.
7:147:44

The application has a new kind of user

The ending turns from individual workflows to product design. Making a landing page friendly to AI discovery does not prepare the application itself for agent use. Composio began by turning popular applications into tools for developers’ agents; Simionescu now describes requests from startups whose clients want to use their services through their own agents.

That user arrives with a goal and a set of tools. In the talk’s phrasing, it has no eyes and will not click the sparkle button. Its useful interface consists of operations it can find, understand, and combine to finish the job. The two demonstrations give that product requirement substance: expose prerequisites for a debugging investigation, and support cross-app data handling without forcing the model to read every intermediate result.

Simionescu’s forecast is that the next era favors products that are easy for agents to use. Her opening moment changes meaning under that forecast. Being lost in Datadog initially felt like falling behind; after months of accomplishing the work elsewhere, it looked more like a preview of how application use could change.

9:009:29

Resources

Transcript download
Full transcript

Loading transcript…