AI Engineer World's Fair 2026
Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest
Read the talk
Medic for Apache Spark: Building an Agent That Can Diagnose Failed Jobs
Pinterest’s Medic evolved from a single prompted agent into a diagnostic system with replayable tests, selective evidence retrieval, and specialized investigators.
From a talk by Drasko Profirovic
Before you start: Basic familiarity with Spark jobs, logs, time-series metrics, and LLM tool calling will help; no prior knowledge of Medic is required.
A failed job or a looming deadline?
Do you help the team whose Spark job is failing, or unblock another team facing a deadline? For a data-platform support engineer, both requests matter, and neither necessarily has an obvious priority. Partner teams depend on the infrastructure, while troubleshooting a distributed system is especially difficult for people still learning Spark. The support queue keeps arriving regardless of how much attention each investigation needs.
Human attention forces a choice. LLM capacity suggested a way to make diagnostic assistance available on demand. Pinterest’s Medic for Apache Spark, described by Drasko Profirovic, began with a concrete goal: ask why a job failed and receive a research report that establishes the root cause with evidence. Suggested fixes should fit that particular job, and the assistance should be accessible where users already work, including Slack and the Airflow UI. These were requirements for the experience, not just requirements for a model response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the model access, then give it a method
The first prototype exposed internal data resources through Model Context Protocol, or MCP. With those tools enabled in a conversation, the model could reason about a Spark job. Access alone was useful, but the human operator still had to prompt carefully to steer the investigation.
The next step was a single ReAct agent, combining reasoning with actions that retrieve information. Its prompt specified the problem-solving approach, the structure of the final report, and examples of common failure patterns. That packaged enough diagnostic guidance into the agent to begin beta trials.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One prompt accumulated too many responsibilities
The beta trials exposed four interacting problems:
- Prompt coupling: adding detail to improve one behavior could degrade another. Tuning a prompt responsible for everything became unsustainable.
- Inconsistent reports: some investigations were shallow; others produced too much text. The system lacked controls to keep the agent on track.
- Context exhaustion: large production log responses consumed the context window and could stop the agent’s reasoning.
- Unrepeatable tests: end-to-end checks depended on live production data. Once retention removed that data, earlier successes became difficult to reproduce, making regressions hard to distinguish from changes in the evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make investigations inspectable and repeatable
Before making larger architectural changes, the team invested in observability and testing. OpenTelemetry traces went to Langfuse, where waterfall views exposed the sequence of steps behind a poor response. A separate end-to-end harness captured production state and paired it with authored offline evaluations, giving prompt changes a repeatable basis for assessment.
The harness separated evidence collection from report generation:
| Mode | Tool responses come from | Result |
|---|---|---|
| Record | Real downstream systems | Fixtures saved to disk and checked into code |
| Playback | Recorded fixtures | Fresh analysis and a report graded by offline evaluations |
Playback preserves the evidence while rerunning the reasoning. Production retention no longer determines whether an old case can be tested again. The agent still performs the analysis and generates a report; it does not merely replay a previously accepted answer.
One example evaluation penalized reports containing more than three suggested fixes. This was a verbosity constraint, not a diagnosis-success metric or a universal limit on remediation. A small Python evaluator can express that particular rule:
python
def suggested_fixes_score(report: dict) -> float:
fixes = report["suggested_fixes"]
return 1.0 if len(fixes) <= 3 else 0.0
The binary scoring choice here simply implements the illustrated penalty; other evaluations would need to assess the substance of the report. As coverage grew, the team could quantify aspects of quality and gain confidence that an improvement had not broken an earlier case.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve exception evidence instead of dumping logs
With test coverage in place, the team turned to log handling. A Spark log can contain many benign exceptions, so the last exception is not necessarily the cause of failure. Medic initially used regex heuristics to exclude certain exceptions, but maintaining those rules did not scale well. The replacement was an exception classifier pipeline that learned which exceptions commonly appeared in successful jobs and treated them as likely red herrings during later investigations.
The pipeline fingerprinted and clustered exceptions, then ranked them by content relevance and how close they occurred to job termination. Recency remained useful evidence, but it was no longer the only reason to investigate an exception.
The agent stopped consuming logs directly. Instead, two MCP tools exposed different levels of detail:
| Retrieval | Purpose |
|---|---|
| Top-K truncated exceptions | Inspect a ranked set of candidate signals |
| Full logs for a selected exception | Investigate a specific candidate in depth |
This made the initial evidence smaller and more selective while preserving a path to detailed inspection. Profirovic reports that the change improved the signal-to-noise ratio and reduced the chance of anchoring an investigation on a misleading exception.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn metric histories into images and return findings
Metrics presented a related context problem. Feeding raw time-series values to a model could work for small inputs, but long-running production jobs made the approach token-inefficient and impractical. Medic moved metrics analysis into a quarantine sub-agent: a separate conversation responsible for interpreting the metrics without filling the parent agent’s context.
Raw time series became graphs, which were assembled into a dashboard-like image. Annotations called out useful details such as minimum and maximum values. The image was attached to the sub-agent’s conversation, and the model was prompted to reason about patterns. Profirovic reports predictable image-input token usage across job durations in Medic’s implementation; that property depends on the chosen image representation and model processing, rather than applying to arbitrary images.
Useful patterns included executor counts dropping to zero or near zero, long plateaus, bottlenecks, and other resource behavior inconsistent with healthy progress. After examining the image, the sub-agent returned a summary of its findings to the parent. The representation controlled the metrics input, while the summary boundary kept the parent’s investigation from accumulating all of the metrics-analysis context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate prompts, tools, and responsibilities
The next overhaul replaced the single ReAct agent with a multi-agent harness built on Deep Agents, which uses LangChain building blocks and the LangGraph runtime. Each agent received a dedicated prompt and a subset of MCP tools. The harness also supplied a to-do list and a virtual filesystem to help keep work on track, resembling the working environment of coding tools such as Claude Code and Codex. That is the configuration described in the talk; current Deep Agents documentation makes task planning opt-in starting with v0.7.
Separating roles also separated maintenance. Developers could change one specialized prompt and use the existing end-to-end harness for focused testing, instead of repeatedly adjusting a single prompt with many competing duties. Profirovic credits this structure with making expansion easier: the team extended Medic to help optimize Spark SQL jobs by adding a new specialized prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a request to an evidence-backed remedy
The resulting diagnostic path divides the investigation into explicit responsibilities:
- Classify intent. Decide whether the request needs a simple answer or a deep diagnostic session.
- Triage the job. For a diagnostic session, determine the Spark job’s lifecycle state. If it failed, use the triage agent’s tools to generate failure hypotheses.
- Research hypotheses in parallel. Gather evidence to validate each candidate explanation.
- Select a root cause. Research agents return a score and a root cause; the supervisor chooses the highest-confidence cause.
- Propose remediation. The healer uses runbooks ingested into a vector database to offer remedies grounded in operational guidance.
- Assemble the report. The supervisor combines the results and ensures the final formatting is correct.
The healer’s output is a set of proposed remediations. This diagnostic flow ends with a report, not with automatically executing changes against the Spark job.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control improved, but fixed workflows proved brittle
Profirovic reports that the multi-agent architecture gave the team its greatest control over system behavior, while improved log handling substantially reduced inaccurate root causes. These are qualitative results: the talk supplies no numerical accuracy or improvement benchmark.
More determinism did not automatically improve the system. Pinterest also tried LangGraph workflows to constrain execution, but found that approach brittle compared with the reasoning-and-acting paradigm. That was a result of this diagnostic trial, not a limitation preventing LangGraph from supporting dynamic agents. The useful boundary was between specialized responsibilities, while leaving agents room to investigate the evidence they encountered.
The next experiment was to incorporate user feedback from previous sessions so the agent could improve automatically. That remained experimental. Beyond Spark, the team saw an opportunity to apply the same diagnostic pattern to other distributed systems, including Flink and Trino—an expansion of the approach, not implementations demonstrated in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original paper explaining how language models interleave reasoning with actions that gather information from external environments.
The November 2024 specification for exposing, discovering, and invoking tools through Model Context Protocol.
An agent harness with subagent delegation, filesystem tools, and context management, including a runnable quickstart.
Current SDK setup and examples for recording application traces and nested observations in Langfuse.
Examples of predetermined workflows and dynamic agents, including routing, parallelization, and orchestrator-worker patterns.
Further reading
Claude's image-processing guidance explains how image dimensions, resizing, and model limits affect token usage and legibility.
Updates since the talk
Current guidance for configuring the harness, including isolated subagents and task planning, which is opt-in starting with v0.7.
Read the complete timestamped transcript
- 0:01
Hi, my name is Drasko Profirovic. I'm a staff engineer at Pinterest. Today, I'll cover Medic for Apache Spark, which is our agentic diagnostics tool built to troubleshoot Spark failures.
- 0:16
We'll dive into why we built the Medic, the journey from prototype to the current architecture, lessons learned along the way, and what's next.
- 0:26
A bit of background about myself. I had the opportunity to work at a few companies under the data platform org. Despite many differences between those companies, there's at least one similarity, the high bar for providing quality support to partner teams who rely on the infrastructure owned by the data platform org.
- 0:46
I'm sure I'm not alone when I say that the support rotation feels like a never-ending stream of questions or problems to resolve.
- 0:55
Moreover, it's easy to forget how difficult it is to troubleshoot Spark or any distributed system for that matter. This is particularly true for anyone just getting started with the framework.
- 1:07
The other challenge with supporting a load-bearing system comes down to ambiguous priorities. Do you focus on helping one team with their failing job, or do you unblock another team with a looming deadline?
- 1:20
It's not always straightforward to rank these asks, but as humans, we often have to decide how we'll spend our time. The same is not true for LLMs. We can easily scale out knowledge and capabilities on demand.
- 1:36
Our vision for a diagnostics agent was to ask it simply, "Why did a job fail?" And get back a deep research document which provides evidence on the root cause of the failure.
- 1:47
The agent would also need to provide suggested fixes that are grounded in the context of the job.
- 1:54
Needless to say, this agent would need to be available on all the surfaces where our users operate today, like Slack or the Airflow UI, to name a few.
- 2:05
We started by exposing our data resources by way of the Model Context Protocol as a way to connect them to the LLMs. At this point, we could start an LLM conversation with the MCP tools enabled and ask the model to reason about our Spark job.
- 2:22
This worked in practice, but it required a lot of careful prompting from the human operator.
- 2:29
We extended our prototype by creating a single reasoning and acting agent, ReAct for short.
- 2:36
The agent was given a single prompt which embodied the problem-solving approach it would take, how to structure the responses as a report, and specific examples to common failure patterns.
- 2:49
At this point, we had enough capabilities to start trialing the solution with our beta users.
- 2:55
From those early trials, we found a lot of shortcomings with our solution.
- 3:01
Prompt tuning became unsustainable. One prompt had to do everything, and adding detail in one area degraded the behavior in another.
- 3:10
Response quality was inconsistent. Sometimes analysis was shallow or other times too verbose. We lacked controls to keep the agent on track,
- 3:22
and we often hit context window issues for production jobs. As an example, large tool outputs from logs would click... quickly consume tokens and brought a halt to the agent's reasoning.
- 3:35
Lastly, our end-to-end testing strategy up to this point relied on manual tests from production.
- 3:42
This felt anecdotal since production data would be retentioned away. Overall, it was hard to know if changes broke earlier wins.
- 3:53
To improve the system, we invested in observability and testability. We used OpenTelemetry to publish traces to Langfuse, and by viewing the agent's execution as a waterfall diagram of steps, we could better understand the cause of lower quality responses.
- 4:11
The reliance on manual end-to-end testing highlighted the need for a more reliable and scalable solution. We built an end-to-end test harness to snapshot production state,
- 4:23
and we could codify expectations as offline evaluations.
- 4:28
Lastly, this allowed us to tune our prompt based on the results.
- 4:34
In practice, the end-to-end test harness is simple. In record mode, the agent calls real downstream systems, and tool responses are captured as fixtures. These are then saved to the file system and checked in as code.
- 4:49
Playback mode, the agent runs against fixtures instead of production data, but this time performs the analysis and generates the report. The test suite then grades the report based on the offline evals we have authored.
- 5:05
For example, an offline eval might check for a limit of three suggested fixes. The eval would score lower if the agent provided too many fixes towards managing the verbosity of the final report.
- 5:19
Our end-to-end tests allowed us to quantify quality instead of relying on intuition, and as we grew our test coverage, we gained confidence that improvements do not introduce regressions.
- 5:32
Once we had the testing coverage in place, we invested deeper in our handling of logs.
- 5:39
Logs are noisy, and many exceptions we see in logs are benign, so simply focusing the last exception may not always be suitable.
- 5:49
Initially, we kept it simple with a heuristics-based approach using regex to filter out certain exceptions, but this didn't scale well. Instead, we built the exception classifier pipeline. The core idea was we would learn which exceptions commonly appear in successful jobs, treat those as likely red herrings, and filter them out in the future analysis.
- 6:13
The agent would fingerprint and cluster exceptions, then rank them based on content relevance and how recently they occurred compared to the termination of the job.
- 6:23
The agent stopped consuming logs directly and instead was given two MCP tools: get the top-K truncated exceptions or get full log details for a specific exception.
- 6:36
This resulted in an improvement to our signal-to-noise ratio and reduced the chance of the LLM to anchor its investigation with a misleading exception.
- 6:47
Like logs, we found that we could improve the overall quality by investing in how we handle metrics.
- 6:54
Raw time-series metrics are not context window friendly. Simply feeding the raw data to an LLM works in the small scale but fails for long-running jobs in production. Not to mention, it's horribly token inefficient.
- 7:09
The approach we took was to perform metrics analysis in a quarantine sub-agent. We would convert the raw time-series data into graphs, which are then collaged into an final image, the appearance of which isn't much different than a Grafana dashboard, albeit with annotations that we found useful, like calling out the min and max values.
- 7:33
The image is then attached to the LLM conversation, and we would prompt the model to reason about patterns in the data. Images worked better because we could guarantee how many input tokens would be used for analyzing any given Spark job, irrespective of its duration.
- 7:52
Examples of useful signals we would be able to get included executors dropping down to zero or near zero, long plateaus or bottlenecks, effectively any resource behavior inconsistent with healthy progress.
- 8:07
Our sub-agent would summarize its findings and return the results back to the parent agent, thereby ensuring the context window is kept healthy.
- 8:17
Lastly, we overhauled our agent harness. We went from a single ReAct agent with multi-agent architecture.
- 8:27
This was accomplished by building on top of LangGraph's Deep Agent library. Each agent now had a dedicated prompt and a subset of MCP tools. Meanwhile, the Deep Agent library itself provided built-in tools to keep the agent on track, like a to-do list or a virtual filesystem.
- 8:46
This approach mirrors what we've come to expect from our coding tools like Cloud Code or Codex, to name a few.
- 8:53
We could finally decompose our single prompt into specialized roles. This refactor provided clearer separation of each agent, and it made it easier for developers to maintain each prompt separately while performing focus testing on the system using our end-to-end test harness.
- 9:10
A pleasant consequence of this architecture was that the effort to expand the scope of the project was as simple as adding a new prompt, and this is how we were able to extend the Medic to also help users optimize their Spark SQL jobs.
- 9:28
Our workflow started... starts with the user's request entering the system, whereupon intent is classified as either requiring a simple answer to a question or a deep diagnostic session. If it's the latter, the triage agent determines the Spark job's lifecycle state, and if it's failed, it'll use a subset of tools to generate a set of failure hypotheses.
- 9:52
Each hypothesis is researched in parallel, where we gather evidence to validate it.
- 9:59
These research agents then return a score and a root cause. The supervisor selects the highest confidence root cause and invokes the healer agent to offer remediations based on runbooks ingested into our vector database.
- 10:15
Lastly, the supervisor agent would assemble the final report, ensuring proper formatting.
- 10:23
The multi-agent architecture proved very effective, offering us the greatest control over the system's behavior.
- 10:32
Enhancements to log handling led to substantial reduction in inaccurate root causes.
- 10:40
We trialed using LangGraph's workflows to make the agent more deterministic, but this approach proved to be brittle compared to the reasoning and acting agent paradigm.
- 10:50
What we're experimenting with now is incorporating user feedback from prior sessions to automatically improve the agent.
- 10:59
Lastly, we see a broader opportunity to apply this pattern to other distributed systems like Flink and Trino.
- 11:08
Medic for Apache Spark project was made possible by the hard work from these contributors. Thank you for your time.