# Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest Drasko Profirovic AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/0RNNfxpdbQk-medic-apache-spark-first-aid-failing-jobs) · [Video](https://www.youtube.com/watch?v=0RNNfxpdbQk) ## Medic for Apache Spark: First Aid for Failing Jobs Pinterest’s Medic evolved from a single prompted agent into a diagnostic system with replayable tests, selective evidence retrieval, and specialized investigators. Basic familiarity with Spark jobs, logs, time-series metrics, and LLM tool calling will help; no prior knowledge of Medic is required. ### A failed job or a looming deadline? [0:26](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=26s) Do you help the team whose Spark job is failing, or unblock another team facing a deadline? For a data-platform support engineer, both requests matter, and neither necessarily has an obvious priority. Partner teams depend on the infrastructure, while troubleshooting a distributed system is especially difficult for people still learning Spark. The support queue keeps arriving regardless of how much attention each investigation needs. ![Motivation slide listing pager and Slack support, disparate data sources, company-specific patches, varying Spark familiarity, and goals of reducing support burden and resolving issues sooner.](https://ai.engineer/api/talk-essay-media/5aa3ed63c84a9dbd94c183ee1ab8ff747488300799b452891992c7d8fa60a4ca) Why diagnosing Spark failures is difficult at Pinterest. Human attention forces a choice. LLM capacity suggested a way to make diagnostic assistance available on demand. Pinterest’s [Medic for Apache Spark](https://ai.engineer/talks/medic-for-apache-spark-first-aid-for-failing-jobs-drasko-profirovic-pinterest), described by Drasko Profirovic, began with a concrete goal: ask why a job failed and receive a research report that establishes the root cause with evidence. Suggested fixes should fit that particular job, and the assistance should be accessible where users already work, including Slack and the Airflow UI. These were requirements for the experience, not just requirements for a model response. ### Give the model access, then give it a method [2:05](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=125s) The first prototype exposed internal data resources through [Model Context Protocol](https://modelcontextprotocol.io/specification/2024-11-05/server/tools), or MCP. With those tools enabled in a conversation, the model could reason about a Spark job. Access alone was useful, but the human operator still had to prompt carefully to steer the investigation. The next step was a single [ReAct agent](https://arxiv.org/abs/2210.03629), combining reasoning with actions that retrieve information. Its prompt specified the problem-solving approach, the structure of the final report, and examples of common failure patterns. That packaged enough diagnostic guidance into the agent to begin beta trials. ### One prompt accumulated too many responsibilities [2:55](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=175s) The beta trials exposed four interacting problems: - **Prompt coupling:** adding detail to improve one behavior could degrade another. Tuning a prompt responsible for everything became unsustainable. - **Inconsistent reports:** some investigations were shallow; others produced too much text. The system lacked controls to keep the agent on track. - **Context exhaustion:** large production log responses consumed the context window and could stop the agent’s reasoning. - **Unrepeatable tests:** end-to-end checks depended on live production data. Once retention removed that data, earlier successes became difficult to reproduce, making regressions hard to distinguish from changes in the evidence. ![Agent shortcomings slide describing growing prompts, difficulty steering reasoning, tool outputs exceeding token limits, and the absence of a regression framework.](https://ai.engineer/api/talk-essay-media/300bfad43bcb37560a15a9a1186fcf3f36ad6c9ad41fc0fa227e8720a8418fbf) Agent shortcomings: prompt tuning, context limits, and manual testing. ### Make investigations inspectable and repeatable [3:53](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=233s) Before making larger architectural changes, the team invested in observability and testing. OpenTelemetry traces went to [Langfuse](https://langfuse.com/docs/observability/sdk/overview), where waterfall views exposed the sequence of steps behind a poor response. A separate end-to-end harness captured production state and paired it with authored offline evaluations, giving prompt changes a repeatable basis for assessment. The harness separated evidence collection from report generation: | Mode | Tool responses come from | Result | |---|---|---| | Record | Real downstream systems | Fixtures saved to disk and checked into code | | Playback | Recorded fixtures | Fresh analysis and a report graded by offline evaluations | **Playback preserves the evidence while rerunning the reasoning.** Production retention no longer determines whether an old case can be tested again. The agent still performs the analysis and generates a report; it does not merely replay a previously accepted answer. ![Testability diagram with Records Mode and Playback Mode. The recording side connects an end-to-end test harness, MCP server, Medic agent, and data sources to fixtures. Playback connects the agent to fixtures and the harness to a report and evaluations.](https://ai.engineer/api/talk-essay-media/d12ec943b557908876222d03841a43a447eba828d2a40e7aeb79fbde8a5050aa) The test harness records fixtures and replays them to evaluate reports. One example evaluation penalized reports containing more than three suggested fixes. This was a verbosity constraint, not a diagnosis-success metric or a universal limit on remediation. A small Python evaluator can express that particular rule: ```python def suggested_fixes_score(report: dict) -> float: fixes = report["suggested_fixes"] return 1.0 if len(fixes) <= 3 else 0.0 ``` The binary scoring choice here simply implements the illustrated penalty; other evaluations would need to assess the substance of the report. As coverage grew, the team could quantify aspects of quality and gain confidence that an improvement had not broken an earlier case. ### Retrieve exception evidence instead of dumping logs [5:32](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=332s) With test coverage in place, the team turned to log handling. A Spark log can contain many benign exceptions, so the last exception is not necessarily the cause of failure. Medic initially used regex heuristics to exclude certain exceptions, but maintaining those rules did not scale well. The replacement was an **exception classifier pipeline** that learned which exceptions commonly appeared in successful jobs and treated them as likely red herrings during later investigations. The pipeline fingerprinted and clustered exceptions, then ranked them by content relevance and how close they occurred to job termination. Recency remained useful evidence, but it was no longer the only reason to investigate an exception. The agent stopped consuming logs directly. Instead, two MCP tools exposed different levels of detail: | Retrieval | Purpose | |---|---| | Top-K truncated exceptions | Inspect a ranked set of candidate signals | | Full logs for a selected exception | Investigate a specific candidate in depth | This made the initial evidence smaller and more selective while preserving a path to detailed inspection. Profirovic reports that the change improved the signal-to-noise ratio and reduced the chance of anchoring an investigation on a misleading exception. ### Turn metric histories into images and return findings [6:47](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=407s) Metrics presented a related context problem. Feeding raw time-series values to a model could work for small inputs, but long-running production jobs made the approach token-inefficient and impractical. Medic moved metrics analysis into a **quarantine sub-agent**: a separate conversation responsible for interpreting the metrics without filling the parent agent’s context. Raw time series became graphs, which were assembled into a dashboard-like image. Annotations called out useful details such as minimum and maximum values. The image was attached to the sub-agent’s conversation, and the model was prompted to reason about patterns. Profirovic reports predictable image-input token usage across job durations in Medic’s implementation; that property depends on the chosen image representation and model processing, rather than applying to arbitrary images. Useful patterns included executor counts dropping to zero or near zero, long plateaus, bottlenecks, and other resource behavior inconsistent with healthy progress. After examining the image, the sub-agent returned a summary of its findings to the parent. The representation controlled the metrics input, while the summary boundary kept the parent’s investigation from accumulating all of the metrics-analysis context. ### Separate prompts, tools, and responsibilities [8:17](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=497s) The next overhaul replaced the single ReAct agent with a multi-agent harness built on [Deep Agents](https://github.com/langchain-ai/deepagents), which uses LangChain building blocks and the LangGraph runtime. Each agent received a dedicated prompt and a subset of MCP tools. The harness also supplied a to-do list and a virtual filesystem to help keep work on track, resembling the working environment of coding tools such as Claude Code and Codex. That is the configuration described in the talk; [current Deep Agents documentation](https://docs.langchain.com/oss/python/deepagents/overview) makes task planning opt-in starting with v0.7. Separating roles also separated maintenance. Developers could change one specialized prompt and use the existing end-to-end harness for focused testing, instead of repeatedly adjusting a single prompt with many competing duties. Profirovic credits this structure with making expansion easier: the team extended Medic to help optimize Spark SQL jobs by adding a new specialized prompt. ### From a request to an evidence-backed remedy [9:28](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=568s) The resulting diagnostic path divides the investigation into explicit responsibilities: 1. **Classify intent.** Decide whether the request needs a simple answer or a deep diagnostic session. 2. **Triage the job.** For a diagnostic session, determine the Spark job’s lifecycle state. If it failed, use the triage agent’s tools to generate failure hypotheses. 3. **Research hypotheses in parallel.** Gather evidence to validate each candidate explanation. 4. **Select a root cause.** Research agents return a score and a root cause; the supervisor chooses the highest-confidence cause. 5. **Propose remediation.** The healer uses runbooks ingested into a vector database to offer remedies grounded in operational guidance. 6. **Assemble the report.** The supervisor combines the results and ensures the final formatting is correct. The healer’s output is a set of proposed remediations. This diagnostic flow ends with a report, not with automatically executing changes against the Spark job. ![Multi-agent architecture slide listing dedicated prompts and MCP tools, supervisor, triage, research and healer roles, and benefits including controlled context, readability, and specialized agents.](https://ai.engineer/api/talk-essay-media/346c845457a5686533dffe029bd123655b0cf649ef780687604ae591c15dffbb) Medic separates supervisor, triage, research, and healer responsibilities. ### Control improved, but fixed workflows proved brittle [10:23](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=623s) Profirovic reports that the multi-agent architecture gave the team its greatest control over system behavior, while improved log handling substantially reduced inaccurate root causes. These are qualitative results: the talk supplies no numerical accuracy or improvement benchmark. More determinism did not automatically improve the system. Pinterest also tried [LangGraph workflows](https://docs.langchain.com/oss/python/langgraph/workflows-agents) to constrain execution, but found that approach brittle compared with the reasoning-and-acting paradigm. That was a result of this diagnostic trial, not a limitation preventing LangGraph from supporting dynamic agents. The useful boundary was between specialized responsibilities, while leaving agents room to investigate the evidence they encountered. The next experiment was to incorporate user feedback from previous sessions so the agent could improve automatically. That remained experimental. Beyond Spark, the team saw an opportunity to apply the same diagnostic pattern to other distributed systems, including Flink and Trino—an expansion of the approach, not implementations demonstrated in the talk. ## Resources - [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629) — The original paper explaining how language models interleave reasoning with actions that gather information from external environments. - [MCP tools specification](https://modelcontextprotocol.io/specification/2024-11-05/server/tools) — The November 2024 specification for exposing, discovering, and invoking tools through Model Context Protocol. - [Deep Agents](https://github.com/langchain-ai/deepagents) — An agent harness with subagent delegation, filesystem tools, and context management, including a runnable quickstart. - [Deep Agents capabilities and configuration](https://docs.langchain.com/oss/python/deepagents/overview) — Current guidance for configuring the harness, including isolated subagents and task planning, which is opt-in starting with v0.7. - [OpenTelemetry-based tracing with Langfuse](https://langfuse.com/docs/observability/sdk/overview) — Current SDK setup and examples for recording application traces and nested observations in Langfuse. - [Image resolution and token budgets](https://platform.claude.com/docs/en/build-with-claude/vision) — Claude's image-processing guidance explains how image dimensions, resizing, and model limits affect token usage and legibility. - [Workflows and agents in LangGraph](https://docs.langchain.com/oss/python/langgraph/workflows-agents) — Examples of predetermined workflows and dynamic agents, including routing, parallelization, and orchestrator-worker patterns. ## Slides ### Medic for Apache Spark Drasko Profirovic introduces Medic, Pinterest’s agentic diagnostics tool for troubleshooting Spark failures. [0:00](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=0s) [Title slide introducing Medic for Apache Spark and speaker Drasko Profirovic, with a QR code and Pinterest imagery.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7a903785029ac70859e24ac5150e55161627f1936649faa9994ae903f7851f3.webp) Medic for Apache Spark First Aid for Failing Jobs Drasko Profirovic Staff Engineer @ Pinterest Interface text: Save ### Agenda Drasko outlines why Pinterest built Medic, its journey from prototype to current architecture, lessons learned, and what comes next. [0:15](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=15s) [Agenda covering motivation, the Medic development journey, lessons learned, and upcoming opportunities.](https://assets.ai.engineer/sites/aiecode2025/sha256/a397cbebcea656c1cd3b783af57b97dde6bd5abb509599c88f791fa406559a99.webp) Agenda Motivation Our Journey building Medic for Apache Spark Lessons learnt Upcoming opportunities ### Motivation Pinterest’s Spark support uses a pager rotation and Slack. Disparate data sources, company-specific patches, and varying familiarity make diagnosis difficult; improving it could reduce KTLO burden and speed remediation. [0:24](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=24s) [Slide describes Spark support, diagnostic difficulties, and the benefits of reducing operational burden and resolving issues sooner.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f3124781bd33aec1532c0f3f1e607225e065017d2e7f3d350f92531bc2844eb.webp) Motivation Initial support for Spark jobs at Pinterest • Pager rotation for critical jobs • Slack channel for less urgent questions What makes diagnosing Spark hard • Disparate data sources to review • Company specific patches • Job owners have varying degrees of familiarity with Spark Business impact by improving the status quo • Reduce KTLO burden • Timelier remediation of issues ### North Star design Pinterest envisioned a Spark diagnostics agent that identifies one primary root cause with cited evidence, recommends prioritized fixes, and meets users in tools such as Slack and the Airflow UI. [1:38](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=98s) [Design goals include evidence-backed root cause identification, actionable remediation guidance, and minimal friction for Spark users.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce9d3e72f7a680851a4841bc56ea2314a20438803fefbf127c3bb92a0c9243d3.webp) North Star design High-confidence root cause identification • Focus on identifying one primary root cause • Evidence cited in the report • Excel with supported use cases, perform decently for less frequent scenarios, and include mechanisms for continuous improvement Opinionated, actionable remediation guidance • Provide prioritized next steps • Tie suggestions to Pinterest’s best practices • Offer ready-to-use code snippets Minimal friction for Spark users • Meet users where they currently operate ### Building the Spark MCP Foundation The MCP server connects Spark job metadata, history, logs, and metrics to LLMs for job analysis. The initial approach worked, but required careful prompting from a human operator. [2:02](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=122s) [Spark MCP server deployment and connections to job metadata, Spark History Server, logs, and metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/31ae29cf63afd34d81ccb9a18e09c23586cd5be5476913d04dc7ea86c6e65da4.webp) Building the Spark MCP Foundation MCP server & deployment • Spark MCP server and K8s production deployment Connecting to data sources • Job Submission Service for job metadata • Spark History Server • Logs ◦ Driver / Executor ◦ K8s Pod • Time series metrics MCP Server SHS JSS Logs Metrics ### Exposed a ReAct Agent Medic placed a ReAct agent inside the MCP server, connected to SHS, JSS, logs, and metrics. A manually tuned prompt guided its problem-solving approach, reports, and handling of common failures. [2:27](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=147s) [A manually tuned prompt guides the ReAct agent's problem solving, report presentation, and handling of common failure patterns; a diagram places it within the MCP server.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5ebc768e399572eabd3a2803daa1f164da19f0607db2133e307d4738cfea00a.webp) Exposed a ReAct Agent Manually tuned generic prompt which outlined • Problem solving approach • How to present the report • How to handle certain common failure patterns, and their suggested fixes MCP Server ReAct Agent (Medic) SHS JSS Logs Metrics ### Agent shortcomings One prompt grew as the team handled edge cases, yet changes could degrade other behavior. Large log outputs exhausted context, and manual testing made regressions hard to detect. [2:57](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=177s) [Slide describing prompt tuning, context window, and manual testing limitations.](https://assets.ai.engineer/sites/aiecode2025/sha256/23b0e2dfa9343f8e422692c319bb509e8ea157511b1ab6e677c5333e537944fd.webp) Agent shortcomings Prompt tuning presented several issues • Unbounded prompt length as we tried to handle edge cases • Difficulty in steering the agent's reasoning and outcomes Limitations with context windows • Tool calls (e.g., logs, metrics) to the MCP would exceed the LLM models' token limits Manual testing lacked confidence • The absence of a regression framework made it impossible to accurately judge the agent's quality across supported scenarios ### Observability and Testability OpenTelemetry traces in Langfuse reveal agent execution steps. An end-to-end harness snapshots production state for replay and uses offline evaluations to grade responses and guide prompt tuning. [3:52](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=232s) [Slide describing tracing and an end-to-end test harness with offline evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/824f93845ac100e38e290d0d32360dc9b4bcbec329e8fc2259d54813b32d282c.webp) Observability and Testability OTel to track traces • Allows us to see the tool invocations and reasoning steps • Used Langfuse Purpose built E2E test harness • Needed a way to snapshot production data for future replay • Used offline evals to grade the Agent’s response based on key metrics • Expanded our corpus of test cases by ○ Identifying production use cases ○ Recording production state ○ Setting expectations in evals ○ Tuning prompts until evals pass ### Testability In record mode, Medic calls real systems and captures responses as fixtures. In playback, it analyzes those fixtures and generates a report, which offline evaluations grade. [4:33](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=273s) [Diagram comparing fixture recording and playback through an end-to-end test harness.](https://assets.ai.engineer/sites/aiecode2025/sha256/84dd406136ad3d1c9e168838d88d6561b3324013d117ba0e5037e4a0b3829b58.webp) Testability Records Mode SHS Fixtures JSS Fixtures Logs Fixtures Metrics Fixtures E2E Test Harness MCP Server ReAct Agent (Medic) SHS JSS Logs Metrics Playback Mode Report Evals E2E Test Harness ReAct Agent (Medic) SHS Fixtures JSS Fixtures Logs Fixtures Metrics Fixtures ### Handling Logs: evaluation results All 7/7 evaluations passed for a diagnosis of a volume column type mismatch. The justification scored 4/5 because it did not explicitly connect the failure to joins or operations on incompatible schemas. [5:06](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=306s) [Evaluation results for a schema mismatch diagnosis and suggested fixes, with seven of seven evaluations passing.](https://assets.ai.engineer/sites/aiecode2025/sha256/3698fa002346cf8bb60e939a4ed220e44949c872b5c1dc77a0309d77bd0ceeed.webp) Handling Logs root_cause_check evaluation: ✅ Score: 5/5 Explanation: The response clearly identifies the expected root cause—schema mismatch or incompatible data types in Spark SQL—by specifying that the 'volume' column was expected as 'bigint' but found as 'INT32'. It provides detailed supporting evidence from exception messages, stack traces, and file paths, and explains why the error occurred. The diagnosis is thorough, with alternative hypotheses considered and ruled out, and actionable fixes are provided. suggested_fix_check evaluation: ✅ Score: 5/5 Explanation: The response provides multiple specific, actionable fixes that directly address schema alignment, including explicit casting, enabling schema merging, and repairing Parquet files for schema consistency. Each fix includes clear implementation steps and validation guidance, fully aligning with the expected fix theme. root_cause_justification_check evaluation: ✅ Score: 4/5 Explanation: The justification provides a clear and technically accurate explanation of a schema mismatch between expected and actual column types in a Parquet file, with supporting evidence and actionable fixes. However, it does not explicitly connect the failure to joins or operations on tables with incompatible schemas. Overall: 7/7 evaluations passed 🎉 All evaluations passed! ### Handling Logs The classifier learns which exceptions commonly appear in successful jobs, filters likely red herrings, and ranks the rest. MCP tools then return top-k truncated logs or full details for an exception. [5:31](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=331s) [Slide outlining log analysis problems, an exception classifier pipeline, and tools for retrieving ranked and detailed logs.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ef9a26e730327eb5e25421c36b181e5f655c36e1006e2481c0c171a75b1854d.webp) Handling Logs Problems • Multiple exceptions per job, often benign • Heuristic if–else rules were brittle & hard to maintain Exception classifier pipeline • Continuous sampling of production jobs • Fingerprinting and clustering of logs • Improve signal to noise ratio by filtering out known red-herrings • Ranking driven by content relevance and time-decay scoring MCP tools • Top-k truncated logs, ordered by their ranking score • Fully detailed logs by exception id ### Handling Metrics A sub-agent analyzes time-series graphs collaged into one image, then returns findings to the parent. The image keeps input token use predictable even for long-running jobs. [6:47](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=407s) [Slide describing a metrics sub-agent and presenting time-series data as graphs.](https://assets.ai.engineer/sites/aiecode2025/sha256/a843d7522ed08b6c57082e8b14a1678c6506fd052eebddda72fe88b41afe5ab4.webp) Handling Metrics • Quarantined analysis of metrics data in sub-agent • Parent Agent provides instructions what observations the sub-agent needs to make with the data • Moved from raw text time series data to encoding the data as a graph, attached as an image to the conversation • Ability to collage multiple time-series datasets into one attached image ### From single Agent to Multi Agent architecture The classifier routes generic queries to an agent and diagnosis requests to triage. Triage routes running jobs to a wait recommendation, successful jobs to optimization suggestions, and failed jobs to diagnosis. [8:21](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=501s) [Flowchart routes a user request through intent classification to triage or a generic agent, then to a response.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8b30ccdd86462859fbbcd5e24ed3a91ca4954fdfd2b01cf6d533c82b177ace0.webp) From single Agent to Multi Agent architecture User Request Job Name Parser - URL Extraction Intent Classifier Diagnosis Request Supervisor Agent - Orchestrator Phase 1: Triage Triage Agent - Job State and Hypotheses job_info Job Info exceptions Exceptions analyze_spark_application_metrics Spark Metrics - Memory, GC, IO Job State? RUNNING Recommend Wait SUCCEEDED Optimization Suggestions FAILED + Hypotheses Failed Job Diagnosis - see Diagram 2 Generic Query Generic React Agent User Response ### From single Agent to Multi Agent architecture Research agents investigate hypotheses in parallel and score the evidence. The supervisor selects the highest-confidence root cause, then a healer agent proposes fixes for the final report. [9:43](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=583s) [Flowchart shows research agents evaluating hypotheses, selecting a root cause, generating fixes, and producing a report.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc3bf75764233761269741281458aa0fd47cbdc9ad10ae9bfcc205f189ae6d2f.webp) From single Agent to Multi Agent architecture FAILED + Hypotheses Phase 3: Investigation Research Agent 1 Hypothesis 1 Research Agent 2 Hypothesis 2 Research Agent 3 Hypothesis 3 Evidence + Confidence Score Evidence + Confidence Score Evidence + Confidence Score Phase 4: Selection Winning Root Cause Highest Confidence Phase 5: Healing Healer Agent Solution Generation ALL 30+ Tools ALL 30+ Tools ALL 30+ Tools ALL 30+ Tools Spark Analysis Tools answer_question_using_spark_documentation Spark Docs Actionable Fixes Up to 3 Phase 6: Final Report Comprehensive Report final_report.md User Response ### Learnings and What’s next The multi-agent design gave the team control, and better log handling reduced inaccurate root causes. Using LangGraph workflows to represent system state proved brittle; the team is looking toward agentic RAG and expansion beyond Spark. [10:24](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=624s) [Lessons include multi-agent tools, better log signals, brittle workflow state, agentic RAG, and extending beyond Spark.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d808ae4da68986ac3634368c5b2db275dee2657fd80b66a459e18b13e52068c.webp) Learnings and What’s next What worked well • Multi-Agent architecture with built-in tools • Investing in ways to boost signal to noise ratio for logs Hard-earned lessons • Using langraph workflows to represent the state of the system is brittle Looking ahead • Moving away from prompt tuning for domain specific knowledge to agentic RAG • Extending the framework beyond Spark ### Thank you The speaker thanks the Medic for Apache Spark contributors and gives special thanks to four others. [11:05](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=665s) [Closing slide lists project contributors and special thanks.](https://assets.ai.engineer/sites/aiecode2025/sha256/86ea64e807448bd287f0480910750808397b3c45a3e01da33b9e31facd8f1e3d.webp) Thank you Project contributors - Frida - Yifei - Vanessa - Tucker - Jingyuan - Drasko Special thanks to - Ang - Zaheen - Chen - Kingsley ## Full transcript [0:01](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=1s) Hi, my name is Drasko Profirovic. I'm a staff engineer at Pinterest. Today, I'll cover Medic for Apache Spark, which is our agentic diagnostics tool built to troubleshoot Spark failures. [0:16](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=16s) We'll dive into why we built the Medic, the journey from prototype to the current architecture, lessons learned along the way, and what's next. [0:26](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=26s) A bit of background about myself. I had the opportunity to work at a few companies under the data platform org. Despite many differences between those companies, there's at least one similarity, the high bar for providing quality support to partner teams who rely on the infrastructure owned by the data platform org. [0:46](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=46s) I'm sure I'm not alone when I say that the support rotation feels like a never-ending stream of questions or problems to resolve. [0:55](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=55s) Moreover, it's easy to forget how difficult it is to troubleshoot Spark or any distributed system for that matter. This is particularly true for anyone just getting started with the framework. [1:07](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=67s) The other challenge with supporting a load-bearing system comes down to ambiguous priorities. Do you focus on helping one team with their failing job, or do you unblock another team with a looming deadline? [1:20](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=80s) It's not always straightforward to rank these asks, but as humans, we often have to decide how we'll spend our time. The same is not true for LLMs. We can easily scale out knowledge and capabilities on demand. [1:36](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=96s) Our vision for a diagnostics agent was to ask it simply, "Why did a job fail?" And get back a deep research document which provides evidence on the root cause of the failure. [1:47](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=107s) The agent would also need to provide suggested fixes that are grounded in the context of the job. [1:54](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=114s) Needless to say, this agent would need to be available on all the surfaces where our users operate today, like Slack or the Airflow UI, to name a few. [2:05](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=125s) We started by exposing our data resources by way of the Model Context Protocol as a way to connect them to the LLMs. At this point, we could start an LLM conversation with the MCP tools enabled and ask the model to reason about our Spark job. [2:22](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=142s) This worked in practice, but it required a lot of careful prompting from the human operator. [2:29](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=149s) We extended our prototype by creating a single reasoning and acting agent, ReAct for short. [2:36](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=156s) The agent was given a single prompt which embodied the problem-solving approach it would take, how to structure the responses as a report, and specific examples to common failure patterns. [2:49](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=169s) At this point, we had enough capabilities to start trialing the solution with our beta users. [2:55](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=175s) From those early trials, we found a lot of shortcomings with our solution. [3:01](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=181s) Prompt tuning became unsustainable. One prompt had to do everything, and adding detail in one area degraded the behavior in another. [3:10](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=190s) Response quality was inconsistent. Sometimes analysis was shallow or other times too verbose. We lacked controls to keep the agent on track, [3:22](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=202s) and we often hit context window issues for production jobs. As an example, large tool outputs from logs would click... quickly consume tokens and brought a halt to the agent's reasoning. [3:35](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=215s) Lastly, our end-to-end testing strategy up to this point relied on manual tests from production. [3:42](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=222s) This felt anecdotal since production data would be retentioned away. Overall, it was hard to know if changes broke earlier wins. [3:53](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=233s) To improve the system, we invested in observability and testability. We used OpenTelemetry to publish traces to Langfuse, and by viewing the agent's execution as a waterfall diagram of steps, we could better understand the cause of lower quality responses. [4:11](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=251s) The reliance on manual end-to-end testing highlighted the need for a more reliable and scalable solution. We built an end-to-end test harness to snapshot production state, [4:23](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=263s) and we could codify expectations as offline evaluations. [4:28](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=268s) Lastly, this allowed us to tune our prompt based on the results. [4:34](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=274s) In practice, the end-to-end test harness is simple. In record mode, the agent calls real downstream systems, and tool responses are captured as fixtures. These are then saved to the file system and checked in as code. [4:49](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=289s) Playback mode, the agent runs against fixtures instead of production data, but this time performs the analysis and generates the report. The test suite then grades the report based on the offline evals we have authored. [5:05](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=305s) For example, an offline eval might check for a limit of three suggested fixes. The eval would score lower if the agent provided too many fixes towards managing the verbosity of the final report. [5:19](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=319s) Our end-to-end tests allowed us to quantify quality instead of relying on intuition, and as we grew our test coverage, we gained confidence that improvements do not introduce regressions. [5:32](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=332s) Once we had the testing coverage in place, we invested deeper in our handling of logs. [5:39](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=339s) Logs are noisy, and many exceptions we see in logs are benign, so simply focusing the last exception may not always be suitable. [5:49](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=349s) Initially, we kept it simple with a heuristics-based approach using regex to filter out certain exceptions, but this didn't scale well. Instead, we built the exception classifier pipeline. The core idea was we would learn which exceptions commonly appear in successful jobs, treat those as likely red herrings, and filter them out in the future analysis. [6:13](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=373s) The agent would fingerprint and cluster exceptions, then rank them based on content relevance and how recently they occurred compared to the termination of the job. [6:23](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=383s) The agent stopped consuming logs directly and instead was given two MCP tools: get the top-K truncated exceptions or get full log details for a specific exception. [6:36](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=396s) This resulted in an improvement to our signal-to-noise ratio and reduced the chance of the LLM to anchor its investigation with a misleading exception. [6:47](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=407s) Like logs, we found that we could improve the overall quality by investing in how we handle metrics. [6:54](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=414s) Raw time-series metrics are not context window friendly. Simply feeding the raw data to an LLM works in the small scale but fails for long-running jobs in production. Not to mention, it's horribly token inefficient. [7:09](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=429s) The approach we took was to perform metrics analysis in a quarantine sub-agent. We would convert the raw time-series data into graphs, which are then collaged into an final image, the appearance of which isn't much different than a Grafana dashboard, albeit with annotations that we found useful, like calling out the min and max values. [7:33](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=453s) The image is then attached to the LLM conversation, and we would prompt the model to reason about patterns in the data. Images worked better because we could guarantee how many input tokens would be used for analyzing any given Spark job, irrespective of its duration. [7:52](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=472s) Examples of useful signals we would be able to get included executors dropping down to zero or near zero, long plateaus or bottlenecks, effectively any resource behavior inconsistent with healthy progress. [8:07](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=487s) Our sub-agent would summarize its findings and return the results back to the parent agent, thereby ensuring the context window is kept healthy. [8:17](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=497s) Lastly, we overhauled our agent harness. We went from a single ReAct agent to a multi-agent architecture. [8:27](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=507s) This was accomplished by building on top of LangGraph's Deep Agent library. Each agent now had a dedicated prompt and a subset of MCP tools. Meanwhile, the Deep Agent library itself provided built-in tools to keep the agent on track, like a to-do list or a virtual filesystem. [8:46](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=526s) This approach mirrors what we've come to expect from our coding tools like Claude Code or Codex, to name a few. [8:53](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=533s) We could finally decompose our single prompt into specialized roles. This refactor provided clearer separation of each agent, and it made it easier for developers to maintain each prompt separately while performing focus testing on the system using our end-to-end test harness. [9:10](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=550s) A pleasant consequence of this architecture was that the effort to expand the scope of the project was as simple as adding a new prompt, and this is how we were able to extend the Medic to also help users optimize their Spark SQL jobs. [9:28](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=568s) Our workflow started... starts with the user's request entering the system, whereupon intent is classified as either requiring a simple answer to a question or a deep diagnostic session. If it's the latter, the triage agent determines the Spark job's lifecycle state, and if it's failed, it'll use a subset of tools to generate a set of failure hypotheses. [9:52](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=592s) Each hypothesis is researched in parallel, where we gather evidence to validate it. [9:59](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=599s) These research agents then return a score and a root cause. The supervisor selects the highest confidence root cause and invokes the healer agent to offer remediations based on runbooks ingested into our vector database. [10:15](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=615s) Lastly, the supervisor agent would assemble the final report, ensuring proper formatting. [10:23](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=623s) The multi-agent architecture proved very effective, offering us the greatest control over the system's behavior. [10:32](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=632s) Enhancements to log handling led to substantial reduction in inaccurate root causes. [10:40](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=640s) We trialed using LangGraph's workflows to make the agent more deterministic, but this approach proved to be brittle compared to the reasoning and acting agent paradigm. [10:50](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=650s) What we're experimenting with now is incorporating user feedback from prior sessions to automatically improve the agent. [10:59](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=659s) Lastly, we see a broader opportunity to apply this pattern to other distributed systems like Flink and Trino. [11:08](https://www.youtube.com/watch?v=0RNNfxpdbQk&t=668s) Medic for Apache Spark project was made possible by the hard work from these contributors. Thank you for your time.