Using RL-based Agent to Detect and Remediate ETL Pipeline Failures

Imagine you are this engineer. A production data job failed hours ago. The dashboard went stale. You have spent all day checking the logs, the schema, and the upstream data, and now it is past midnight.

The same question keeps coming back. What changed? The failure itself may be small, but expensive part is everything around it. Inspection, diagnosis, choosing a safe response, rerunning the job, and confirming that we did not make the data worse.

Hi, I'm Anna Marie Benzon. In this talk, I will show an RL-guided system that selects bounded remediation action for ETL failures. The central question is not simply whether an agent can act, but whether it can act usefully, explainably, and within boundaries that an operations team would actually trust.

Cloud ETL failures are rarely arrived as one clean, well-labeled exemption. We see late or unavailable sources, schema drift, date-time incompatibilities, null rate spikes, type changes, and runtime errors that do not match anything in the runbook.

The usual response is a human workflow. Inspect the logs, form a diagnosis, attempt a repair, rerun the job, and validate the output. Each step is reasonable. The latency comes from handoffs, incomplete context, and the need to avoid an un-unsafe fix.

In the capstone evaluation, the manual recovery baseline was modeled at roughly two point five working days. This represents an incident moving through normal queuing, investigation, and approval. So the engineering objective is specific.

Compress that loop for routine, recognizable failures while escalating the cases that are uncertain, novel, or high risk.

This diagram shows the end-to-end AWS architecture from my capstone. An existing AWS Glue ETL job emits a job failed event. Amazon EventBridge catches that event and triggers the Lambda function that runs the agent.

Lambda gathers evidence from two read-only sources. CloudWatch provides the error logs, while the Glue Data Catalog provides the current schema metadata. The system uses those signals to classify the failure, assess the data quality and operational risk, and construct the state passed to the RL decision engine.

The policy then proposes a bounded response. The safety layer checks that proposal before the executor can use the Glue API to re-trigger the job or apply an approved remediation.

Amazon S3 stores agent artifacts, audit logs, and quarantined outputs.

Finally, the job is rerun and validated. So this is closed operational loop. Monitor, diagnose, score, decide,

check safety, act, and verify recovery. The capstone implementation uses synthetic data provided by the client. The public repository preserves this pattern through a sanitized, generalized deployment template. The intelligence layer deliberately separates three concerns.

Deterministic anomaly rules establish observable facts. A field disappeared, a type changed, or the null rate crossed a threshold. The Q-learning policy handles contextual action selection. Given the current incin-incident state, should the system retry, coerce the schema, roll back, quarantine, escalate, or simply log the event?

Then a safety override sits outside the learned policy. For example, if the anomaly is critical and the policy proposes a passive action such as logging, the override converts that choice into an escalation.

This separation is the design thesis of the project. Rules for facts, learning for bounded choices, and guardrails for authority. Before selecting an action, the system has to establish what actually happened.

The schema profiler extracts structure, types, nesting, and null rate statistics. The drift detector compares the current profiler with the baseline a-and identifies additions, removals, and type changes. The data quality analyzer checks completeness, validity, and consistency.

The error classifier maps log patterns into failure families, and the risk scorer turns those signals into an operational risk level. These components are deterministic by design. For directly observable data conditions, an explicit rule is easier to validate, explain, and audit than an opaque inference.

With richer and representative incident history, some classifiers could become learned components. But ML-ready is not the same as ML required. The simplest reliable component should own each decision. The policy receives a compact state, failure category, risk level, retry count, drift severity, and data quality condition.

It then selects from six actions: retry, coerce, rollback, quarantine, escalate, or log. I use tabular Q-learning because the state and action spaces are small. The Q table is cheap to evaluate, and every decision can be inspected directly.

For this state, these were action values, and this action won. Technically, each incident is modeled as single-step contextual decision implemented with tabular Q-learning rather than as a long horizontal control task.

That formulation is deliberate. The system needs to choose one safe operational response from a bounded action set.

The value of the learned policy here is not sophistication for its own sake

It is a structured way to learn action preferences from outcomes while retaining a decision service that an engineer can spec.

The learned policy does not have final authority. It proposes an action. The safety layer evaluates the proposal against the anomaly severity and the system's operational constraints. Passive actions are overridden for critical conditions, and high-risk or unknown cases are escalated.

Every proposal, override, execution result, and validation outcome is written to an audit record. Notice that escalation is included in the action space. That's not agent giving up. It is the system correctly recognizing the boundary of its evidence or authority.

For an operational agent, the ability to say, "I should not do this automatically," is a capability. If success is measured only by non-escalation, the optimization target is wrong. Here is one failure path.

The agent receives a Glue-style job failure event. The log classifier detects a date-time format incompatibility with zero point nine confidence. Based on the encoded state, the policy proposes schema coercion.

The safety overrides does not fire because this is not classified as a critical anomaly. But the executor then discover, discovers that automatic coercion is not available for this specific case.

The system does not pretend that the fix happened. It records the proposed action, reports that execution was unavailable, and sends the incident for manual review.

This example shows two distinct controls: policy safety and implementation capability. An action can be safe in principle and still be unavailable in the current environment. A robust agent must represent both conditions explicitly.

To make the work independently reviewable without exposing, uh, the client context, I built a sanitized public benchmark around a generalized AWS Lambda-style architecture. The capstone implementation used client-provided synthetic data.

The public repository uses newly generalized synthetic schemas, records, logs, and incident scenarios. It contains no client documents, infrastructure identifiers, or business-specific values. I ran four controlled experiment groups and repeated the robustness evaluation across thirty seeds from forty-two through seventy-one.

The reported aggregates include ninety-five percent confidence intervals. This preserves the system design and experimental logic in a form that other engineers can inspect and rerun while maintaining the confidenti- confidentiality boundary.

On the controlled benchmark, the rule-based anomaly detector achieved precision of one, recall of zero point eight, and an F1 score of zero point eight eight nine. That means the detector was conservative.

The anomalies it flagged were correct in this benchmark, but it still missed some positive cases. For operations, that distinction matters. Perfect precision does not mean perfect detection. For cases where our-- where the RL-guided workflow resolved the incident successfully, mean resolution time was about five point twenty-four minutes.

Across the thirty runs, the simulated success rate was seventy-four point sixty-three percent, plus or minus one point fifty-one percentage points. The non-escalation rate was eighty-eight point sixty-three percent, plus or minus zero point eighty-nine points.

The chart compares that minute scale result with the modeled manual baseline of two and a half working days, or that is two hundred sixteen thousand seconds. Within the benchmark, that is approximately a ninety-nine point eighty-five percent reduction in MTTR.

These figures quantify performance within the controlled benchmark. Within that scope, they show that the architecture can automate the fast path for known failure conditions. Production validation is the next evaluation boundary.

The ablation results are, in my view, the most useful part of the project. The RL policy matched the equivalent deterministic policy, a difference of zero percentage points within a zero point nineteen point confidence interval.

On this compact state space, the learned policy maintained the same success level as the hand-defined policy. By contrast, the deterministic action selection beat random selection by fifteen point sixty-three points.

And enabling the safety override reduced non-escalation by about fifteen point zero three points. That decrease is intentional. The guarded system escalates more often when autonom-autonomy would be inappropriate. So where did the reliabi-reliability come from?

Primarily from structured state, sensible decision logic, and external safety constraints, not from RL alone. That is a useful engineering result. In the current benchmark, RL provides an inspectable learned decision surface rather than an immediate success rate advantage.

Its value becomes more significant as incident histories become richer. Action outcomes vary by context, and manually maintaining every preference becomes difficult. This slide defines the current validation boundary.

The results come from synthetic scenarios. The agent responds after a failure signal. It does not predict a failure before it happens. Real incident diversity may exceed the current state space.

Some remediation actions are simulated or deliberately bounded. And online learning in a production environment would require strict approval gates, versioned policies, rollback support, and continuous monitoring. The result is credible feasibility demonstration of the system design with a clear path toward production validation.

The next step is a shadow mode deployment on representative in incident traces, where recommendations can be compared with human decisions before the agent receives execution authority.

There are five takeaways I would leave with an engineering team. First, use deterministic logic for facts that can be measured directly. Second, use learning only where contextual action selection adds real value.

Third, place safety constraints outside the learned policy, so a policy update cannot silently redefine its own authority. Fourth, treat escalation and post-action validation as first-class outcomes, not exemption paths.

And fifth, evaluate across repeated seeds and compare against simple baselines. A single favorable run is a demo, not evidence. A practical self-healing system does not need the largest possible model.

It needs a clear state, bounded action, reprodu- reproducible evaluation, observable decisions, and the discipline to stop when uncertainty exits its authority. This brings us back to the engineer in the opening video.

The goal is not to eliminate human judgment. It is to stop spending that judgment on the same recognizable failure at two in the morning. Before, the response is manual log inspection, schema tracing, delayed dashboards, and recovery process measured in working days.

After, the routine path becomes event-triggered diagnosis and RL-guided, but safety constraint action, explicit validation, and recovery measured in minutes when the case is supported. The unusual or high-risk failures are still go to the humans.

That is the point. Human attention is reserved for incidents where context, trade-offs, or authority genuinely require it. The code, synthetic benchmark, experiment scripts, and reproducibility instructions are available in the GitHub repository on-screen.

If you work on agent reliability, data quality, or production incident automation, I would especially value your feedback on state representation, reward design, and safety boundary. Thank you for watching.