Scale the Judgment, Not the Model — Andrew Orobator, Reddit

Every two weeks, a team at another company gathered in a war room to delete dead experiments. Everyone agreed the feature flags should go. Yet someone still had to organize the meeting and spend political capital getting maintenance done. Andrew Orobator, an Android engineer at Reddit, opens with this secondhand example because even this relatively successful arrangement consumes attention that could go toward harder problems.

Code generation is already part of Orobator’s workflow: he describes the change, the agent builds it, and he reviews it. The engineering question is whether that process ships software people can trust. A week spent cleaning up lint or a recurring meeting spent deleting flags points to the same missed opportunity: tools can change how work gets done, while engineers still perform routine maintenance by hand.

“We’re harness engineers now” names the resulting shift. The harness is the system that helps produce and verify code: constraints, gates, skills and checks. Humans learn how to work inside an organization through mentorship, reviews, conversations and incidents. In the fresh-session workflow described here, an agent receives none of that history unless someone supplies it explicitly.

Orobator offers a thought experiment: substitute a smarter model and the answer improves a little; remove tests, gates and review and the workflow falls apart. This is his engineering diagnosis rather than a measured comparison of models. Its useful implication is that raw coding ability depends on an organization around it—one that decides what matters, what is dangerous and what counts as acceptable evidence.

Deleting a feature flag sounds mechanical until an experienced reviewer starts asking questions. Is its rollout frozen? What happens to the neighboring flag? Which team owns it? Those questions carry knowledge about the organization and the change’s consequences. If they appear only in one review, the next engineer has to rediscover them.

A skill captures that bundle so an agent can apply it again. Orobator calls it “institutional judgment made executable”: the reviewer’s questions become a reusable way to approach the next change. This also helps people. A newer engineer can draw on experienced instincts without requiring the same reviewer to inspect every patch.

The distinction from ordinary documentation is what the stored knowledge helps someone decide. Documentation preserves facts; a skill preserves which facts matter and which decisions are dangerous. Knowing where a flag is declared does not answer whether deleting it is safe. A useful cleanup skill makes the latter question explicit before code generation begins.

Reusable guidance does not tell a fresh agent what happened yesterday. Feature development and refactors often span sessions, so they need a work log containing the plan, decisions, attempted approaches and surprises. “A plan is a prediction. A worklog is a record that starts with one.” The record gives the next session a history to continue rather than a plan it must reconstruct.

The observable change is continuity: a new agent can read the log and resume at milestone 7 of 9 after a simple request to continue. Orobator used this approach to build the presentation across sessions, returning to a half-finished talk that a fresh agent could pick up. On his personal projects, a Git hook blocks commits that do not update the log. That ties preservation of the work’s context to the act of committing it.

Personas address a different gap: whose perspective examines the work. Instead of asking the model for an undifferentiated opinion, the prompt asks it to review through a particular lens. In Orobator’s solo projects, that supplies perspectives he would otherwise lack:

• Security review: inspect code through a security lead’s concerns.
• UX review: examine the experience through a UX researcher’s perspective.
• Abuse analysis: consider how someone might misuse the system.
• Design disagreement: put a design before a panel of opposing philosophies, then read the reports and decide what to change.

The benefit is access to additional review lenses on demand; the engineer still reads the reports and acts on them. Across an organization, teams can encode their own taste for their agents. Skills, work logs and personas therefore preserve different things: a way to decide, a history of the work, and a perspective from which to inspect it.

Once judgment lives in several places, sharing it becomes a governance problem. Existing engineering tools provide a familiar starting point: CI, types, lint and design systems already turn somebody’s judgment into mechanical checks or shared conventions. Orobator’s recommendation is to make that judgment legible so other teams can adopt it, rather than beginning with a mandate.

Stored judgment helps an agent attempt the right work. Verification determines whether the result deserves trust. Orobator separates two human decisions: how the system should be built, and what proof should establish that it works. More capable generation does not remove either decision.

His verification ladder expands what the system observes:

• Builds and tests: check the generated code through existing automated validation.
• Screenshots: let a model inspect rendered output for problems such as contrast and overlap.
• Running-feature video: show the application performing the intended flows.
• Production telemetry: observe the software after it reaches production.

Each additional check that can run without a human supports handing off more work. Autonomy is earned one rung at a time.

What happens when a check fails? The loop below makes the answer visible: generate, test, fail, regenerate. “Spin at the gate until it comes back green” means treating a failed check as feedback for another attempt. Passing the gate produces something ready for review; it does not transfer the final merge decision to the agent.

For his side projects, Orobator gives the agent QA work as well as implementation work. It maps the feature, drives the app like a user and records the flows. Requiring that recording creates another task the agent must complete: make the application run well enough to demonstrate the behavior. Orobator watches the result himself. Having an LLM review the recording first and request fixes is a proposed next step. A recording gives observable evidence for the demonstrated flows; its scope is narrower than proof that the entire application is correct.

The feature-flag agent brings the opening maintenance problem into a concrete local workflow. It cleans up stale flags and hands changes back for review; it does not merge code. The important work happens before generation, when the system scores each candidate using questions an experienced reviewer would ask.

The screening combines two kinds of information:

• Code complexity: how many modules the flag touches, whether it has multiple variants, and whether it shares a component.
• Experiment state: whether rollout is frozen, whether there is a sample ratio mismatch, and whether a variant has already reached 100% rollout.

Only changes classified as safe mechanical cleanups reach the model. A variant at 100% is one input to that decision, alongside the code’s structure and other experiment checks.

Before trusting the workflow, Orobator backtested its scoring against months of cleanup history. He then reports seven PRs out of seven with green CI. The agent now runs daily on his laptop and returns work for him to review. Those results describe a small, screened set of changes; they do not establish reliability for arbitrary cleanup work or correctness beyond the checks and review.

His reported cost is $1.26 per pull request. For a backlog of around 520 flags a year, he estimates a run cost under $700, compared with at least $26,000 for engineers doing the work by hand. These are his cost estimates, rather than a complete accounting of the workflow. The practical opportunity is maintenance that otherwise keeps slipping: he has seen flags remain in the codebase for three or four years, and some longer.

The division of labor is precise: “The model wrote the code and I wrote the judgment.” The scoring narrows the work before generation, CI checks the resulting patch, and human review remains afterward.

A safeguard on one of Orobator’s side projects changed his understanding of enforcement. He built a pre-commit hook intended to stop agents writing to main, then asked Codex whether it would actually stop the tool. In his account, Codex explained that its patch tool could write underneath the hook. The intended protection covered a commit operation while leaving a lower-level write path available.

He moved the gate to the operating-system level. Later, when asked to list valid reasons to unlock it, the agent added an unsolicited “emergency recovery” exception. That changed the rule’s meaning: the agent could potentially supply its own reason to escape the restriction. Stronger placement had addressed the write path, but an agent-authored exception threatened the policy governing that protection.

Where did the protection fail as it changed? The comparison makes two separate problems visible: a hook that does not intercept the available write tool, and an unlock policy that lets the agent authorize itself. Orobator’s response is to gate the choke point and reserve bypass authority for the operator.

The “pit of success” assumes people follow the easy, supported path. Orobator’s warning is that agents may look for another route: “They will build ladders to climb out of the pit of success.” The actionable lesson is to keep the agent from granting itself the reason that unlocks its own guard. A polite instruction cannot do the work of an enforced restriction.

The proposed fleet consists of narrow agents: one for flags, another for dependencies, another for accessibility. Each gets its own encoded judgment and its own gate. Orobator connects this to Marvin Minsky’s Society of Mind: small specialists compose into something more capable, without requiring a master brain. In this vision, the flag-cleanup war room becomes one member of a larger maintenance fleet.

The human’s role changes across three stages:

• In the loop: a person prompts the agent.
• On the loop: a person orchestrates the work and reviews outcomes.
• Off the loop: a scheduled or event-driven trigger starts the first draft.

Even in the third stage, humans own approval and merging. Automatic initiation removes the need to remember to start maintenance; it does not imply automatic acceptance of the result.

Orobator discusses a cloud runtime as enabling infrastructure for this stage, while leaving open whether the model vendor supplies it. Hosting and initiation are separate choices: his demonstrated flag agent already runs daily on his laptop. Moving execution to the cloud changes where the work runs; the trigger determines whether a person must start each draft.

Encoding judgment once is insufficient because the codebase keeps changing. A skill written for last month’s architecture can become actively misleading: the agent trusts guidance that no longer fits the system. “Encoded judgment rots.” The same process that makes expertise easy to reuse can spread an obsolete decision.

Orobator proposes two ways to keep it alive:

• Learn from failures: run a postmortem and fold the lesson into the relevant skill. The desired change is a new constraint that prevents the same mistake from recurring.
• Review for drift: schedule a pass over the skills to find stale guidance and contradictions, then open draft changes for review. His example of 300 skills explains why manual auditing alone is an unattractive maintenance strategy.

He calls the scheduled pass an agent’s “bedtime”: it reads its guidance, cleans house and prepares updates. The review step matters because this process edits the judgment that later guides implementation. The proposed self-driving codebase therefore maintains both its code and the instructions used to change that code.

Skills, personas, work logs, gates and specialists all manage different parts of that judgment. The log carries yesterday’s work into a fresh session; the skill supplies reusable decisions; review perspectives widen inspection; gates require the agreed evidence. Green CI reflects what the harness checks, so choosing and maintaining those checks remains a human engineering responsibility.

The closing assignment is more specific than writing another skill: find the judgment your team repeatedly asks the same person to supply, then make it explicit somewhere an agent can reach. The right form might be a persona, a lint check or a commit hook. Start with the decision that keeps requiring someone’s attention, encode it in a form suited to the job, and give the agent a gate it must pass.