Loophole: Adversarial Agents To Stress Test Your Morality — Brendan Rappazzo, Morgan Stanley

Brendan Rappazzo introduces Loophole as an independent open-source project, separate from his work as a machine learning researcher at Morgan Stanley. Its original form is a terminal-based game: state your morals, have one agent convert them into formal rules, and let adversarial agents search for cases where the rules and the morals disagree.

The project began with a concrete consent problem. After submitting DNA for ancestry testing, Rappazzo broadly opted out of other uses because he worried about where permission might lead. Yet a blanket refusal did not express his actual judgment. If someone presented a particular murder investigation or cold case, he might approve that use while rejecting another.

Case-by-case judgment could capture the nuance, but anticipating every future use would be “cognitively prohibitive.” Loophole reframes that burden as search: instead of asking a person to enumerate every exception in advance, generate situations that probe the boundaries of the stated principles.

The legal analogy supplies the model. Loophole treats a legal system as an attempt to translate moral beliefs into written rules. Broad language misses distinctions; highly specific language creates corner cases of its own. Rappazzo connects synthetic cases to the role of case law in common-law systems, where concrete disputes help judges interpret boundaries that legislation cannot fully anticipate.

What happens after the user enters a principle? A drafting agent first expands the natural-language statement into a legalistic code. Two agents then attack that translation from opposite directions:

• Loophole finder: searches for conduct that violates the user’s morals but remains legal under the generated code.
• Overreach finder: searches for conduct the user considers acceptable but the generated code prohibits.
• Judge: compares the original principles, formal code, and synthetic case, then decides whether the code can be patched or the value question must return to the user.

Here, “legal” means permitted by Loophole’s generated code, not necessarily permitted by real law.

That final distinction separates editing from moral judgment. If the principles already imply a clear answer and the code merely translated them poorly, the judge can revise the code automatically. If the case exposes an underspecified principle or a genuine tension between principles, an automatic edit would invent a value judgment. The system therefore escalates the case to the user.

The diagram answers the central implementation question: where does machine-driven refinement stop and human judgment begin? Both adversaries feed cases to the same judge, but the judge has two exits. A translation error returns to the code as a patch; an unresolved value question leaves the automated loop and goes to the user.

The DNA demonstration makes the distinction observable. The user supplies several principles, and the drafting agent produces a document shaped like legislation, with a preamble, articles, and sections. The formal structure gives the adversaries explicit wording to test, but precise-looking language does not guarantee that its scope matches the original intent.

The first generated case finds a loophole. An insurance company does not train a predictive model directly on the person’s DNA; it trains on artifacts derived from that DNA. The restriction covers the source representation but misses its derivatives, even though the resulting use still violates the stated principles. The judge classifies this as a translation problem, patches the code, and displays the change as a Git-style diff. The talk does not provide the amendment’s wording, so the example establishes the need to cover derived artifacts without specifying the final legal definition.

The second case finds overreach. A person submits DNA for genetic research, and a researcher discovers a rare but treatable disorder. The current code appears to prevent the researcher from telling the participant. Unlike the artifacts case, this raises a new consent question: does permission for research also permit an unexpected, potentially beneficial disclosure? The judge cannot derive that answer from the existing principles and asks the user to decide. After that decision, the system patches the code, although the talk does not state which choice was made.

The project’s unusually large response after its open-source release encouraged Rappazzo to test whether the same loop could do more than provoke personal reflection. The next three branches preserve the same shape—formalize intent, attack the formalization in both directions, and repair or escalate—but change what the “code” governs.

The first branch applies Loophole to customer-facing chatbots and agents. A company states both its moral policy and the subjects the assistant should or should not discuss. A drafting agent turns those requirements into a codified system prompt rather than a legal document.

The two attack directions map cleanly onto chatbot behavior. One adversary tries to make the bot discuss a prohibited subject; the other asks legitimate questions and looks for false refusals. Testing only prohibited answers would miss the second failure mode, where a supposedly safe assistant becomes useless by declining work it should perform. Rappazzo presents this as an exploratory prompt-construction method, without reporting a measured improvement or showing that the final prompt enforces every intended rule.

The second branch treats a person’s preferences as a contract to compare against external terms. A user might state how online services should handle their data, refine those preferences through adversarial testing, and then compare the resulting code with a company’s terms of service. Synthetic cases make the disagreement concrete: they show a situation where the company can do something the user’s principles reject.

For a large company, finding a conflict does not create bargaining power. As Rappazzo puts it, “It’s not a negotiation.” The practical gain is better information about what the user is accepting. Loophole can clarify the gap between two documents, but it cannot compel the company to change its terms.

A more symmetric version gives both parties a voice. Two people entering a cross-border work agreement could separately specify expectations around payment and conduct. The system would stress-test each side, identify conflicting cases, and surface them before anyone signs. That may improve the wording and the conversation, but the proposed mechanism does not enforce performance or resolve a later breach; confidence in the text is different from confidence in the counterparty.

The most aspirational branch applies the framework to government. A constituent could compare a refined moral code with a bill or politician’s positions and inspect concrete disagreements. A legislator could run a proposed bill against simulated colleagues before submission. Rappazzo acknowledges substantial privacy and logistical problems before setting them aside for this experiment.

The Senate simulator builds profiles from senators’ public voting histories and other public information, infers a moral system for each senator, and converts it into a codified rule set. A submitted bill is then evaluated against every profile, producing a simulated vote, leaning, and explanation.

Once simulated support becomes a score, bill revision becomes an optimization loop. The system proposes a language change, evaluates the revised bill against the senator profiles, and keeps changes that increase support while attempting to preserve the bill’s core principles. It can also rank candidate changes, leaving the user to choose which tradeoffs are acceptable.

A Medicare example reportedly begins around a 50–50 simulated split and reaches 52 supporting votes after hill climbing. The observable change is the vote count; the causal path is proposed wording change, reevaluation against each inferred profile, updated aggregate support, and another iteration. The talk does not show the textual revisions or specify the search algorithm, so it remains unclear whether the system preserved substance or merely judged that it had.

The diagram answers what “hill climbing a bill” means here. It is a feedback loop over a model-generated score, not a direct interaction with legislators. The user remains outside the automatic loop to reject compromises that improve the score while changing the bill in an unacceptable way.

The final experiment replaces senators with synthetic constituent profiles. Using an NVIDIA dataset of USA personas, Rappazzo takes 500 personas per state. For each persona, the model drafts morals and a corresponding legal contract, then compares a bill with those contracts. The same hill-climbing process can optimize for greater agreement across the modeled population rather than greater support among modeled senators.

The consequential limitation sits at the measurement boundary. The talk provides no comparison between simulated and actual Senate votes, and no validation that the 500 personas per state, their generated morals, or their contract comparisons reproduce real constituent preferences. The 52-vote result therefore means 52 votes inside the simulator, while a higher persona-agreement score means better agreement with the constructed personas—not evidence that real voters would support the revised bill. The privacy and logistical problems of collecting comparable preferences from actual constituents also remain unsolved.

Loophole’s immediate value is more modest and more defensible: it turns vague principles into concrete questions. A user can add nuance after each difficult case and continue until the adversaries stop finding contradictions. That stopping point can increase confidence in the rules tested so far, but it cannot prove that the moral system is complete or consistent; the agents may simply have exhausted the cases they can generate.

That leaves the project with a useful split personality. It is already a game for discovering what your written principles accidentally permit or forbid. Its larger applications—system prompts, decentralized contracts, and legislation—are experiments in translating intent into rules and attacking the translation before the stakes become real. Rappazzo closes by inviting people to try, fork, and contribute to the open-source project.