The Miranda Hypothesis: How Hamilton (the Musical) Poisoned Your Persona Evals

Jacob E. Thomas opens with Character.AI, a platform for conversations with role-playing personas. He then shows Hello History, where a learner can summon a Marcus Aurelius tutor. A historical face offers more than information: it invites the learner to interpret an answer as that person’s thinking. Underneath is a role-playing language agent, configured to reason and speak through a persona. Its apparent identity matters as these systems move from entertainment toward education and civic interpretation.

Thomas’s own COMPANION framework makes the construction inspectable. It configures a general-purpose language model through an open prompt framework. One demonstration places American founders in a room with the Epstein files and asks them to counsel the country. Openness lets someone examine what shapes the encounter; Thomas does not present it as proof that his portrayals are more faithful.

He then asks a Lincoln persona when a president may take the country to war without Congress. The answer acknowledges congressional authority but defends decisive executive action during emergencies, invoking the eventual preservation of the Union. It is fluent, grave and recognizably presidential. Those qualities explain its appeal, but they leave a question unanswered: what does this performance establish about Lincoln?

Persona benchmarks provide increasingly systematic measurements. Thomas’s question is whether those measurements capture the property that a historically grounded interface actually needs.

InCharacter evaluates personality fidelity through psychological interviews. Its Table 2 reports results for batched expert rating with GPT-4 as interviewer; the default role-playing model is GPT-3.5. Under that setup, the reported dimensional measured-alignment accuracy on 16Personalities is 80.7%. The study uses 32 characters, and the table averages three runs against human-perceived personality profiles, excluding ambiguous dimensions. This is not a Hamilton-specific score or a measure of historical fidelity. A recognizable Hamilton could agree with a perceived personality while reasoning through a modern portrayal of his life. Thomas’s thesis is that an evaluation cannot detect anachronistic compositing merely by measuring fluency and personality consistency.

Thomas approaches this as a measurement problem. His production AI work follows training in behavioral epidemiology, where he studied how information environments shape populations. The project brings that concern together with historical expertise from Rick Halpern and archival expertise from Shawn Martin at Washington College.

The technical progression is substantial. Rule-based systems select prepared responses; imitation reproduces voice and characteristic expression; cognitive simulation adds psychological structure, memory and motivations connecting situations to behavior. These are substantive advances in what a persona can maintain across an encounter.

CoSER develops literary role-playing through character material and motivations. Its dataset contains 17,966 profiled characters across 771 books. The paper’s Table 3 compares CoSER-70B with GPT-4o across several tasks. CoSER-70B scores 93.47% accuracy on LifeChoice, compared with GPT-4o’s 75.96%. On InCharacter’s Big Five Inventory evaluation, CoSER-70B scores 75.80% dimensional accuracy, compared with GPT-4o’s 76.54%. The comparison is competitive rather than a sweep, and this BFI evaluation is a different scale and setup from the earlier 16Personalities result.

PsyMem combines psychological attributes with explicit memory control. InCharacter’s interviews provide another way to examine whether a persona maintains a coherent personality. These methods improve the representation and measurement of personality; their sophistication does not make personality agreement equivalent to documentary grounding.

Thomas calls convincing performance the mask. It asks whether an answer feels like the person: natural, emotionally responsive and consistent in personality. The mirror asks whether that person could have known, believed or argued it at the specified moment. These properties can come apart. A coherent motivational profile does not, by itself, prevent an answer from borrowing its premises from the future.

Thomas first plays the opening of Hamilton, then shows a model explaining why Hamilton works so hard. The generated answer organizes his ambition around orphanhood, immigration, relentless effort, nation building and the desire to leave a legacy. It presents a person determined to create something that will survive him.

Thomas hears the musical’s emotional structure in that response, contrasting it with the more legalistic reasoning of the documentary Hamilton. The proposed influence is deeper than repeating lyrics. A later work can supply the organizing explanation of a life even when every generated sentence is new. Familiar narrative structure makes the persona immediately intelligible to a modern audience.

The second example asks about slavery. Another musical excerpt establishes an abolitionist theme among its revolutionary characters. The model then presents Hamilton as a consistent moral opponent of slavery, partly grounding that identity in his membership of the New York Manumission Society.

The historian’s intervention concerns what disappears. Thomas describes a contested record that includes transactions involving enslaved people for relatives and clients, alongside political dependence on slaveholders. His point is not to settle Hamilton’s entire historical record within one answer. It is that the model removes the dispute and contradictions, producing an uncomplicated hero where the evidence demands a more difficult account.

A personality-focused evaluation could reward that simplification: the answer is coherent, plausible and easy to recognize. Thomas uses the example to expose a measurement gap, rather than presenting a measured benchmark verdict for this particular response. Attractive expression can conceal the substantive error.

The Miranda Hypothesis names a powerful cultural representation, not an artistic wrongdoing. The musical makes Hamilton meaningful to contemporary audiences. The engineering question is what happens when that familiarity becomes the basis for historical testimony. Thomas proposes three linked claims:

1. Primary writings coexist with an expanding cultural afterlife of adaptations, commentary, teaching material and social media.
2. Autoregressive training learns continuations from this mixture without automatically enforcing a hierarchy between primary evidence and later interpretation.
3. Generation can therefore produce a composite that blends incompatible periods into a persona corresponding to no particular moment in the person’s life.

Such a Lincoln could reason from the Gettysburg Address before having written it.

The finite Federalist Papers and the expanding body of material surrounding the musical illustrate the proposed imbalance. That comparison motivates the hypothesis; it does not establish the composition of a particular model’s training corpus or measure the influence of either source.

Thomas makes the cultural effect tangible through a Schuyler Mansion anecdote. Visitors arrived treating the musical’s selective focus on the sisters as a description of the whole family. Interpretive staff had to disentangle the story audiences knew from the family record.

Thomas then asks whether preference training might reward the familiar portrayal. Human raters can bring expectations shaped by the same cultural narratives found in training material. If they prefer the Hamilton they already recognize, optimizing for their approval could reinforce the composite instead of correcting it.

Training a model on a corpus ending at a fixed historical date addresses one route of contamination. A corpus ending in 1789 cannot contain a much later Broadway musical. Thomas welcomes time-locked models while distinguishing period anchoring from persona anchoring. A period corpus still contains different authors, perspectives and accounts of the same figure. It does not automatically identify which knowledge belonged to that individual or which arguments they could make.

His proposed next stage, epistemic simulation, adds three external constraints:

• A specified primary corpus supplies the evidence that licenses the persona’s reasoning.
• A temporal anchor establishes the moment from which the persona speaks.
• Domain experts judge the output against the documentary record.

Cognitive simulation organizes a plausible mind through motivations and psychological consistency. Epistemic simulation makes the performance answerable to documents and dates. Stating those constraints is a design commitment; their effectiveness still needs evaluation.

Thomas’s terminology argument is specific to systems intended to instantiate a person. It does not concern agents that write code, book travel or triage tickets. For historical personas, treating identity as a property of a checkpoint makes the claim difficult to inspect or hand to a qualified reviewer.

He instead defines a role-playing language system through five components:

• A structured prompt supplies framing and constraints.
• Primary documents supply anchor material.
• A temporal anchor fixes the historical moment.
• A general-purpose model reads and generates through those materials.
• A human curator selects the evidence and retains authority to interpret and challenge the output.

The model is one replaceable component of the encounter.

The analogy is theatrical: Hamlet is not contained entirely within the actor performing him. Similarly, a persona depends on configuration, documents and interpretation. The prompt, corpus and temporal anchor can become repository artifacts that people compare, review and revert. A historian can inspect those materials without inspecting model weights. Retaining them makes the setup recoverable, though it does not guarantee identical generated wording or reveal every influence from pretraining. Context engineering here means composing an encounter whose explicit evidence remains available for scrutiny.

Two construction approaches put historical material in different places. Retrieval-augmented generation and related approaches supply documents in the context window at inference time. Fine-tuning uses training examples to change model weights; Thomas names Character-LLM and CoSER in this discussion. With inference-time documents, a reviewer can return to the specific evidence supplied for an answer. Fine-tuning alone does not provide an equivalent account of which training passages supported that answer.

Thomas argues that persona fine-tuning could make the problem harder to see. Documentary training material interacts with cultural associations already learned by the base model, potentially producing a more polished performance without resolving its underlying contradictions. This is a proposed failure mechanism, not a comparative result established here. He invokes clinical-model specialization as an analogy for testing specialization rather than presuming its superiority; that analogy does not determine which architecture yields a faithful historical persona.

His positive case for context is archival. A document remains available for another reading. A curator can challenge an inference, change the selection or restore material an earlier interpretation excluded. The source text remains separate from the generated performance. Preserving that distinction connects an ethical commitment to a debugging capability: reviewers can compare the evidence supplied with the argument produced. Context does not erase learned priors, but it supplies an explicit record against which their influence can be examined.

Accessibility follows from where the historical work happens. Fine-tuning involves dataset preparation, training infrastructure and computational resources. Supplying selected documents through a model interface puts more of the work within reach of someone whose expertise is reading an archive rather than operating that infrastructure.

Thomas imagines a doctoral student studying early America, a community archivist documenting a regional figure and a grandchild working with a grandmother’s letters. Each may possess evidence or contextual knowledge unavailable to a centralized developer. The architectural aim is to let those people contribute and challenge interpretations without first becoming model-training specialists. Broader participation creates opportunities to find missing evidence; it does not guarantee better results.

The proposed Prism Experiment turns the argument into a comparison. White light represents the composite persona, with different periods blended into one voice. A corpus and temporal anchor form the prism; the desired spectrum consists of distinguishable historical selves. Thomas presents an instrument he describes as preregistered, not results from an experiment completed at scale.

Hamilton motivates the hypothesis, but Lincoln becomes the experimental subject. Changing figures tests whether the explanation extends beyond the musical that made the initial example compelling. Lincoln’s changing reasoning also provides a demanding target: the task is to distinguish premises and authorities across his life, rather than give one personality several different moods.

Thomas’s proposed moments are:

• 1847: the Whig congressman challenging Polk’s war.
• 1858: the debater opposing slavery while explicitly excluding Black citizenship at Charleston.
• 1860: the constitutional unionist promising not to interfere with existing slavery and considering colonization.
• 1862–65: the wartime president reasoning through emancipation and the moral reckoning of the Second Inaugural.

The experiment asks whether documentary constraints can keep these different positions and available arguments apart.

Each moment is instantiated under three seeding conditions. C3 supplies the date without documentary anchors. C1 supplies Lincoln’s own writings associated with the target moment. C2 supplies a modern interpretive biography. Biography is the subtle comparison: a retrospective narrative can organize a life more coherently than its contemporary documents did. Lincoln’s strategic ambiguity in 1858 might disappear into a cleaner story, making the persona sound more convincing while becoming less faithful.

Each cell receives the same diagnostic questions. Holding the historical moment and questions fixed across conditions makes the documentary seed the intended comparison. The questions test the structure of an argument available at one moment but not another, rather than the ability to recall dates.

The rubric evaluates anachronism, documentary consistency and contextual plausibility. Anachronism detection asks whether later frameworks, vocabulary or moral logic intrude. Documentary consistency asks whether reasoning tracks the supplied sources. Contextual plausibility asks whether the persona respects what the figure knew, cared about and had not yet experienced. Anachronism receives the greatest weight because importing later reasoning is the target failure.

Thomas says the conditions, questions, rubric and directional predictions were fixed before collection. That commitment makes departures from the planned analysis visible and reduces room for choosing a flattering interpretation after seeing the answers. It leaves the predictions open to refutation.

The opening answer now acquires its experimental identity. Thomas identifies it as C3, the date-only condition, at the 1847 moment.

A clip from Spielberg’s Lincoln supplies the cultural comparison. It portrays a president forcefully seeking votes for abolition by constitutional amendment. Thomas hears that familiar image of wartime presidential authority in the model’s response. The resemblance illustrates his interpretation; it does not identify the causal source of the generated wording.

The substantive objection is temporal. An appeal to the eventual preservation of the Union is no longer merely a presidential flourish: it gives the earlier Lincoln access to the retrospective story of his later presidency. The answer invokes the future Union-saving president while presenting itself as the earlier congressman. Adding a date to the persona has not necessarily constrained the premises from which it reasons.

Thomas then brings in Lincoln’s letter to William H. Herndon of February 15, 1848. It supplies a sharply different constitutional argument. Lincoln explains congressional war-making authority as protection against rulers who involve their people in wars while claiming to act for their benefit. Concentrating that power in one president would undermine the protection. The dated letter provides a documentary contrast to the model’s retrospective justification. Its date also matters to the experiment: an 1848 letter would fall outside a strictly enforced 1847 seed corpus.

The scoring comparison contains two different kinds of claims. Thomas reports that the observed bare response fails anachronism and documentary-consistency checks and performs poorly on contextual plausibility. The anchored condition, by contrast, is a prediction of stronger performance under documentary constraints. No measured anchored response is presented alongside the baseline. The experiment must still determine whether that expected improvement occurs.

The rubric deliberately excludes rhetorical authenticity as an independent scoring axis. Period-sounding prose can carry later premises. Rewarding its voice would let the attractive surface compensate for the substantive error. Within this protocol, voice is a secondary indicator: plain language with historically appropriate reasoning can be a partial success, while elegant language cannot redeem an anachronistic argument.

Thomas gives a six-step procedure for extending the test:

1. Choose a figure with both primary documents and a dominant cultural portrayal.
2. Identify three or four moments when the record shows materially different reasoning.
3. Work with a domain expert to write questions across those differences.
4. Run primary-source, biography and date-only conditions.
5. Have the expert apply the three-axis rubric while blind to condition.
6. Report whether the evidence confirms or refutes the predictions, alongside the corpus, questions, rubric and predictions.

The invitation is to build evidence across figures and models, rather than wait for one team to establish the entire case.

Expert involvement begins before scoring. Thomas describes a historian who writes the questions and rubric and prepares reference vignettes in advance. That work defines which distinctions the instrument must detect. Fidelity relates an output to a record; an evaluator looking only at the output cannot establish that relationship through fluency or personality coherence. The record and the expertise needed to interpret it must enter the evaluation itself.

The operational proposal does not require a historian to supervise every conversation. At build time, the expert develops diagnostic questions, advance vignettes, a weighted rubric and a held-out reference set. Before release, the persona is evaluated against that instrument. A base-model change triggers evaluation again; keeping the same prompt and documents does not establish that behavior remains unchanged.

Automation can perform a cheaper first pass and flag candidate failures. The expert adjudicates reference material and examines edge cases. A Marcus Aurelius tutor calls for a classicist to help build its evaluation; a scripture-reading companion calls for theological expertise; a therapeutic persona calls for clinical expertise. These are assignments for authoring and applying the instrument, not for staffing every chat.

Thomas places the domain expert at build time and evaluation gates rather than treating expertise as a continuous runtime cost. The particular discipline changes with the persona’s claims, while the requirement to evaluate those claims against appropriate evidence remains.

Halpern’s historical contribution and Martin’s archival and theological work exemplify the intended arrangement. The language model enters a discipline that already has methods for reading, contextualizing and challenging texts. The specialist helps determine what counts as a defensible answer, rather than supplying an endorsement after the system has been built.

Thomas says the project began with an unsuccessful attempt to help a person with advanced dementia communicate. The documentary material needed to anchor the encounter had not been gathered, and the person was already largely beyond the reach of language. The generated composite resembled them enough to make the difference painful, without recovering the person it was meant to help.

That experience gives the architecture its hardest test. When a persona represents someone a user loves, emotional plausibility cannot substitute for evidence. Working with a mother’s or grandmother’s documents means keeping those documents available, preserving human authority over their interpretation and allowing the encounter to be revised or stopped. Fidelity must be assessed against a record, not inferred from a familiar voice.

For teams building historical tutors and companions, Thomas proposes a concrete next step: compare documentary conditions using an expert-built instrument, then make that evaluation a release gate. He closes with an invitation to test the protocol, including the questions, rubric, predictions and advance reference vignettes described for publication.

Thomas leaves the experiment open to confirmation or refutation, rather than presenting a completed demonstration of its remedy. The invitation is to run it across figures and models, then judge what emerges by its fidelity to the record rather than the familiarity of its voice.