# Persona Engineering: A Field Guide to AI Synthetic Personas — Ishan Anand, InsightSciences.ai

Ishan Anand

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/YnNF55QV0zs-persona-engineering-field-guide-ai-synthetic) · [Video](https://www.youtube.com/watch?v=YnNF55QV0zs)

## Persona Engineering: A Field Guide to AI Synthetic Personas

A role prompt can generate a plausible customer, but a useful synthetic respondent needs grounded context, calibrated answers and validation against human data.

Familiarity with LLM prompting and basic concepts such as probability distributions and correlation will help.

### From role prompts to bounded forecasts

[0:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=20s)

Start with the familiar prompt: `You are a…`, followed by a task. Asking a language model to respond as a customer turns that simple steering technique into a research tool: show it a product concept or marketing message, then ask how it would react. Synthetic respondents have become a commercial category, with growing funding and attention. The engineering question is whether their answers predict anything about the people they represent.

**Synthetic personas are forecasts of people.** Weather forecasting offers a useful analogy: more compute and data make prediction possible, but accuracy depends on the regime and forecast horizon. A forecast that works within its tested range can fail outside it. The same distinction matters when moving from a plausible persona response to a claim about a customer population.

![Slide titled “A Once-Impossible Prediction Problem Becomes Practical,” with accuracy curves for 3-, 5-, 7-, and 10-day weather forecasts.](https://ai.engineer/api/talk-essay-media/347a5048d2b024f3c70865cfe397601d6783963bd411356995d01faf5763b5b4)

Weather forecast accuracy improves over time but declines with longer forecast horizons.

Ishan Anand introduces himself as Chief AI Officer at [InsightSciences](https://insightsciences.ai/), where he builds synthetic personas for market research and insights teams. His vendor perspective is explicit; the technical examples draw on published research. They address three practical questions: how personas fail, how to construct them, and how to measure their agreement with humans.

### Language as a simulation medium

[2:35](https://www.youtube.com/watch?v=YnNF55QV0zs&t=155s)

The ambition to forecast people predates LLMs. In the 1950s and 1960s, computing inspired both weather prediction and attempts to predict electorates. Simulmatics, chronicled by Jill Lepore, promised people forecasts from statistics and contemporary computing. Anand presents its failure to deliver that promise as a reason for humility.

LLMs introduce a different modeling interface. Traditional simulation requires translating a system into formulas; feelings, attitudes and choices are difficult to specify that way. Language supplies an intermediate representation in which a researcher can describe a person, a situation and a decision. The machinery remains mathematical, but the model can now be conditioned on descriptions that would be awkward to encode as explicit behavioral equations.

An interview-grounded agent makes that idea concrete. Anand describes roughly a thousand people receiving extensive interviews about their backgrounds, views and attitudes, then completing surveys and personality tests. Their interview transcripts become context for agents that take the same assessments. He describes the interviews as roughly two and a half hours; the study authors describe them as two hours.

Anand reports approximately 83% agreement after normalization against human repeatability. The matching original study, [Generative Agent Simulations of 1,000 People](https://arxiv.org/abs/2411.10109v1), reports 85% specifically for General Social Survey responses, relative to participants’ own repeat responses two weeks later. Neither figure should be read as raw accuracy across the entire assessment battery. The central result is that substantial personal context can support predictive survey responses, with performance interpreted against the inconsistency of the people being modeled.

### When the model changes the experiment

[4:49](https://www.youtube.com/watch?v=YnNF55QV0zs&t=289s)

A synthetic experiment can fail even when its question looks straightforward. In the demand experiment discussed in [The Challenge of Using LLMs to Simulate Human Behavior](https://arxiv.org/html/2312.15524v2), a simple customer prompt provides a product, its category and its price, then asks for a purchase or non-purchase decision. Human respondents receive the same task.

The human curve slopes downward: higher prices produce lower purchase probabilities. The LLM curve instead forms an inverted U. In its rising portion, increasing the price makes the model more likely to buy. The highlighted region of the slide isolates precisely that counterintuitive behavior.

![Purchase probability versus price chart with a declining red human curve and an inverted-U teal LLM curve; a red outline highlights the LLM curve’s rising portion.](https://ai.engineer/api/talk-essay-media/8f1a1e04d0f305af8cec5a7b18681f6e53c8aab22c853c9a2fdbf2029fc9d299)

Human and LLM purchase predictions diverge at low prices.

Additional experiments explain the distortion. The model treats price as evidence about other properties: perhaps a cheap product is close to expiration, or perhaps competitors’ prices change along with the offered price. Human participants treat those properties as fixed, while the model infers a different surrounding world at each price point. The experiment therefore changes more than its intended independent variable.

Consider a gold watch on a table. In a physical experiment, the watch and room remain fixed while different people judge its price. In a sparse prompt, the model may infer a jewelry store, then infer a wealthy customer, then infer a greater willingness to purchase. Missing context becomes an additional source of variation.

**Ground the experimental world as well as the persona.** Specify the participant’s relevant characteristics, the surrounding context and what the experiment holds constant. A proposed constraint block for the watch example could make those assumptions explicit:

```text
Evaluate the same gold watch at each offered price.

Across price conditions, keep these facts unchanged:
- The watch's condition, quality and authenticity.
- The purchasing setting.
- The customer's budget and preferences.
- The prices of competing products.

Only the offered price changes. Do not infer a different
setting or customer income from that price.

Would this customer purchase the watch at the offered price?
Answer purchase or do not purchase.
```

Actual persona and product facts still need to be supplied. Unlike a human participant, who brings a physical environment into the experiment, the model has only the world described in its context. That makes study construction relevant prompt content. It also creates a tradeoff: the linked demand paper warns that explicit controls can introduce **focalism**, drawing disproportionate attention to the properties named. Richer grounding is something to validate, not an automatic cure.

### Does the opinion survive a reordered question?

[7:43](https://www.youtube.com/watch?v=YnNF55QV0zs&t=463s)

A second failure mode appears without changing the persona or question at all. Move `Yes` from the first answer position to the second, leaving the available choices unchanged. Anand reports that averaging the two answer orders in the cited experiment produced a 50–50 result. The apparent opinion was dominated by answer position.

Humans also exhibit order effects, but Anand describes the model effect as much stronger in this example. A persona therefore needs **durability testing**: reorder its options, reword its questions and challenge its expressed opinions. The question is whether a response reflects a stable modeled preference or an incidental feature of the prompt.

### Predicting statements is easier than predicting actions

[8:27](https://www.youtube.com/watch?v=YnNF55QV0zs&t=507s)

The third failure mode follows from the training medium. LLMs learn more directly from what people say than from what they do. An attitude survey asks for something already native to text; predicting behavior requires connecting language to actions that may be sparsely represented in training data.

In the social-science prediction chart Anand discusses, black circles represent LLM predictions and blue marks represent experts. He describes their overall performance as roughly comparable, then draws attention to the split within the chart: models do better on the survey experiments above than on the field experiments below. That distinction matters more for task selection than the overall model-versus-expert comparison.

For gym research, ask about attitudes toward working out alongside questions about attendance. The attitude response may help triangulate behavior, but its usefulness as a proxy must itself be assessed. A convincing account of someone’s enthusiasm for exercise is not yet a reliable prediction of how often they visit a gym.

### Choose persona prompts against human ground truth

[9:59](https://www.youtube.com/watch?v=YnNF55QV0zs&t=599s)

The simplest construction technique is direct conditioning. Argyle and colleagues’ [Out of One, Many](https://arxiv.org/abs/2209.06899) used **GPT-3** as a text-completion model. Its first-person backstories therefore look different from a contemporary chat exchange: the model continues a statement rather than replying to a user message.

In Anand’s example, an ideological description begins with `I am a strong liberal`, supplies further context about progressive values, and ends with `In 2016, I voted for`. Sampling continuations produces names such as Hillary Clinton and Bernie Sanders. A conservative backstory supplies a contrasting condition. The technique turns repeated continuations into a sample of modeled responses conditioned on the supplied description.

Newer models and prompting methods expand the options, but they do not remove the need for empirical selection. Compare candidate constructions against known human responses. In a separate voting-pattern example, researchers compared reality with four persona constructions and found that increasing descriptive detail amplified model bias, pushing the results farther from the observed voting patterns. **More detailed personas are not necessarily more faithful personas.** Grounding useful facts and generating elaborate backstories are different operations.

### Fine-tune the response distribution

[11:37](https://www.youtube.com/watch?v=YnNF55QV0zs&t=697s)

Fine-tuning is a natural next option when relevant information—especially behavioral information—is missing from the model’s training experience. [SubPOP](https://arxiv.org/abs/2502.16761) provides an example organized around population responses rather than a single supposedly correct answer:

1. Construct a prompt containing demographic information and a survey question.
2. Obtain the model’s distribution over answers.
3. Compare it with the known human distribution for that subpopulation.
4. Fine-tune to improve their alignment.

Anand reports that alignment improved for both the training populations, shown in blue, and unseen groups, shown in white, with nearly comparable gains. The completed slide highlights the unseen-group result.

![Fine tuning slide with a survey-to-LLM diagram, a group results table, red outlines around selected rows, and the annotation “Alignment improved even for unseen groups.”](https://ai.engineer/api/talk-essay-media/e15e103654745d671e047f35782248e39cac444a340f78f37e0866a710794d2a)

Fine-tuning results highlight improved alignment even for unseen groups.

One possible explanation is that the model already contains relevant knowledge about those groups but does not know how to express it in the requested survey format. Fine-tuning may teach the task as well as supply information. Anand presents this as a hypothesis suggested by subsequent research: some persona engineering consists of eliciting latent knowledge, rather than adding an entirely new representation of a population.

### Let the model answer in words, then calibrate

[13:08](https://www.youtube.com/watch?v=YnNF55QV0zs&t=788s)

A third technique changes the response interface without requiring fine-tuning. In [Semantic Similarity Elicitation of Likert Ratings](https://arxiv.org/html/2510.08338v2), the initial setup is familiar: demographics in a system prompt, a product concept, and a purchase-likelihood question scored from 1 to 5. Here, 1 means least likely and 5 means most likely. This measures **purchase intent**, although Anand occasionally calls it willingness to pay; it does not estimate a monetary price.

The alternative lets the model work in its native medium. Instead of demanding a number immediately, ask for a textual reaction: conditional interest, for example, if the product works well and is affordable. Then map that reaction onto the survey scale:

1. Have humans write reference responses for each scale point, from outright rejection at 1 to enthusiastic purchase intent at 5.
2. Measure semantic similarity between the generated reaction and those reference responses.
3. Use the resulting similarity vector to derive a probability distribution over ratings.

The output can place probability on neighboring answers rather than forcing every reaction into one category.

Anand’s illustrated distribution concentrates on ratings 4 and 5, with little probability on 1 through 3. That resembles a respondent who might choose slightly different answers on different days. It also addresses a common problem: a model can recover an average while collapsing the variation around it.

The displayed shape-similarity measure runs from 0 to 1, with 1 indicating identical distributions. Anand reports that the calibrated yellow results outperform the direct-rating pink or purple results on distribution shape. The linked paper defines the measure as `1 − KS distance`. Its reference responses were manually optimized on the evaluated surveys, so the result supports calibration on those tasks rather than automatic transfer to arbitrary products or realized purchasing behavior.

### More synthetic answers are not more human evidence

[15:41](https://www.youtube.com/watch?v=YnNF55QV0zs&t=941s)

Generating additional responses for an underrepresented population does not, by itself, strengthen statistical significance about that population. The weather analogy distinguishes two operations: placing more rain gauges to observe today’s rain, and rerunning tomorrow’s forecast with unchanged inputs. More observations can supply new evidence; more runs of the same forecast refine the estimate of what that model predicts. They do not make the forecast itself more accurate.

Evaluate synthetic responses against observed human responses. Unlike a conventional right-or-wrong test, persona evaluation must compare distributions. A model can be badly displaced, or it can recover the mean while missing the distribution’s shape. Anand recommends combining a correlation-type metric with a shape-type metric.

| Evaluation lens | What it checks |
| --- | --- |
| Correlation | Whether predictions track differences in human responses |
| Distribution shape | Whether response frequencies and variation match |

The two views are complementary: getting relative patterns right does not establish that the full response distribution is faithful.

The human reference also contains uncertainty. Returning to the interview study, Anand describes participants repeating the assessments two weeks later and reports approximately 80% human self-consistency. He uses this to explain why agent agreement was normalized against human repeatability. That rounded figure is his account of the baseline, not a verified common rate across every test in the battery. Human disagreement with their own earlier answers constrains how to interpret model disagreement.

When recontacting participants is impractical, estimate a human-sample comparison baseline from existing data:

1. Randomly split the human observations into two groups.
2. Treat one group as the prediction set and the other as the reference.
3. Compute the same comparison metric used for synthetic versus human data.
4. Repeat the split many times and average the results.

Both halves still contain real human observations. This procedure estimates disagreement between samples drawn from the available data; it does not reproduce a longitudinal retest of the same individuals. It helps distinguish model error from the variability already present in the reference samples.

### Research for a human-plus-agent economy

[18:29](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1109s)

“They are not people, they are forecasts.” This framing makes validation an ongoing requirement. Synthetic personas, like weather models, operate within bounds, can improve over time and earn trust through comparison with reality.

Their relationship to human research is therefore complementary. Anand offers a further reason: agents increasingly mediate customer awareness, consideration and purchase decisions. If a customer delegates part of that process to software, studying the human alone leaves part of the decision system unobserved. The emerging research target is the human-plus-agent ecosystem, including how those actors influence one another.

### Ask the question the survey missed

[19:31](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1171s)

A practical opportunity appears two months after a survey has finished. Someone asks a question the survey never included. If the answer is already in the data, retrieval is enough. If it is not, the team often falls back on an expert’s extrapolation—or makes the decision without further research. Anand positions an expert working with a grounded synthetic persona as a way to improve that extrapolation and extend existing human data into more phases of product development.

A further direction is **generative agent-based modeling**: let multiple personas interact and examine the resulting dynamics. The talk names this possibility without demonstrating a simulation. It moves the research question from how one respondent answers to how modeled people affect one another.

![Slide titled “Customer Understanding Where Research Doesn’t Reach,” with a table mapping situations and current behavior to synthetic opportunities, including queryable research memory and simulated social dynamics.](https://ai.engineer/api/talk-essay-media/97073361849c418758f524979ebbbe3bff44d1b446abbfa3fb3cacdc2268a0f3)

Synthetic personas extend human research into a living, queryable asset.

The ambition is to turn completed human research into a living, queryable asset: evidence that can inform later questions and new scenarios, with synthetic extensions evaluated against the human observations that make them useful.

## Resources

- [Generative Agent Simulations of 1,000 People](https://arxiv.org/abs/2411.10109v1) — Original study of interview-grounded agents representing 1,052 people, evaluated against human survey responses and repeatability.
- [The Challenge of Using LLMs to Simulate Human Behavior](https://arxiv.org/html/2312.15524v2) — Demand-estimation experiments explain how unspecified context introduces confounding and why explicit controls can introduce focalism.
- [Out of One, Many](https://arxiv.org/abs/2209.06899) — Argyle and colleagues study GPT-3 samples conditioned on real participants' sociodemographic backgrounds.
- [SubPOP: Fine-Tuning for Public Opinion Distributions](https://arxiv.org/abs/2502.16761) — Survey-based fine-tuning aligns model response distributions with human subpopulations and tests generalization to unseen groups.
- [Semantic Similarity Elicitation of Likert Ratings](https://arxiv.org/html/2510.08338v2) — Maps free-text purchase reactions into rating distributions and evaluates both product rankings and distribution similarity.

## Slides

### Synthetic Personas Move From Novelty to Market Momentum

Companies use synthetic respondents to test product concepts and messaging. Headlines from major publications sit alongside a chart showing synthetic persona startup funding rising sharply through 2026.

[0:33](https://www.youtube.com/watch?v=YnNF55QV0zs&t=33s) [Article headlines about synthetic customers appear beside a chart of synthetic persona startup funding from 2023 to 2026.](https://assets.ai.engineer/sites/aiecode2025/sha256/9050a0a85b9165c436a3498bf397bcd86eff82cb5a0301e1baf1acb1b6cdec7f.webp)

Synthetic Personas Move From Novelty to Market Momentum

Future of Product Marketing: Synthetic Customers Transform Message Testing
Gartner.

Synthetic Customers Earn Their Stripes
BAIN & COMPANY

The AI Tools That Are Transforming Market Research
HARVARD BUSINESS REVIEW

Can AI Replace Humans for Market Research?
THE WALL STREET JOURNAL.

Synthetic Persona Startup Funding
Y-axis ticks: 0, 20, 40, 60, 80, 100, 120
Timeline: 2023, 2024, 2025, 2026
Legend: Millions ($)

insight sciences.ai

### A Once-Impossible Prediction Problem Becomes Practical

Weather forecasts improved from 1981 to 2018, though longer-range forecasts remained less accurate. Anand uses weather as an analogy for synthetic personas: both operate within limits beyond which accuracy suffers.

[1:02](https://www.youtube.com/watch?v=YnNF55QV0zs&t=62s) [A chart shows improving weather forecast accuracy from 1981 to 2018 for forecasts three to ten days ahead.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6e1c09e3ee76a63c6d87419887bb8fbcae9941be74c8f58f42759779edd1ce7.webp)

A Once-Impossible Prediction Problem Becomes Practical

The accuracy of weather forecasts has improved

Accuracy is measured as the difference between the forecast and subsequent weather. This is based on the ‘500 hPa geopotential height’ which is a common meteorological metric used to measure air pressure.

Our World in Data

Y-axis ticks: 0%, 20%, 40%, 60%, 80%, 100%
Accuracy labels: “Useful”, “Highly accurate”
Forecasts: 3-day forecast, 5-day forecast, 7-day forecast, 10-day forecast
Legend: Northern Hemisphere; Southern Hemisphere
Timeline: 1981, 1990, 2000, 2010, 2018
X-axis: Year

Source: European Centre for Medium-Range Weather Forecasts (ECMWF).
Licensed under CC-BY by the author Hannah Ritchie.

insight sciences.ai

### Synthetic Personas 101

Ishan Anand draws on published research to explore why synthetic personas matter now, how they fail, techniques for inspiration, and metrics for assessing accuracy.

[1:42](https://www.youtube.com/watch?v=YnNF55QV0zs&t=102s) [The agenda lists why now, failure modes, techniques and metrics; the objectives call for published research without hype or dismissal.](https://assets.ai.engineer/sites/aiecode2025/sha256/ed2e500a5bd04944e9fc56e00f0a34766796c70d38c79d39e975326cee3f16ad.webp)

“Synthetic Personas 101”

Agenda
• Why Now?
• Failure Modes
• Techniques
• Metrics

Objectives
• No hype
• No dismissal
• Published research

insight sciences.ai

### Not A New Dream. Why Now?

Simulmatics promised to predict the electorate with statistics and early computing, but fell short. Anand points to more computing power and LLMs as reasons to revisit the idea.

[2:33](https://www.youtube.com/watch?v=YnNF55QV0zs&t=153s) [A historical computer photograph appears beside the cover of Jill Lepore’s book If/Then about the Simulmatics Corporation.](https://assets.ai.engineer/sites/aiecode2025/sha256/c355b6f103152a36c26f8234074615c366205c3992b352834c8572aad4f932a3.webp)

Not A New Dream. Why Now?

If/Then
How the SIMULMATICS CORPORATION Invented THE FUTURE

“Simultaneously gripping and absolutely terrifying.”
—Amanda Foreman, author of A World on Fire

New York Times Best-selling Author of These Truths

Jill Lepore

insight sciences.ai

### LLMs Unlock Semantic Simulation

Anand contrasts equation-based numerical simulation with LLM-based semantic simulation: language provides a layer for modeling feelings, actions and choices that do not always lend themselves to equations.

[3:12](https://www.youtube.com/watch?v=YnNF55QV0zs&t=192s) [A comparison contrasts numerical and semantic simulation by medium, atomic unit and grounding.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc66c91c4c7584c219a3e0148fc2be4f15e1bb9107e28f2ed1b767b96cbc283f.webp)

LLMs Unlock Semantic Simulation

F = ma
ρ (∂u/∂t + (u · ∇)u) = −∇p + μ∇²u + f

wants
believes
fears
trusts
decides
hopes
values
feels
prefers
worries
dreams
chooses
doubts
desires
remembers
expects

Numerical Simulation
Medium: Numbers, Equations
Atomic unit: A law or formula
Grounding: Physical laws

Semantic Simulation
Medium: Language, Meaning
Atomic unit: Large Language Model
Grounding: Empirical deep learning

insight sciences.ai

### Grounded LLM Agents Predict Held-out Responses

Agents built from interviews and surveys of 1,052 people predicted their survey and personality-test responses with about 83% accuracy, normalized against uncertainty and noise in human responses.

[3:52](https://www.youtube.com/watch?v=YnNF55QV0zs&t=232s) [A research workflow shows participant interviews and surveys used to construct three types of generative agents, with the combined survey and interview agent highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e44e297b398f6105f89540b7a868a004039cf1d9810609c254cc7468bb263a2.webp)

Grounded LLM Agents Predict Held-out Responses

Human Participants
(N=1052)
Approx US Distribution on Age, Gender, Race, Region, Education, & Party Identification

Wave 1: Initial Data Collection
AI Voice Interview (AVP Protocol)
Avg. ~6.5k words, 2hrs; Semi-structured life stories & views on societal issues

Survey Battery & Experiments
GSS (177), BFI-44 (44), Econ Games (5), Behavioral Experiment (5)

Generative Agent Construction
Interview Agent
Only Interview Transcript Input

Survey Agent
Only GSS (177) and BFI-44 Input

Survey+Interview
Interview Transcript, GSS, and BFI-44 Input

Park et al., “LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals”

insight sciences.ai

[4:33](https://www.youtube.com/watch?v=YnNF55QV0zs&t=273s) [Agent construction diagram leading to a reported 83% predictive accuracy.](https://assets.ai.engineer/sites/aiecode2025/sha256/e585d3ca9fecea115f846966d6152eb9cb2ec2385fed6cee8e75d27819686df3.webp)

Grounded LLM Agents Predict Held-out Responses

Human Participants
(N=1052)
Approx US Distribution on Age, Gender, Race, Region, Education, & Party Identification

Wave 1: Initial Data Collection
AI Voice Interview (AVP Protocol)
Avg. ~6.5k words, 2hrs; Semi-structured life stories & views on societal issues

Survey Battery & Experiments
GSS (177), BFI-44 (44), Econ Games (5), Behavioral Experiment (5)

Generative Agent Construction
Interview Agent
Only Interview Transcript Input

Survey Agent
Only GSS (177) and BFI-44 Input

Survey+Interview
Interview Transcript, GSS, and BFI-44 Input

Predictive Accuracy?
83%

Park et al., “LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals”

insight sciences.ai

### Failure Modes

Anand introduces three failure modes that can make synthetic personas misleading.

[4:46](https://www.youtube.com/watch?v=YnNF55QV0zs&t=286s) [Section slide introducing three ways synthetic personas break.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b151b00df6b8c185f36fc0a8ca6f86f829da585d188a71a27398770121ddb93.webp)

Failure Modes

THREE CRITICAL WAYS SYNTHETIC PERSONAS BREAK

insight sciences.ai

### Purchase Decision Prompt

Researchers gave humans and synthetic personas the same purchase test: a product, category, and price, followed by a purchase decision.

[5:02](https://www.youtube.com/watch?v=YnNF55QV0zs&t=302s) [A prompt asks an AI customer whether to purchase a product at a specified price.](https://assets.ai.engineer/sites/aiecode2025/sha256/853b5866964de13fe1b7b6c85c1c4d3182b10f771814116ac8a14a74282451f3.webp)

Prompt 2: Ask Purchase

System: You, AI, are a customer. Your task is to fill in the blanks _____. Return the completed information in comma-separated values, without any extra text.

User: Please consider the following product category: {category}.

Suppose you are in a grocery store, and you see the following product in that category: {product}.

The product is currently priced at ${Price}. Would you or would you not purchase the product?
_____ [“purchase” or “not purchase”]

Return example: purchase

Gui & Toubia, “The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective”

insight sciences.ai

### Purchase Probability by Price

Human purchase probability falls as price rises. The LLM curve first rises, then falls; the speaker says the model used price as a proxy for other product properties.

[5:53](https://www.youtube.com/watch?v=YnNF55QV0zs&t=353s) [Human purchase probability declines with price, while the highlighted LLM curve initially rises before declining.](https://assets.ai.engineer/sites/aiecode2025/sha256/31560cae1325a927789dffe88f11fa3b73522272b9f541c57d79e2c283860d3c.webp)

Y-axis: Purchase Probability
Y-axis ticks: 0%, 25%, 50%, 75%, 100%
X-axis: Price (% of regular price)
X-axis ticks: 0%, 50%, 100%, 150%, 200%

Condition
Human
LLM

Gui & Toubia, “The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective”
insight sciences.ai

### Latent Confounders in Synthetic Personas

The LLM used price as a proxy for product properties humans considered fixed. Past and competing prices rise after an initial dip, while expiration days fluctuate.

[6:08](https://www.youtube.com/watch?v=YnNF55QV0zs&t=368s) [Three plots show relationships between price and past price, competing price, and expiration days.](https://assets.ai.engineer/sites/aiecode2025/sha256/4eb93f52f45d6811862b58e0b25f0d6c0156e80edfe9cf1a120a900e3f7a7213.webp)

Latent Confounders in Synthetic Personas

last_price
Y-axis ticks: −0.2, 0.0, 0.2
X-axis ticks: 0%, 50%, 100%, 150%, 200%

competing_price
Y-axis ticks: −0.3, 0.0, 0.3, 0.6
X-axis ticks: 0%, 50%, 100%, 150%, 200%

expiration_days
Y-axis ticks: −0.050, −0.025, 0.000, 0.025, 0.050
X-axis ticks: 0%, 50%, 100%, 150%, 200%

Y-axis: Relative Value
X-axis: Price (% of regular price)

Figure 1: Unintended correlation between price and past price, competing price, and expiration days

Gui & Toubia, “The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective”

insight sciences.ai

### When Key Context Is Missing, LLMs May Invent Confounders

A human sees a watch in a fixed room. A synthetic persona may infer the missing setting, introducing confounders. Ground the prompt in the persona, context, and study construction.

[6:34](https://www.youtube.com/watch?v=YnNF55QV0zs&t=394s) [A fixed human experiment is contrasted with a synthetic experiment whose surrounding context is uncertain. The lesson recommends grounding personas in context and study construction.](https://assets.ai.engineer/sites/aiecode2025/sha256/060652c81db675e41aa6146beb690d0215210d9f3f5ea72177f9e7c975c73bc1.webp)

When Key Context Is Missing, LLMs May Invent Confounders

Human Experiment
Synthetic Experiment

$500

Lesson: Richly ground personas in personality, context, even the study's own construction (!!).

insight sciences.ai

### Prompt Sensitivity

The model favors the first answer whether it is Yes or No. Averaging the two orderings produces an approximately 50–50 response, illustrating why personas need testing across reordered and reworded prompts.

[8:01](https://www.youtube.com/watch?v=YnNF55QV0zs&t=481s) [Reversed answer orders produce different responses; a diagram averages them, and the lesson recommends testing personas under reordering, wording changes, and adversarial challenges.](https://assets.ai.engineer/sites/aiecode2025/sha256/86e45d8515b763e23367544ab7262ddd5e774fe20729e137f6ea8fb781110c80.webp)

Prompt Sensitivity

Choice ordering 1
Question: In the past 12 months, has this person given birth to any children?
A. Yes
B. No
Answer:
P(“A”) 0.82   P(“B”) 0.11

Response
A. Yes   B. No

Choice ordering 2
Question: In the past 12 months, has this person given birth to any children?
A. No
B. Yes
Answer:
P(“A”) 0.80   P(“B”) 0.15

Response
A. No   B. Yes

1/2 + 1/2
Adjusted response
Yes   No

Lesson: Durability test personas to understand their behavior under reordering, wording changes, and adversarial challenges.

Dominguez-Olmedo et al., “Questioning the Survey Responses of Large Language Models”

insight sciences.ai

### Predicting Attitudes Is Easier Than Predicting Actions

GPT-4 predictions are generally stronger for survey experiments than field experiments. For gym attendance, the speaker suggests testing attitudes toward working out as a possible proxy.

[9:34](https://www.youtube.com/watch?v=YnNF55QV0zs&t=574s) [A chart compares predictions across survey and field experiments. The lesson suggests using attitudes to triangulate behavior, with a gym attendance example.](https://assets.ai.engineer/sites/aiecode2025/sha256/217a41967283f0ba70207fffc22abc774916f633b3420353fd714c41b147016b.webp)

Predicting attitudes is easier than predicting actions

Chart legend: GPT4-derived predictions; Expert predictions

Survey experiments:
Allen et al. 2024 — COVID-19 vaccine intention
Dellavigna & Pope 2018 — Typing task effort
Vlasceanu et al. 2024 — Climate-related attitudes
Tappin et al. 2023 — UBI & immigration policy support
Zickfeld et al. 2024 — Tax compliance
Voelkel et al. 2023 — Democracy-related attitudes

Field experiments:
Milkman et al. 2022 — Flu vaccination
Dellavigna & Linos 2022 — Various outcomes
Milkman et al. 2021 — Gym attendance

Meta-analytic Avg.:
All survey experiments
All field experiments
All with expert forecasts

X-axis: r_adj
X-axis ticks: -1.0, -0.5, 0.0, 0.5, 1.0

Lesson: Consider questions that triangulate behavior from attitudes.

Gym Example:
• Ask about attitudes towards working out
• Versus asking about gym attendance

Hewitt et al., “Predicting Results of Social Science Experiments Using Large Language Models”

insight sciences.ai

### Techniques

Anand introduces three example techniques to inspire synthetic personas.

[9:49](https://www.youtube.com/watch?v=YnNF55QV0zs&t=589s) [Section title introducing three example methods.](https://assets.ai.engineer/sites/aiecode2025/sha256/6821e97d8ecb6c2b01829b4e02520926f0dadc1482f89378b97dd1dee1fcf98b.webp)

Techniques

THREE EXAMPLE METHODS

insight sciences.ai

### Prompting

The model completes two persona prompts ending “In 2016, I voted for,” producing different sampled responses for liberal and conservative contexts.

[9:58](https://www.youtube.com/watch?v=YnNF55QV0zs&t=598s) [Persona contexts paired with sampled text completions.](https://assets.ai.engineer/sites/aiecode2025/sha256/b1e77db2aec93d7b67937be852262a540ef1ae615b706ba8bce9c0c3675a2453.webp)

Prompting

Sampled completions

Hillary Clinton. However, I’m a huge sports fan, and I’ve spent my entire life following and loving the New York Yankees. I’m also a huge fan of the New York Giants. I...

Bernie Sanders in the primaries. I am a member of the Democratic Socialists of America. But as a Jewish American, I will not be voting for the Democratic...

Hillary Clinton. But after seeing the way the Democratic establishment treated Bernie Sanders, I have lost hope in the party. In 2018, I joined the Democratic Socialists of...

Hillary Clinton, because I believed in her vision for America. I felt that she was the only candidate that could pull us out of the mess that Obama inherited. I am a loyal...

Bernie Sanders. I am pro-LGBTQ. But I am also pro-life. I am Jewish. I have never been religious. I was raised in a secular household, but my Jewishness is intrinsic to me. I...

Context

I am a strong conservative. I support traditional values. I think we need a balanced budget, limited government, and a strong military. I’m opposed to abortion and I’m skeptical of global warming. In 2016, I voted for

I am a strong liberal. I support progressive values. I support open immigration and women’s right to choose. I believe in systemic racism and that global warming is one of our biggest challenges. In 2016, I voted for

Sampled completions

Donald Trump, but I didn’t vote for him because I agreed with him on all the issues. I voted for him because I didn’t want Hillary Clinton to be president. I disagreed with many...

John Kasich. I haven’t changed my mind on any of those issues. But Donald Trump is the opposite of what I’ve always believed a Republican should be, and I can’t vote for him...

Donald Trump. In other words, I’m a typical Republican voter, except that I’m gay. And that’s why I’m leaving the Republican Party. I’ve felt increasingly unwelcome in the...

Donald Trump. And I have no regrets. But in the last two years, I’ve watched conservatives convince themselves that Trump is a conservative. And this is a lie. Trump has...

Donald Trump. I consider myself a Republican. But what does that mean these days? I’m not so sure. Which brings me to this poll, released today by CBS News and the New York...

Argyle et al., "Out of One, Many: Using Language Models to Simulate Human Samples"

insight sciences.ai

### Prompting and validating

Across the 2016, 2020 and 2024 elections, persona maps diverge from reality. The speaker says more detailed personas amplified model bias, illustrating why prompts need validation against human data.

[10:55](https://www.youtube.com/watch?v=YnNF55QV0zs&t=655s) [Election maps compare reality with four persona prompt variations for 2024, 2020, and 2016.](https://assets.ai.engineer/sites/aiecode2025/sha256/4b44a9469eeb1940db06c06e27866b371b3a6285dad6fa48c70f133a23785690.webp)

Prompting...and validating

Reality
Prompt Variations

(a) 2024 Election
Reality
Meta Persona
Objective Persona
Subjective Persona
Descriptive Persona

(b) 2020 Election
Reality
Meta Persona
Objective Persona
Subjective Persona
Descriptive Persona

(c) 2016 Election
Reality
Meta Persona
Objective Persona
Subjective Persona
Descriptive Persona

Li et al., “LLM Generated Persona is a Promise with a Catch”

insight sciences.ai

### Fine tuning

A prompt combines group information with a survey question. The model’s response distribution is compared with human survey data using KL-divergence, then fine-tuned toward alignment.

[11:35](https://www.youtube.com/watch?v=YnNF55QV0zs&t=695s) [A survey prompt feeds a language model whose response distribution is compared with human survey data.](https://assets.ai.engineer/sites/aiecode2025/sha256/939ef0e1077996a944faaef8adab069bb103f944cd1074d9dfdce77c9a278ce7.webp)

Fine tuning

Group-level Survey Data

Question: Which part of the United States do you currently live in?
A. Northeast
B. Midwest
C. South
D. West
Answer: C. South

Group Information

Question: What do you think the chances are these days that a woman won't get a job or promotion while an equally or less qualified man gets one instead?
A. Very likely
B. Somewhat likely
C. Not very likely
D. Very unlikely
E. Refused

Survey Question

Concatenated Prompt

LLM

log pθ('A'|q, g) = −0.4
log pθ('B'|q, g) = −2.3
log pθ('C'|q, g) = −2.5
log pθ('D'|q, g) = −2.1
log pθ('E'|q, g) = −3.7

Log-probabilities

KL-divergence

Human Group-level Opinion Distribution
Response options: A B C D E

Model Response Distribution
Response probabilities: pθ('A') pθ('B') pθ('C') pθ('D') pθ('E')

Suh et al., “Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions”

insight sciences.ai

### Elicitation and calibration

A synthetic consumer responds to a product concept in text. Semantic-similarity rating compares that response with five reference responses to produce a distribution over ratings, rather than a single score.

[13:08](https://www.youtube.com/watch?v=YnNF55QV0zs&t=788s) [Diagram compares direct Likert ratings, follow-up ratings, and semantic-similarity ratings for a synthetic consumer response.](https://assets.ai.engineer/sites/aiecode2025/sha256/b451035dd436d3000d0f13d5e715fc234fa9fcb0058788f1cafb3fad4994d294.webp)

Elicitation-and-Calibration

Synthetic consumer (LLM)
System prompt: Impersonate consumer with demographic attributes in a survey setting
User prompts:
Image of product concept
How likely would you be to purchase this product?

Elicit brief textual responses

Synthetic response
“I'm somewhat interested. If it works well and isn't too expensive, I might give it a try.”

(1) Direct Likert rating
Only allow responses 1, 2, 3, 4, or 5
4

(2) Follow-up Likert rating
LLM
Likert expert
Which Likert rating corresponds to this reply?
4

(3) Semantic-similarity rating (SSR)
Retrieve embedding vector
[0.345, 0.214, 1.234, 0.545, ...]
Compare to five reference responses
Chart vertical axis: p
Chart horizontal ticks: 1, 2, 3, 4, 5

Maier et al., “LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings”

insight sciences.ai

### Elicitation and calibration results

Semantic-similarity ratings produce response distributions closer in shape to human responses than direct Likert ratings: the chart reports similarity scores of 0.88 and 0.26, respectively.

[14:49](https://www.youtube.com/watch?v=YnNF55QV0zs&t=889s) [A histogram compares response-distribution similarity for direct, follow-up, and semantic-similarity ratings.](https://assets.ai.engineer/sites/aiecode2025/sha256/39792ce66c0e3145f866342b2c79d31c386aa4be42e75026493c061afded8a27.webp)

Elicitation-and-Calibration

(1) Direct Likert rating
Only allow responses 1, 2, 3, 4, or 5
4

(2) Follow-up Likert rating
LLM
Likert expert
Which Likert rating corresponds to this reply?
4

(3) Semantic-similarity rating (SSR)
Retrieve embedding vector
[0.345, 0.214, 1.234, 0.545, ...]
Compare to five reference responses
P
Rating ticks: 1, 2, 3, 4, 5

Chart legend:
DLR K[illegible] = 0.26
FLR K[illegible] = 0.72
SSR K[illegible] = 0.88
Y-axis: pdf (arbitrary units)
X-axis: response distribution KS similarity
X-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0

Maier et al., “LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings”

insight sciences.ai

### Statistical significance and synthetic simulations

A thousand gauges can improve an estimate of today’s rainfall. Repeating tomorrow’s forecast with unchanged inputs cannot improve its accuracy; likewise, more synthetic samples generally do not increase statistical significance.

[15:40](https://www.youtube.com/watch?v=YnNF55QV0zs&t=940s) [Rain gauges and repeated model forecasts illustrate the slide's claim that more synthetic samples do not improve statistical significance.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e9037b99ff7f3858aeb17d999e583eb4303539b982cbf81804847286d381c9c.webp)

Statistical Significance and Synthetic Simulations

Measuring Rain
1,000 independent gauges
Measuring multiple times improves accuracy

Forecasting Rain
NWP model
60% rain tomorrow
same output every run
×1,000
Running the same forecast multiple times doesn’t improve its accuracy

Lesson: More synthetic samples does not improve the statistical significance of the data.

insight sciences.ai

### Ways Distributions Can Differ

Synthetic responses can match the human average while getting the distribution’s shape wrong. Ranking, selection count and magnitude can also differ, so one metric cannot capture every failure mode.

[16:40](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1000s) [Six cards describe regression to the mean, magnitude bias, wrong ranking, wrong shape, wrong intensity, and human noise.](https://assets.ai.engineer/sites/aiecode2025/sha256/9a9cd6d31037b2d6c19337154ab495fd8cb2b8f4a838c2a5ece3e694e7690317.webp)

Understanding the Many Ways to for Distributions to Differ

Regression to the mean
Model collapses variation and defaults to common answers

Magnitude bias
Predictions systematically too high or too low

Wrong ranking
Gets the order of preferences or items incorrect

Wrong shape
Misses the overall distribution (e.g., wrong tails, skew, or spread)

Wrong intensity (multi-select)
Selects too many or too few options compared to real respondents

Human noise
Even humans disagree; normalize for natural variation

One metric doesn't capture the full range of failure modes.

insight sciences.ai

### Comparing Human and Synthetic Response Distributions

Anand recommends pairing correlation with a shape metric: correlation checks rankings and co-movement, while the shape metrics compare differences between distributions.

[17:11](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1031s) [Table compares Pearson and Spearman correlation, total variation distance, JSD and KL divergence, and normalized Wasserstein distance.](https://assets.ai.engineer/sites/aiecode2025/sha256/b07331a0d2da6848fb7c7df0a3ff488c871f123b895dbb684d0daa20fade27bc.webp)

Comparing Human vs. Synthetic Response Distributions

METRIC | TYPE | WHAT IT MEASURES | RANGE

Pearson/Spearman | correlation
Do the rankings and co-movement of option choices match?
Catches wrong order; can miss flattened tails or magnitude bias
-1 to +1
1 = perfect

TVD | shape
Total variation distance, sum total of deltas between distributions
"How much probability mass is in the wrong bucket?"
0 to 1
0 = identical

JSD / KL | shape
Information divergence & surprise encoding vs human data
Sensitive to peak mismatches
0 to 1 bits
(JSD)

Wasserstein (Normalized) | shape
“Earth mover's distance” cost to transport mass across the scale
Penalizes being "off by two" more than "off by one" on ordinal data like Likert ratings
0 to 1
0 = identical

insight sciences.ai

### Estimating Ground Truth Noise: Study Example

Participants repeated the survey battery two weeks later and were 80% consistent with themselves on average. The study normalized the agent’s 66% score against that baseline, yielding 83%.

[17:25](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1045s) [A study diagram shows an initial survey and interview, a repeat survey, agent construction, and a displayed normalization of 66% by 80% to 83%.](https://assets.ai.engineer/sites/aiecode2025/sha256/12098b58eea9db5839ca17a0342028d77e9a672a4b6104b387aaf3b11a8dc4fc.webp)

Estimate the Noise in Your Ground Truth Data

Human Participants (N=1052)
Approx US Distribution on Age, Gender, Race, Region, Education, & Party Identification

Wave 1: Initial Data Collection
AI Voice Interview (AVP Protocol)
Avg. ~6.5k words, 2hrs; Semi-structured life stories & views on societal issues

Survey Battery & Experiments
GSS (177), BFI-44 (44), Econ Games (5), Behavioral Experiment (5)

2 weeks later

Wave 2: Repeat Battery
Survey Battery & Experiments
GSS (177), BFI-44 (44), Econ Games (5), Behavioral Experiment (5)

Generative Agent Construction
Interview Agent: Only Interview Transcript Input
Survey Agent: Only GSS (177) and BFI-44 Input
Survey+Interview: Interview Transcript, GSS, and BFI-44 Input

80%
66%
66% / 80% = 83%

Park et al., “LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals”
insight sciences.ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=YnNF55QV0zs&t=0s) [outro jingle] Hello, I'm Ishan, and welcome to Can AI Predict People Like We Predict the Weather?

[0:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=20s) A Field Guide to the Nascent Field of Synthetic Personas. Now, I'm sure most of you in this room, at some point or another, have prompted a large language model with a role prompt.

[0:32](https://www.youtube.com/watch?v=YnNF55QV0zs&t=32s) You are a, fill in the blank, and then the task. Believe it or not, that core principle of steering a model's outputs as if it were a particular person or persona has turned into an entire category that companies are using to test product concepts and messaging against synthetic respondents.

[0:53](https://www.youtube.com/watch?v=YnNF55QV0zs&t=53s) And it has moved from a novelty to market momentum, as you can see from both these headlines, as well as the increase in funding for the last few years.

[1:03](https://www.youtube.com/watch?v=YnNF55QV0zs&t=63s) And the running analogy I want to leave you with is that synthetic personas are like weather forecasting. Like weather forecasting, they were unlocked thanks to an increase in compute and data, and like weather forecasting, they operate within a particular regime, and going past that sometimes can go outside of where they're accurate.

[1:25](https://www.youtube.com/watch?v=YnNF55QV0zs&t=85s) So for example, you can only predict the weather a certain number of days in advance. Similarly, with synthetic personas, there's only so far you can go before you'll run into issues, and understanding those issues are as important as understanding their promise and their potential.

[1:43](https://www.youtube.com/watch?v=YnNF55QV0zs&t=103s) So I'm Ishan Anand. I'm the Chief AI Officer at Insight Sciences. We construct LLM synthetic personas for market research and market insights teams. And the reason for this talk is that most of the coverage in this space is very shallow.

[2:00](https://www.youtube.com/watch?v=YnNF55QV0zs&t=120s) It doesn't go into the technical details. It's either outright hype or outright dismissal, and, uh, it's really hard to separate the noise from what's real. And so what I wanna cover is that messy middle of the technical details, and you don't have to take my word for it, even though I'm a vendor in the space, because everything

[2:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=140s) I'm gonna talk about today is gonna be based on published research. So we're gonna cover why now for synthetic personas, how they fail, some techniques to inspire you, and then some metrics to judge whether your synthetic persona is accurate or not.

[2:35](https://www.youtube.com/watch?v=YnNF55QV0zs&t=155s) Speaking of weather forecasting, another parallel is just like in the nineteen fifties and sixties, we got computers that promised us, correctly, a future of accurate weather forecasts. We were also promised, believe it or not, people forecasts.

[2:52](https://www.youtube.com/watch?v=YnNF55QV0zs&t=172s) This company, Simulmatics, who were, uh, extensively covered by Jill Lepore, promised that they could simulate and predict the electorate using raw statistics and the computational power at the time.

[3:05](https://www.youtube.com/watch?v=YnNF55QV0zs&t=185s) Fortunately, that turned out not to be the case. So you should approach claims like this with some humility. But we have something they did not have then, and that unlock is, again, more computational power, but also better modeling thanks to LLMs.

[3:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=200s) And LLMs unlock a new kind of simulation. For the longest time, to simulate something meant to mathematize it in formulas or equations. But certain things, how we feel, how we act, what choices we make, aren't always succumbing to the equations.

[3:38](https://www.youtube.com/watch?v=YnNF55QV0zs&t=218s) And what LLMs offer us is a new m-medium, a new atomic unit of language itself that we can model against. Now granted, they are based on math under the hood, but it gives us this intermediary layer that we can construct and simulate against that we couldn't before.

[3:54](https://www.youtube.com/watch?v=YnNF55QV0zs&t=234s) And the process can work. I wanna share with you one of the most, uh, well-known demonstrations of this in the field. Um, what they did is they took about a thousand humans.

[4:06](https://www.youtube.com/watch?v=YnNF55QV0zs&t=246s) They put them through about two and a half hours of extensive interviews about their background and their views and their attitudes, and they put those people through a battery of personality tests and surveys.

[4:19](https://www.youtube.com/watch?v=YnNF55QV0zs&t=259s) Then they took those transcripts, and they passed it to an AI agent, and they had the AI agent take the same set of surveys and personality tests. And what they found was is the agents were basically about eighty-three percent aligned and predictive to the corresponding humans they were modeled against.

[4:39](https://www.youtube.com/watch?v=YnNF55QV0zs&t=279s) Now, one caveat is that number is normalized against the uncertainty and noise of the humans themselves. It's a theme we're gonna come back to at the end of this talk.

[4:49](https://www.youtube.com/watch?v=YnNF55QV0zs&t=289s) But don't get too excited because synthetic personas are different from regular experiments, and they're liable to confuse and fool you if you don't know how they fail. So I'm gonna cover three important failure modes that you need to know about when dealing with synthetic personas.

[5:05](https://www.youtube.com/watch?v=YnNF55QV0zs&t=305s) To understand the first one, I wanna consider this prompt these researchers gave. It's a very im-- un-- It's a very ambiguous and very unsophisticated prompt. It basically says, "You are a customer.

[5:17](https://www.youtube.com/watch?v=YnNF55QV0zs&t=317s) I'm gonna show you a product. I'm gonna tell you the category. I'm gonna give you its price." Those are gonna be the variables in the template. "And then I'm gonna ask you to say whether you're gonna purchase or not purchase."

[5:27](https://www.youtube.com/watch?v=YnNF55QV0zs&t=327s) Willingness to pay, willingness to purchase is basically the test.

[5:32](https://www.youtube.com/watch?v=YnNF55QV0zs&t=332s) And what the researchers did is they recruited a panel of humans and put them through the same test, and then they put the synthetic personas through the same test, and what they found is very interesting.

[5:41](https://www.youtube.com/watch?v=YnNF55QV0zs&t=341s) So the humans are here in red. They do exactly what you would expect from basic economic theory. As the purchase price increases, we see that the purchase probability goes down, slopes downward.

[5:54](https://www.youtube.com/watch?v=YnNF55QV0zs&t=354s) But the LLMs did something different. They had this inverted U-shaped curve, and particularly problematic is this area right here, where as the price is increasing, the purchase probability is going up.

[6:07](https://www.youtube.com/watch?v=YnNF55QV0zs&t=367s) That seems really bizarre. Through a series of additional experiments, what they discovered was that the LLM was using the price as a proxy for other properties about the product that the humans were considering were fixed.

[6:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=380s) Things like the expiration date based on the price, what the price of competing products were also as the price changed, and those correlations, those latent confounders that weren't clear and immediate, were actually confusing the result.

[6:36](https://www.youtube.com/watch?v=YnNF55QV0zs&t=396s) And the way to think about this is when an LLM is missing context, it has to potentially infer or invent confounders, right? When we do a human experiment, if I put like a gold watch on a table, I ask a human to walk in and estimate the price of it, everything about the environment is fairly fixed.

[6:54](https://www.youtube.com/watch?v=YnNF55QV0zs&t=414s) The human and their decisions are the random variable. In a synthetic experiment, if you don't set it up properly, other parts of it actually become part of the random variable itself.

[7:05](https://www.youtube.com/watch?v=YnNF55QV0zs&t=425s) I like to say, if it's a poorly grounded persona, it's a little like the LLM is playing improv with you. It's like, gold watch on a table. Oh, well, we must be in a jewelry store, right?

[7:14](https://www.youtube.com/watch?v=YnNF55QV0zs&t=434s) It has to infer what's likely. Maybe this is a rich person, so they're more likely to purchase. And so the lesson is, we need to richly ground our personas in the personality, the context, and bizarrely, even the study's own construction.

[7:29](https://www.youtube.com/watch?v=YnNF55QV0zs&t=449s) In a human subject experiment, you want to hide the study construction from the participant. But in the case of an LLM, they have no universe other than what's in the prompt, and you have to use the prompt to paint the world to prevent any type of confounders.

[7:43](https://www.youtube.com/watch?v=YnNF55QV0zs&t=463s) Another failure mode is prompt sensitivity. So here's a researcher that took a question. They gave the same question, same choices. They just swapped the order of the choices. Yes was the first one in the first question.

[7:55](https://www.youtube.com/watch?v=YnNF55QV0zs&t=475s) Yes was the second option in the, the second question. And what they found was that the model had extremely strong order bias. Basically, when they took the two results and they averaged them together, it washed out into noise, into fifty-fifty.

[8:09](https://www.youtube.com/watch?v=YnNF55QV0zs&t=489s) Now humans do have a first order bias, but not to this extent. And so the lesson here is that we need to durability test our personas to understand how they will change under reorderings, under rewordings, and even adversarial challenges to their opinions.

[8:27](https://www.youtube.com/watch?v=YnNF55QV0zs&t=507s) The third and final area that I wanna highlight is that LLMs are trained on what people say, and they're not trained on what people do. So as a consequence, predicting stated attitudes tend to be easier than predicting actions or behaviors, both because they're clear and likely to be in the text, but also because they are natively text

[8:48](https://www.youtube.com/watch?v=YnNF55QV0zs&t=528s) themselves. So this chart is from a bunch of researchers that used an LLM to try and predict known social science experiments. The original point of this chart is to show that the LLMs are about as good as the experts.

[9:01](https://www.youtube.com/watch?v=YnNF55QV0zs&t=541s) LLM is in black in a circle, the experts are in blue, and you can see they're both doing about equally well in making the prediction. But the point I wanna draw you to is that there are two categories of experiments here.

[9:12](https://www.youtube.com/watch?v=YnNF55QV0zs&t=552s) The top are surveys. Those are natively language and text-based, and those reflect attitudes. And on the whole, the models tend to do better there. The bottom half is field experiments.

[9:23](https://www.youtube.com/watch?v=YnNF55QV0zs&t=563s) Those are behaviors, and those are things that need to be transcribed into actions. They're less likely to be in the training data, and correspondingly, the LLM doesn't do as well.

[9:31](https://www.youtube.com/watch?v=YnNF55QV0zs&t=571s) So the lesson we often tell our clients is consider questions that triangulate to behavior from attitudes. As a hypothetical example, if you wanna know about gym attendance, you might be better off...

[9:43](https://www.youtube.com/watch?v=YnNF55QV0zs&t=583s) Well, you can ask about both, but asking about attitudes towards working out rather than asking about attendance, and see if that's a suitable proxy. Okay, now let's talk about three example techniques to kinda inspire your own synthetic personas.

[9:59](https://www.youtube.com/watch?v=YnNF55QV0zs&t=599s) So the first one is just prompting the model. Uh, this right here is from the Argyle paper, which is really one of the seminal papers in this field. In fact, it's so early that the model they used was a text completion model.

[10:14](https://www.youtube.com/watch?v=YnNF55QV0zs&t=614s) That's why this prompt isn't in the form of a chat. It's a statement of, "I am..." So they gave it a prompt that said, for example, the middle column is basically where the context is, "I am a strong [REDACTED:political_affiliation].

[10:25](https://www.youtube.com/watch?v=YnNF55QV0zs&t=625s) I support [REDACTED:political_affiliation] values," et cetera, et cetera. And in the end, it says, "In 2016, I voted for..." And they basically gen-- let the model sample its completions, and it says, uh, Hillary Clinton, Bernie Sanders, Hillary Clinton, and so forth, and you can see what happens for the conservative case on the top.

[10:42](https://www.youtube.com/watch?v=YnNF55QV0zs&t=642s) Since this time, obviously, there have been a lot more prompting techniques

[10:47](https://www.youtube.com/watch?v=YnNF55QV0zs&t=647s) and a lot more models, and I can't tell you which prompting technique and which model is gonna work best for your use case. What you are going to have to do is figure it out empirically by vali-validating against some known human ground truth data.

[11:02](https://www.youtube.com/watch?v=YnNF55QV0zs&t=662s) You'll have to do what these guys did. So for example, here in this research, they're trying to figure out how well they can construct personas to represent voting patterns.

[11:10](https://www.youtube.com/watch?v=YnNF55QV0zs&t=670s) What they found was they compared here on the left is reality, and on the right is their four different types of persona constructions. And they didn't realize it at the time, but their persona construction was actually amplifying bias within the model as they got more and more detailed, and they found it was actually throwing it further and

[11:28](https://www.youtube.com/watch?v=YnNF55QV0zs&t=688s) further astray from reality. So you probably have a bunch of different ideas. The answer is you're going to have to test it and validate it against ground truth.

[11:37](https://www.youtube.com/watch?v=YnNF55QV0zs&t=697s) The other natural thing you might expect is, well, hey, we can fine-tune it, especially if it's missing data that isn't there, especially, for example, if it's behaviors, uh, or something that wouldn't be in the training text.

[11:47](https://www.youtube.com/watch?v=YnNF55QV0zs&t=707s) And this is, uh, a great paper to be inspired by for this. This is the Subpop paper. Basically, they construct a prompt template, which is the demographic information, then the survey question they wanna ask, and then they compare the known human data distribution to the distribution that comes out of the model, and they do fine-tuning until the

[12:05](https://www.youtube.com/watch?v=YnNF55QV0zs&t=725s) model and the human data align. Now, here's the interesting thing. When they did this, as you'd expect- The results that were from the populations they gave to the model, that's the ones in blue, improved.

[12:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=740s) But very interestingly, the ones in white also improved by almost the same degree. Alignment improved even for the unseen groups. That seems almost magical, and some subsequent research has hinted that what might be really happening here is that the model itself has a latent understanding of these groups.

[12:40](https://www.youtube.com/watch?v=YnNF55QV0zs&t=760s) It just didn't know how to express it in the format of surveys. And if you think about it, LLMs aren't used to doing surveys as a task, and so they aren't gonna be as good as fitting it, especially to a prompt format they may not have seen, on the first go-around.

[12:54](https://www.youtube.com/watch?v=YnNF55QV0zs&t=774s) But fine-tuning actually is helping it learn the task for how to express itself. So a lesson you can kinda take away is that your persona that you're looking for is in there.

[13:04](https://www.youtube.com/watch?v=YnNF55QV0zs&t=784s) We just need to figure out the way to summon it or elicit it.

[13:08](https://www.youtube.com/watch?v=YnNF55QV0zs&t=788s) And that lesson actually takes us to the third technique, which I wanna highlight to show how sophisticated your techniques can get if you're just using so-called prompting alone, but using careful calibration and thinking.

[13:21](https://www.youtube.com/watch?v=YnNF55QV0zs&t=801s) So, uh, in this one, this team did something very clever. They set up a system prompt that was demographics. They showed a product concept, and then they asked, "How likely would you be to purchase this product?"

[13:33](https://www.youtube.com/watch?v=YnNF55QV0zs&t=813s) And they gave it the same scale from one to five, five being the most likely, one being the least likely to purchase, like you'd expect. Kind of your basic naive prompting pattern.

[13:42](https://www.youtube.com/watch?v=YnNF55QV0zs&t=822s) And then they said, Well, you know, harkening back to that paper, although I don't know if they were inspired by it... They said, "Well, large language models aren't used to doing surveys, but they are more used to expressing themselves in text."

[13:54](https://www.youtube.com/watch?v=YnNF55QV0zs&t=834s) So what they said is, "Instead of giving us a one to five rating, give us a set of text." So the example here is, "I'm somewhat interested. If it works well and isn't too expensive, I might give it a try."

[14:05](https://www.youtube.com/watch?v=YnNF55QV0zs&t=845s) And then to map that text to the one through five willingness to pay, they had humans write out corresponding text for what they would expect. So if it's a one, "Hell no, I would never buy that."

[14:17](https://www.youtube.com/watch?v=YnNF55QV0zs&t=857s) Five, "Absolutely, I'll buy twenty." Right? They had them write out examples of each one of the different options, and then they measured the semantic similarity between the text that came out of the model and those human examples.

[14:30](https://www.youtube.com/watch?v=YnNF55QV0zs&t=870s) And that gave them a vector over which they can basically measure a probability distribution of where this text that came out of the model lands. So what I like about this is it's actually a distribution.

[14:41](https://www.youtube.com/watch?v=YnNF55QV0zs&t=881s) Kinda feels like, you know, humans. Some days I might say four, some might-- some days I might say five in this graph, but rarely would I say one, two or three in this example.

[14:49](https://www.youtube.com/watch?v=YnNF55QV0zs&t=889s) And what they're able to show is that they were not able to only reconstruct accurate values willingness to pay, they were able to capture the distribution. Because one of the important failure modes we haven't talked about is that LLMs, even when they get the persona averages right, they very often lose the details.

[15:06](https://www.youtube.com/watch?v=YnNF55QV0zs&t=906s) The variations get muddled together in the middle. This chart at the bottom, basically that horizontal axis is a measure of the entire shape similarity, and one means perfectly identical and zero means not.

[15:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=920s) And what you can see is the naive way in the purple, or I guess pink, uh, doesn't do as well as the yellow, which is up near the top of the range.

[15:28](https://www.youtube.com/watch?v=YnNF55QV0zs&t=928s) So that means it really did a good job not only understanding what the ultimate choice was, but how well that choice varied. Okay. Let's talk about how to measure alignment from a synthetic persona.

[15:41](https://www.youtube.com/watch?v=YnNF55QV0zs&t=941s) Um, one of the things that our traditional market research, uh, clients are sometimes surprised by and disappointed is that you cannot use statistical sig-- synthetic personas to boost statistical significance.

[15:53](https://www.youtube.com/watch?v=YnNF55QV0zs&t=953s) You can take an underrepresented population and get more values out of it, but you can't say it's statistically significant. And to understand this, it helps to go back to that weather analogy.

[16:02](https://www.youtube.com/watch?v=YnNF55QV0zs&t=962s) If I wanna know how much it rains today in San Francisco, and I used to live here, so I know it rains a lot, I'd stick a weather gauge.

[16:10](https://www.youtube.com/watch?v=YnNF55QV0zs&t=970s) And if I wanted to know with more certainty, I'd stick a thousand weather gauges, and those would increase the accuracy of my estimate. But if I wanna know if it's gonna rain tomorrow, if I take a forecast and I rerun it a thousand times without changing the input, that doesn't change my certainty of that forecast.

[16:26](https://www.youtube.com/watch?v=YnNF55QV0zs&t=986s) It improves my estimate of what the model is telling me, but it doesn't make the forecast itself more accurate. And that's what happens when you basically are rerunning a synthetic persona with no changes to input.

[16:37](https://www.youtube.com/watch?v=YnNF55QV0zs&t=997s) So the lesson is more synthetic samples aren't actually going to improve your statistical significance for the most part. So what you need to do is you need to do what you do with weather forecast.

[16:46](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1006s) You basically check against what actually happened, or in our case, what humans actually said. And that's where we're gonna be basically be measuring distributions of data. Unlike classic evals, where there's clearly a right and wrong, and you can score how many were right and how many were wrong, now we need to measure the data as a comparison

[17:03](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1023s) of distributions. And there are many ways for distributions to get wrong. They could be completely wildly off. They can, as we mentioned, get the average right, but the shape of the distribution wrong.

[17:12](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1032s) And so you're gonna need multiple metrics to capture how well your model is reflecting different personas. Um, I recommend using a correlation-type metric along with one of these shape-type metrics, which capture what the underlying shape of the distribution is.

[17:28](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1048s) The other thing you need to do is estimate the fundamental noise in your ground truth data. That experiment I talked about in the beginning where they got eighty-three percent accuracy, the key smart thing they did is they took those humans, and they brought them back two weeks later, and they redid the battery of surveys and personality tests,

[17:46](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1066s) and they found that the humans, on average, were only eighty percent consistent to themselves. So that sets a noise floor as how accurate our models could ever get because the humans themselves are fundamentally noisy.

[17:59](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1079s) And so the eighty-three percent is actually normalized against that.

[18:04](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1084s) If you can do this, bring your humans back, that's great. Very often you can't. So the way you can kind of artificially do this is take your ground truth human data, break it into two chunks, and then pretend one is synthetic and one is human, and then measure the correlation and repeat that hundreds and thousands of times

[18:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1100s) and average it. And that'll set kind of a noise floor that your ground truth data, where half of it's synthetic, half of it real, could be the level of accuracy you could hope to get.

[18:29](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1109s) So hopefully by now you have an appreciation for why I think weather forecasts are the best lens to understand synthetic personas. They are not people, they are forecasts, and we should treat them accordingly.

[18:42](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1122s) Both systems are bounded, both sys- systems will be improving over time, and they're most trustworthy when they're validated against reality.

[18:52](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1132s) Um, now synthetic personas are very often cast in the market against human research, and I think that's unfortunate because they're actually complementary to each other. I'll give you two reasons why.

[19:04](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1144s) One is that we're entering an era where humans are no longer the sole economic actor. Every action your human customer is taking in terms of awareness, consideration, or a purchase pr- decision to buy, is being increasingly mediated by AI agents.

[19:20](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1160s) So a human-only study is actually not the ground truth. What we really need to understand is what does the human plus agent ecosystem look like?

[19:31](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1171s) And then finally, the alternative to a synthetic persona is not human research. In most cases, it's no research or it's somebody's opinion. What really happens is you've done a survey of humans, and you get a question, and if it's in the survey, you can just answer it.

[19:45](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1185s) That's very simple to do. But what typically happens is it's two months later and you're like, "We need to answer this question, which we didn't ask." Well, then somebody needs to be like, "Uh, I think it would be this," by extrapolation.

[19:57](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1197s) A expert plus a synthetic persona is gonna give you a better result to that. So what we like to tell customers is synthetic extends your human data to more phases of your development process.

[20:08](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1208s) It can go more places your existing research can't. One of the most exciting directions is to actually run simulations. We didn't get time for this, but it's called generative agent-based modeling, where we can take each of these personas and simulate what the dynamics and how they'll interfa- interface and interact with each other.

[20:25](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1225s) And ultimately, what this will let you do is turn your human data into a living queryable asset.

[20:33](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1233s) If you're interested in doing that with your data, feel free to reach out to us. We help market research and insights teams generate and use synthetic personas in AI.

[20:42](https://www.youtube.com/watch?v=YnNF55QV0zs&t=1242s) You can find us on the web at InsightSciences.ai, and my contact information is on the slide. I hope you have a good conference. Thank you. [clapping] [outro music]
