# Local Agentic Theory For Mobile Games — Shafik Quoraishee & Joanne Song, The New York Times

Shafik Quoraishee · Joanne Song

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/418t26CVz-w-local-agentic-theory-mobile-games) · [Video](https://www.youtube.com/watch?v=418t26CVz-w)

## Local Agentic Theory For Mobile Games

On-device agents could make mobile games more responsive and accessible, but their decisions must share memory, frame time, and battery with the game itself.

Familiarity with basic game loops and model inference is helpful; reinforcement learning and accessibility concepts are introduced as needed.

### What can a local agent do for a human-made game?

[0:32](https://www.youtube.com/watch?v=418t26CVz-w&t=32s)

Can an agent help determine whether a puzzle is solvable, understand how it plays, or support a player without generating the puzzle itself? That is the experimental territory Shafik Quoraishee and Joanne Song explore. Quoraishee establishes the boundary first: the New York Times puzzles discussed here are made by people, and he describes the published games as having no AI features, explicitly excluding WordleBot from that classification. The work concerns possible supporting capabilities, not an announcement of deployed game features.

The starting point is solvability and playability. Quoraishee’s earlier [Connections solver presentation](https://www.shafikquoraishee.com/talks) explored reinforcement learning in greater depth. Here, the question broadens from how to solve a particular game to what an intelligent system could do while running on the player’s own device.

### From Pac-Man’s ghosts to intelligence on the phone

[1:50](https://www.youtube.com/watch?v=418t26CVz-w&t=110s)

Game AI long predates language models. Pac-Man’s ghosts provide a familiar example: a **finite-state machine** selects behavior through explicit states and conditions. Events in the game change what the ghosts do. This is conditional, symbolic intelligence, with the rules authored in advance. The move toward local agents builds on that history while asking whether more capable decision-making can also live on a personal device.

![Slide titled “AI has always been part of games,” showing a Pac-Man maze with pellets and ghosts beside a connected state diagram.](https://ai.engineer/api/talk-essay-media/9e98a915f4bffbdb602a3663e30fa652270dc917002e9ee4231a78229842cef7)

Pac-Man gameplay alongside a ghost state-machine diagram.

A cloud-backed mobile application sends an inference request upstream and waits for the result. That adds service cost and a network round trip to the computation. Moving inference onto the phone removes that round trip; the device still has to perform the work, so local execution is an opportunity to reduce delay rather than a guarantee of instant decisions.

Local execution also changes where gameplay information needs to travel. If the system is configured appropriately, computation and the data it uses can remain inside the device’s security boundary. Information needed only for the immediate gameplay experience need not be sent upstream as telemetry.

The practical example is a subway tunnel. A game may still require connectivity for some features, but its local intelligence can continue operating when HTTP requests cannot get through. Keeping relevant features on the device also creates room for personalization: the game can tailor its behavior to the person using it without making every adaptation depend on a remote service.

### Training a player versus reasoning during play

[4:57](https://www.youtube.com/watch?v=418t26CVz-w&t=297s)

**AlphaGo** and **AlphaZero** anchor the reinforcement-learning history. Through repeated training iterations, a model becomes tuned to playing a game. [**EfficientZero**](https://arxiv.org/abs/2111.00210) and [**EfficientZero V2**](https://arxiv.org/abs/2403.00564) extend that direction through sample efficiency: learning with less game-interaction data. They are examples of progress in limited-data reinforcement learning, not evidence that the same models meet a phone’s inference budget. This training approach continues alongside the agent systems discussed here.

The useful distinction is where adaptation happens:

| Approach | What changes | How the game informs behavior |
|---|---|---|
| Reinforcement-learning training | Model weights | Repeated training cycles |
| The described language-model agent | Context and subsequent decisions | Observations and local tool calls |

In the first case, training changes the model itself. In the second, a language model reasons over the current game state through **in-context learning**, chooses actions, and uses device functionality through tools. The contrast concerns the execution loop; it does not imply that a language model could never have undergone reinforcement learning during training.

### Inside the Space Invaders agent

[7:03](https://www.youtube.com/watch?v=418t26CVz-w&t=423s)

The Space Invaders demonstration makes the loop concrete. The local framework controls a player that must shoot approaching ships while avoiding their bullets. Although reinforcement-learning players can perform the same task, Quoraishee identifies this prototype as an agentic system.

Its behavior follows four stages:

1. **Perceive:** inspect what is happening in the scene.
2. **Predict:** anticipate the attacking ships’ next moves and incoming threats.
3. **Decide:** determine whether a nearby ship or bullet calls for an attack or evasive action.
4. **Act:** execute the chosen action, then return to perception to inspect the resulting state.

The return path matters. An action does not complete the task; it changes the situation the agent must evaluate next.

![Slide titled “Inside the agent that plays the shooter,” with four stacked stages connected to an arcade-shooter game box by a return loop.](https://ai.engineer/api/talk-essay-media/501a91baa69355a27ce5632731f6deffeeb6800f81cb49026d1ee6dea29ce2ad)

The shooter agent’s perceive, predict, decide, and act loop.

### The agent shares the game’s resource budget

[8:09](https://www.youtube.com/watch?v=418t26CVz-w&t=489s)

Running that loop on a device means fitting more than model weights into memory. The agent also needs a history of prior states and a working set of plans or intermediate artifacts. Those may require compression, and the game still needs enough headroom to render and respond to the player. **Agent capacity is only useful if the game remains playable.**

At 60 Hz, a frame interval is approximately 16.7 milliseconds, rounded to 16 milliseconds in the talk. That interval is a deadline shared with rendering and other game work, not an entire budget reserved for inference or a measured planning latency. Agent execution that overruns the relevant budget can produce visible jank.

Energy imposes another limit. Repeated inference and planning consume battery, and the speakers describe mobile hardware as still evolving toward these workloads. Neural processing units provide specialized inference hardware, but the agent loop must still minimize unnecessary processing. A system that makes good decisions while rapidly draining the phone is not a satisfactory mobile design.

The proposed constraint graph balances these demands rather than optimizing one in isolation:

| Constraint | Resource being protected | Design pressure |
|---|---|---|
| Space | Weights, history, working state | Compress and preserve render headroom |
| Time | Responsive frames | Penalize disruptive overruns |
| Energy | Battery life | Limit agent processing |

The penalties need not be equally strict. Quoraishee describes allowing a softer penalty when a plan needs a little extra space while applying stronger pressure against time overruns that disrupt the user experience. The graph expresses a tradeoff policy, not simply three independent maxima.

### From gaze signals to crossword constraints

[10:29](https://www.youtube.com/watch?v=418t26CVz-w&t=629s)

Gaze estimation adds another input to the decision process. A convolutional neural network can estimate gaze and supply information that an agent translates into behavior. Richer sensing does not, by itself, supply better reasoning: Quoraishee separately raises base-model intelligence as a limit on complex game decisions. He mentions [ARC-AGI](https://github.com/fchollet/ARC-AGI) as a capability reference without reporting a model score. Its reasoning tasks are distinct from measuring mobile responsiveness or accessibility.

The Mini Crossword prototype shifts from reacting to moving threats to satisfying a set of linked constraints. Its **constraint-satisfaction graph** guides word placement so that the puzzle can be filled consistently. A candidate can be wrong; when placements conflict or the constraints become difficult to satisfy, the agent backtracks and tries to recover a valid layout. Quoraishee describes the graph as optimized for efficient revision, with natural-language information from the graph helping the agent choose how to backtrack.

These experiments sit within a wider landscape. [**SIMA**](https://deepmind.google/blog/sima-generalist-ai-agent-for-3d-virtual-environments/) is the named example of a generalist game agent; its research direction should not be confused with an established mobile deployment of this framework. Phone-based visual estimation supplies another capability, while AI-generated worlds create additional environments for agents to operate in. Together, these directions expand both what an agent can perceive and the kinds of worlds it may need to understand.

![Three cards labeled “One generalist agent, many games,” “Agents that see, on the phone,” and “The engine becomes a neural net,” each with a diagram.](https://ai.engineer/api/talk-essay-media/671f7148c66af90e0a537e1c922a628ac12c7cddd91e525dc99dffddb8fd9752)

Three research directions on the “Where we are now” slide.

### Accessibility needs more than an easy-mode toggle

[12:30](https://www.youtube.com/watch?v=418t26CVz-w&t=750s)

Joanne Song turns the same local-agent architecture toward the player. A fixed state model can offer a toggle or static menu, but it cannot necessarily respond when someone’s needs change during play. The design foundation is [WCAG 2.2](https://www.w3.org/TR/WCAG22/), organized around four requirements: an experience should be **perceivable, operable, understandable, and robust**. These provide a framework for evaluating interaction rather than treating accessibility as an isolated game setting.

A mobile crossword makes the failures tangible. Tiny grid cells can cause mis-taps. An active-cell highlight can be too difficult for a low-vision player to see. Manual logins can create cognitive barriers, and unforgiving timers can exclude people who need more processing time. Lowering puzzle difficulty does not repair any of those interface problems.

Song uses the proposed Bronze, Silver, and Gold levels in the [WCAG 3.0 Working Draft](https://www.w3.org/TR/wcag-3.0/) to motivate a broader range of assistance. The draft is not a finalized continuous scoring standard, and its proposed levels still include core requirements. The gameplay idea is to move beyond a single binary setting toward adjustable dimensions of support.

Two such dimensions are:

- **Input tolerance:** how forgiving the interaction is of physical constraints or imprecise input.
- **Step granularity:** how operations are divided within a sequence, allowing complex tasks to be broken into manageable steps or presented with more nuance.

The agent’s task is to adjust these dimensions together against the player’s live context. A useful intervention depends on their combination, not just turning every assistance setting upward.

### Sense friction, then repair the interaction

[14:58](https://www.youtube.com/watch?v=418t26CVz-w&t=898s)

On-device sensing could help identify where an interaction is becoming difficult. Gaze may indicate search friction, shaky taps may indicate trouble operating controls, and handwriting recognition can offer another input method. These are cues and alternatives, not definitive diagnoses of the player. The vision pipeline shown here connects gaze dwell, handwriting ink, and touch jitter through LiteRT to an agent.

![Slide titled “Computer-vision agents sense the player,” showing gaze dwell, handwriting ink, and touch jitter connected through LiteRT to an agent.](https://ai.engineer/api/talk-essay-media/37a70eeb40d3c71ea3349c6190b8ef81cd5548c31109813374f765751129e794)

Gaze dwell, handwriting ink, and touch jitter feed an on-device vision pipeline.

The next example turns observation into action: a simulated broken dialog traps a keyboard or switch user in a focus loop with no usable way back. The agent monitors the focus path, flags the problem, and supplies an exit locally. The defect is the missing exit, not focus cycling alone: the [accessible modal-dialog pattern](https://www.w3.org/WAI/ARIA/apg/patterns/dialog-modal/) intentionally contains focus while also providing dismissal and restoring focus afterward.

A small TypeScript example makes the distinction explicit. Repeated navigation is a reason to inspect the dialog; a repair is proposed only when that evidence coincides with no usable dismissal route:

```typescript
type DialogObservation = {
  focusPath: string[];
  hasUsableDismissal: boolean;
  returnFocusId: string;
};

type ExitProposal = {
  kind: "add-exit";
  controlId: string;
  returnFocusId: string;
};

function proposeExit(
  dialog: DialogObservation
): ExitProposal | null {
  const path = dialog.focusPath;
  const repeatedCycle =
    path.length >= 4 &&
    path[path.length - 4] === path[path.length - 2] &&
    path[path.length - 3] === path[path.length - 1];

  if (!repeatedCycle || dialog.hasUsableDismissal) return null;

  return {
    kind: "add-exit",
    controlId: "dialog-close",
    returnFocusId: dialog.returnFocusId,
  };
}

const proposal = proposeExit({
  focusPath: ["hint", "settings", "hint", "settings"],
  hasUsableDismissal: false,
  returnFocusId: "crossword-cell",
});
```

Here, `proposal` describes an exit to add and where focus should return; it does not mutate the interface. The repeated-pair check illustrates the observation-to-proposal boundary, rather than a complete detector for every keyboard trap.

The next intervention concerns target size. For someone with limited dexterity, a tiny control can turn an intended selection into a miss. Song describes an agent acting as a live layout auditor: it measures the interface, identifies inadequate controls, and dynamically resizes them. The layout changes to accommodate the person during interaction.

This produces a continuously adjusted relationship between assistance and challenge. Song frames them as ends of a dial that responds to the moment, with the broader goal of a device that senses frustration and responds usefully. In these experiments, responsive accessibility means changing the interaction itself, not merely observing that the player is struggling.

### What useful local agents still need

[16:42](https://www.youtube.com/watch?v=418t26CVz-w&t=1002s)

The remaining work starts with speed. Planning must respect the frame deadline already discussed so that assistance does not introduce stutter. It also needs prediction: before applying a layout change, the system should be able to anticipate what that change will do. Reacting to difficulty is only part of the problem; avoiding a disruptive intervention is another.

Long-term memory would let an agent learn one person’s habits and needs over time instead of treating every interaction as new. A shared game-state language would address a different limitation: portability. If games exposed a common representation, an agent could operate across releases without being rebuilt separately for each one.

Better chips must be paired with honest benchmarks that establish whether the agent actually improves the experience. The destination is not necessarily one centralized intelligence serving every player in the same way. It could be many small local agents, each running on a personal device and shaped by the individual it serves.

## Resources

- [Shafik Quoraishee's conference talks](https://www.shafikquoraishee.com/talks) — Speaker's talk archive, including the June 2025 Connections solver presentation.
- [Mastering Atari Games with Limited Data](https://arxiv.org/abs/2111.00210) — The original EfficientZero paper on model-based reinforcement learning with limited game interaction data.
- [EfficientZero V2](https://arxiv.org/abs/2403.00564) — Extends sample-efficient reinforcement learning to discrete and continuous control with visual and low-dimensional inputs.
- [EfficientZero implementation](https://github.com/YeWR/EfficientZero) — Official code for experimenting with the original EfficientZero algorithm.
- [SIMA: a generalist agent for virtual environments](https://deepmind.google/blog/sima-generalist-ai-agent-for-3d-virtual-environments/) — DeepMind's original introduction to an agent that follows language instructions using screen observations and keyboard or mouse actions.
- [Android rendering budgets and jank](https://developer.android.com/topic/performance/vitals/render) — Guidance for diagnosing slow frames and understanding rendering deadlines.
- [WCAG 2.2](https://www.w3.org/TR/WCAG22/) — Accessibility requirements covering interaction, perception, navigation and target sizes.
- [WCAG 3.0 Working Draft](https://www.w3.org/TR/wcag-3.0/) — Work-in-progress accessibility guidelines proposing Bronze, Silver and Gold conformance levels.
- [Accessible modal dialog pattern](https://www.w3.org/WAI/ARIA/apg/patterns/dialog-modal/) — Keyboard behavior, dismissal and focus-management guidance for modal dialogs.
- [ARC-AGI-1 tasks and testing interface](https://github.com/fchollet/ARC-AGI) — The original grid-reasoning benchmark, with task data and a browser interface for solving examples.

## Slides

### Local Agentic Theory for Accessible Mobile Games

The talk explores small on-device AI agents that change how easy or hard a game feels for each player, without cloud processing or fixed difficulty settings.

[0:18](https://www.youtube.com/watch?v=418t26CVz-w&t=18s) [Title slide describing on-device agents that adjust game difficulty for each player.](https://assets.ai.engineer/sites/aiecode2025/sha256/c29317d9b73173bec9a362c53c0ca12487e72fafa4f664c1eac8c2c7a8f42d8f.webp)

ON-DEVICE · EDGE AGENTS · GAME THEORY · ACCESSIBILITY

Local Agentic Theory for Accessible Mobile Games

Small AI agents that run on the phone and change how easy or hard a game feels, per player. No cloud, no fixed difficulty settings.

Shafik Quoraishee
Staff Games Engineer · CV & ML Research

Joanne Song
Games

Day 4 · Jul 2, 2026 · 2:50pm · Room [illegible]

Interface text:
Notes
2 / 32
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship

### Disclaimers

The presenters say NYT puzzles are made by people and its Games have no AI features. The work in this talk is personal, exploratory research, not a product.

[0:29](https://www.youtube.com/watch?v=418t26CVz-w&t=29s) [Three disclaimers state that puzzles are made by people, the games do not use AI features, and the presented work is experimental.](https://assets.ai.engineer/sites/aiecode2025/sha256/610da2938734ffbfa90dc7399397f670e27d0d5a3a1657707c910b6bcba2da45.webp)

BEFORE WE START

Disclaimers

Puzzles are made by people
The New York Times's game puzzles are all made by human constructors and editors.

No AI in the Games
The New York Times does not use AI features in its Games.

Experimental work
This is experimental work we are doing on our own. It is research, not a product.

This is personal, exploratory work. The experiments and views here are our own.

setting expectations up front

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
Ask Gemini
Work
NYT Bookmarks
XXW-Board
TODO
ViewFinder
Chapterizer
WorldCup
XXW
SlopDetection
AndroidAIClub
IntMaterials
CUNY
Mentorship
Notes
2 / 32

### A deeper dive: AI Engineer World's Fair 2025 talk

The speaker points to his 2025 World's Fair talk for a deeper look at building a Connections solver, including RL solvers.

[1:24](https://www.youtube.com/watch?v=418t26CVz-w&t=84s) [A slide introduces a 2025 talk with a video thumbnail about AI and game theory in Connections.](https://assets.ai.engineer/sites/aiecode2025/sha256/303f3be3f516b75f8cad7881cbe1a9e1a30c878f7d0c0aa31c53f96c86f171d8.webp)

INTRO · BACKGROUND

A deeper dive: my AI Engineer World’s Fair 2025 talk

A longer version of where these ideas come from, from my talk at the AI Engineer World's Fair, 2025.

Video thumbnail:
AI Engineer World’s Fair
Connections May 8, 2025
Create four groups of four!
COMMIT TO PAPER
AUTHOR, COMPOSE, PEN, WRITE
NEEDS FOR PLAYING YAHTZEE
CUP, DICE, PENCIL, [illegible]
YARD   CHOP
JOY   JONES
System 1
System 2
• Slow, logical reasoning (e.g., [illegible])
The New York Times
AI & Game Theory:
A CASE STUDY ON NYT'S CONNECTIONS

AI Engineer World’s Fair · 2025
the longer version of this talk

Interface text:
Local Agentic Theory For Acc...
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
Ask Gemini
Work
NYT Bookmarks
XXW-Board
TODO
ViewFinder
Chapterizer
WorldCup
XXW
SlopDetection
AndroidAIClub
IntMaterials
CUNY
Mentorship
Notes
Watch on YouTube
4 / 32

### AI has always been part of games

Pac-Man’s ghosts switch behavior as conditions change: grabbing a power pellet sends a chasing ghost into fright mode. The diagram also shows transitions among scatter, chase, fright and eaten.

[1:48](https://www.youtube.com/watch?v=418t26CVz-w&t=108s) [A Pac-Man-style maze appears beside a diagram of ghost states: scatter, chase, fright, and eaten.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5ba3f2c4f61945a8ab28e97afc6731eb29b17f502e6ffa0b0f2610323771e2f.webp)

AI has always been part of games

INTRO · GAMES & AI

Game AI isn't new. Pac-Man's ghosts ran a hand-coded finite-state machine in 1980, switching between chasing you, scattering to their corners, and fleeing when you grab a power pellet. Deep Blue beat Kasparov at chess in 1997. AlphaGo beat the world's best at Go in 2016. Today's systems learn whole games from scratch. Each was more capable than the last, and all of them were built to win.

GHOST STATE MACHINE
SCATTER
CHASE
FRIGHT
EATEN
Transitions: timer, power, timeout, pellet, eaten, home

one ghost, four states
red ghost · CHASE

Pac-Man, 1980 · ghosts run a finite-state machine
AI has been in games all along

Interface text:
Notes
5 / 32
Local Agentic Theory For Ac…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Work · Ask Gemini

### Run the model on the device, not the cloud

The diagram contrasts a shared cloud model with processing on individual phones. It labels the cloud round trip ~200 ms and local processing ~3 ms, illustrating the latency benefit of avoiding a network call.

[2:37](https://www.youtube.com/watch?v=418t26CVz-w&t=157s) [A slide compares cloud and on-device inference and highlights the speed benefit of local processing.](https://assets.ai.engineer/sites/aiecode2025/sha256/2e104420ea3caca5a04aa66ccccc31023559d929ae6afbd772b5e34ddcca25bb.webp)

INTRO · WHY ON-DEVICE

Run the model on the device, not the cloud

A local model runs entirely on the phone's own chip, with nothing sent to a server. For an agent meant to tune the game to each player, it is the only approach that scales: you cannot run an always-on agent for billions of people in a data center, but you can run it on the device they already own. Doing it locally also changes what the agent can do:

• Faster. A frame lands about every 16 ms; a round trip to a server can take 200.

CLOUD · then
EDGE · now
model
~200 ms · congested
~3 ms · local

cloud round-trip ~200 ms · on-device ~3 ms
local is the only thing that scales to billions

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
6 / 32

### Reinforcement learning: trained by trial and error

The slide compares DQN’s roughly 200M frames to reach human-level Atari with EfficientZero’s roughly 0.4M frames and two hours of play for superhuman performance: about 500× less data.

[5:00](https://www.youtube.com/watch?v=418t26CVz-w&t=300s) [Chart compares Atari training experience for EfficientZero and DQN, with a timeline of game-playing models.](https://assets.ai.engineer/sites/aiecode2025/sha256/0675af5be66aac4904d8b2fb8a367497525481ad2f65f9ec3c31e8119f2fb24d.webp)

AGENTIC AI · REINFORCEMENT LEARNING

Reinforcement learning: trained by trial and error

Reinforcement learning is how most game-playing AI is trained. The system plays, earns a reward when it does well, and nudges its own weights over millions of attempts until the play is good. It works, but it is data-hungry: DQN needed about 200 million frames to reach human-level Atari. EfficientZero is the efficiency leap, above-human Atari from roughly two hours of play, about 500× less data, with V2 pushing the scores higher in 2024. That drop in cost is what puts this kind of learner within reach of a phone.

game experience to reach human-level Atari play
≈ 500× less data
EfficientZero: ~0.4M frames · ~2 hrs · superhuman
DQN · 2015: ~200M frames · weeks of play
X-axis ticks, log scale: 100K, 1M, 10M, 100M, 1B

Timeline: AlphaGo ’16; AlphaZero ’17; MuZero ’19; EfficientZero ’21; EZ-V2 ’24; still advancing

AlphaGo → AlphaZero → MuZero → EfficientZero → V2
reinforcement learning · cheaper to train every year

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
8 / 32

### Trained for the game, or reasoning about it

The RL curve illustrates 3.00M training episodes; the agent side depicts reasoning at each step. The speaker explains that RL changes model weights, while an agent reasons about the game state.

[6:06](https://www.youtube.com/watch?v=418t26CVz-w&t=366s) [A learning curve for traditional reinforcement learning is contrasted with an agent reasoning about a situation.](https://assets.ai.engineer/sites/aiecode2025/sha256/7643fac96981d95e27ee5cd46534ba0ddc22299c050c5039f68c4362e2afc2c2.webp)

Trained for the game, or reasoning about it

AGENTIC AI · REINFORCEMENT LEARNING VS AGENTIC AI

Both can learn to play, and they borrow heavily from each other. The difference is how each gets good, and how it behaves once it's running.

TRADITIONAL RL
Y-axis: skill
X-axis: episodes
3.00M episodes to train

vs

AGENTIC AI
agent
situation
no training · reasons each step

train a policy over time · or reason over a trained model live
they overlap, but they behave differently

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
Ask Gemini
Work
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Notes
9 / 32

### Before it can adapt a game, an agent has to understand it

In a Space Invaders-style demo, an on-device agent fires at ships and dodges bullets. A diagram beside the game shows its perception, decisions and actions.

[7:04](https://www.youtube.com/watch?v=418t26CVz-w&t=424s) [An arcade shooter appears beside a diagram of an on-device agent's perception, decisions, and actions.](https://assets.ai.engineer/sites/aiecode2025/sha256/c74b8e7d9629e59961a336e3fb999a35f587707f100560d2f2edd1fc5b072ff9.webp)

AGENTIC AI · A LIVE AGENT, ON-DEVICE

Before it can adapt a game, an agent
has to understand it

A faster game than the crossword. On the phone, the agent watches an arcade shooter, predicts what's coming, and fires and dodges in real time. Same loop, watch then decide then act, just 60 times a second.

GAME · arcade shooter
SCORE 120
WAVE 1

THE AGENT · on-device
perceive
aliens
bullets
self
gaps
decide
act
fire

state
action

› track left
on-device · ~4 ms · 60 fps

watch → decide → act · about 4 ms each
it all runs on the device, with no cloud

Interface text:
Local Agentic Theory For Acc
05:00 - Google Search
storage.googleapis.com/projects-prototype/aiewf/presentation2.html
NYT Bookmarks
XXW-Board
TODO
ViewFinder
Chapterizer
WorldCup
XXW
SlopDetection
AndroidAIClub
IntMaterials
CUNY
Mentorship
Work
Ask Gemini
Notes
10 / 32

### Inside the agent that plays the shooter

Each frame, the on-device agent reads the board, predicts bullet paths, decides, and acts. It puts survival first: a dodge interrupts aiming. The slide estimates about 4 ms per loop.

[7:41](https://www.youtube.com/watch?v=418t26CVz-w&t=461s) [A flowchart shows the agent perceiving, predicting, deciding, and acting in a repeated loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/5421b508eceafc253cfe6096e5f3c54aa55d3669f3443057a87213db4a15b96b.webp)

AGENTIC AI · THE AGENTIC MODEL

Inside the agent that plays the shooter

One loop, repeated every frame. The same four jobs run in order, all on the device, nothing sent to a server.

The policy's priority is clear: stay alive first, then attack. A dodge always beats a shot, so the agent breaks off aiming the instant a bullet lines up.

GAME · arcade shooter

THE AGENT · on-device
PERCEIVE
read the board into a state
PREDICT
where each bullet is heading
DECIDE
survive first, then aim
ACT
move · fire · dodge

act on the game
one loop, every frame · ~4 ms each · on-device

perceive → predict → decide → act · every frame
one small loop, all on the device

Interface text:
Notes
11 / 32
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html

### Challenges

This section introduces three challenges: running on a phone, sensing the player without crossing a line, and keeping the game solvable.

[8:12](https://www.youtube.com/watch?v=418t26CVz-w&t=492s) [Section divider introduces challenges of running on a phone, sensing the player, and keeping the game solvable.](https://assets.ai.engineer/sites/aiecode2025/sha256/f23897383506829f87c762245cc0f8a7939daaad0d2f792d061f9f6814465c2e.webp)

PART ONE

Challenges

The hard parts of doing this for real: running on the phone, sensing the player without crossing a line, and keeping the game solvable.

03

Interface text:
Local Agentic Theory For Ac[illegible]
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
12 / 32

### Three budgets, all on the phone at once

The RAM budget reserves 42% for model weights and 14% for rendering. At 60 Hz, the speaker says agent planning must also fit within roughly 16 ms per frame while conserving battery.

[8:19](https://www.youtube.com/watch?v=418t26CVz-w&t=499s) [A device RAM budget panel divides memory among model weights, state history, planning, and render headroom.](https://assets.ai.engineer/sites/aiecode2025/sha256/0f4a7f4626ac52e40a3944bfe72e8b6bca31082a89cb0748cdfffcc06fdf2642.webp)

CHALLENGES · THE DEVICE BUDGET

Three budgets, all on the phone at once

Everything runs on the device, sharing it with the game. That puts the agent inside three hard limits at the same time: the space in memory, the time between frames, and the energy in the battery.

Space · memory in RAM
Model, recent-move history, and the live working set all share one limited pool.

DEVICE RAM BUDGET
model weights 42%
compressed state history 20%
working set / planning 24%
render headroom 14%

state history → compressed for cheap backtrack
ring

space · time · energy, shared with the game
on-device · nothing sent to a server

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
13 / 32

### Reading where the player is looking

A small on-device CNN turns camera frames into a rough screen region for the agent. Head pose, lighting and other factors limit precision; the slide says frames are discarded after processing.

[10:28](https://www.youtube.com/watch?v=418t26CVz-w&t=628s) [Slide illustrating gaze estimation from an eye and phone camera through an on-device model to a predicted screen region.](https://assets.ai.engineer/sites/aiecode2025/sha256/2dda483785dd76e55305071edaf144c0be433b05217f488af0787b618b1e1534.webp)

CHALLENGES · ESTIMATING GAZE

Reading where the player is looking

Where a player looks is a powerful, hands-free signal: what they're focused on, whether they've noticed a hint, whether they're lost or scanning. For someone who can't give precise input, gaze can stand in for a pointer entirely.

Estimating it on a phone is the hard part. A front camera is noisy and low-res, and gaze shifts with head pose, distance, lighting, and glasses, so without calibration you get a rough region, not a pixel. The agent works with that general area, every frame, on a small model, reading from the most sensitive sensor on the device.

eye
looking at the screen
frame

gaze model
small CNN · on-device · ~3 ms
x
y
frame is used, then discarded

predicted area
agent
looking: top-left

eye → camera frame → on-device model → agent
frames are processed here and discarded · never uploaded

Interface text:
Local Agentic Theory For Acc[illegible]
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
15 / 32

### The ceiling on reasoning

ARC-AGI-3 asks agents to find a new game’s rules without instructions. The slide says untrained people clear every level, and charts humans at 100% versus 0.4% for the best AI.

[10:51](https://www.youtube.com/watch?v=418t26CVz-w&t=651s) [Slide with a visual rule-finding puzzle and a chart showing the slide's reported human and best-AI scores.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6476289f6eae558242066d933afedfad96ea43542b0a99e1c770550d063d713.webp)

CHALLENGES · THE CEILING ON REASONING

Humans solve it. The best AI scores under 1%

ARC-AGI-3 drops an agent into a game it has never seen, with no instructions, and asks it to work out the rules and win. Untrained people clear every level; the best models score under 1%. That's the ceiling: our agents can search a defined problem like the crossword, but can't reason out a brand-new one the way a person can.

ARC-AGI-3 · find the rule, with no instructions
example
example
your turn
a person gets the rule in seconds

who can actually do it
this is the gap
100% — Humans
0.4% — Best AI

best models
Gemini 3.1 Pro 0.37%
GPT-5.4 0.26%
Opus 4.6 0.25%

ARC-AGI-3 · interactive games, no instructions given
humans 100% · the best models under 1%

Interface text:
Local Agentic Theory For Acc[illegible]
05:00 - Google Search
Ask Gemini
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
Work
NYT Bookmarks
XXW-Board
TODO
ViewFinder
Chapterizer
WorldCup
XXW
SlopDetection
AndroidAIClub
IntMaterials
CUNY
Mentorship
Notes
17 / 32

### Solving a crossword with backtracking

A crossword agent checks HARE against its crossing words using a constraint graph. The demo shows 3 of 10 words placed and zero backtracks; the agent backs up when a choice conflicts.

[11:19](https://www.youtube.com/watch?v=418t26CVz-w&t=679s) [Crossword demonstration showing a partially filled grid, the candidate HARE, and a constraint graph.](https://assets.ai.engineer/sites/aiecode2025/sha256/359561fd46a76427c82308c3d0e3de449e2962d12a2d5ada405d74f3be85a8e4.webp)

CHALLENGES · SOLVING WITH BACKTRACKING

4 down A fast cousin of the rabbit

Crossword grid:
■ H U S H
V □ □ □ A
E □ □ □ R
A □ □ □ E
L □ □ □ ■

HARE fits every crossing
backtracks so far · 0

across and down spell different words · every white square has to satisfy one of each at once

CONSTRAINT GRAPH
1A, 5D, 4D, 6A, 1D, 5A, 2D, 7A, 3D, 8A
now trying HARE
words placed 3 / 10

pick a word · check the crossings · back up on a conflict
a constraint-satisfaction search, running live

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
Ask Gemini
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Notes
17 / 32

### Where we are now

Generalist agents such as Google SIMA transfer across games, while on-device vision agents read the screen and act. The speaker also points to AI-generated worlds as a third frontier.

[11:59](https://www.youtube.com/watch?v=418t26CVz-w&t=719s) [Slide showing cards for generalist game agents and on-device vision agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/4dcb62c43a3cecaf7a9119c83ef2077f523dc8ebce78b33372516eee8c24a2ed.webp)

Where we are now
THE STATE OF THE ART · WHERE THE FIELD IS HEADING

Three research frontiers are moving fast, and all three point at the same place: capable game agents that live on the device.

A · FOUNDATION MODELS FOR GAMING
One generalist agent, many games
e.g. Google SIMA
agent
language in · actions out
Follows plain-language instructions and transfers across titles it was never trained on.

B · LOCAL VISION-AGENT FRAMEWORKS
THIS TALK
Agents that see, on the phone
on-device · real time
perceive · decide · act · on-device
Small vision models read the screen and the player, and act every frame, with no cloud.

generalist agents · on-device vision · neural game engines
three frontiers, converging on the phone

Interface text:
Local Agentic Theory For Acc
05:00 - Google Search
Ask Gemini
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Work
18 / 32

### Accessibility

Joanne introduces accessibility by asking how a system can serve everyone. She argues that fixed menus and toggles cannot respond to players’ real-time needs.

[12:25](https://www.youtube.com/watch?v=418t26CVz-w&t=745s) [Part two title slide introducing accessibility, computer vision, and systems that serve everyone.](https://assets.ai.engineer/sites/aiecode2025/sha256/055bb6e2afe9e6adeca5d223d134cc8ffe1dfe59834a4ff52743f57909d35ca8.webp)

PART TWO OF TWO
Accessibility

Why all of this matters. The accessibility rules we build on, the computer vision that reads the player, and one system that serves everyone.

04

Interface text:
Notes
19 / 32
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/aiewf/presentation2.html
Ask Gemini
Work
NYT Bookmarks
XXW-Board
TODO
ViewFinder
Chapterizer
WorldCup
XXW
SlopDetection
AndroidAIClub
IntMaterials
CUNY
Mentorship

### Accessibility criteria games can miss

Tiny crossword cells invite mis-taps, invisible focus hides the active cell from low-vision players, and login hurdles and rigid timers add cognitive barriers.

[13:19](https://www.youtube.com/watch?v=418t26CVz-w&t=799s) [Six cards cover target size, dragging movements, focus appearance, redundant entry, authentication and timing.](https://assets.ai.engineer/sites/aiecode2025/sha256/289863293513a9141fd651ea9c8a76fba155c65a23b6b269dc204b6a0d6dcc8e.webp)

ACCESSIBILITY · WHERE GAMES KEEP FAILING

The 2.2 criteria games quietly break

SC 2.5.8
Target Size
Interactive targets must be ≥ 24×24 CSS px. Dense puzzle grids and tap controls routinely miss this.
MOTOR

SC 2.5.7
Dragging Movements
Every drag needs a single-pointer alternative. Tile-drag and swipe mechanics often have none.
MOTOR

SC 2.4.13
Focus Appearance
Keyboard and switch focus has to be clearly visible and big enough. In most game screens you cannot see it at all.
LOW VISION

SC 3.3.7
Redundant Entry
Don't make players re-enter what they already gave. Timed re-prompts punish cognitive load.
COGNITIVE

SC 3.3.8
Accessible Authentication
Logging in or resuming should not require a memory or puzzle test, which is a barrier for some players.
COGNITIVE

SC 2.2.x
Timing & Pressure
Time limits must be adjustable. Fixed countdowns exclude motor and processing-speed differences.
MOTOR

these aren't rare cases · they're the core of the game
one fixed setting can't meet all of them at once

Interface text:
Local Agentic Theory For Acc[illegible]
05:00 - Google Search
storage.googleapis.com/projects-prototype/aiewf/presentation2.html
Ask Gemini
Work
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Notes
22 / 32

### A proposed graded accessibility scale

The slide contrasts WCAG 2.2 pass/fail checks with draft WCAG 3.0’s proposed Bronze, Silver and Gold scores, linking the shift to a sliding scale for gameplay.

[13:50](https://www.youtube.com/watch?v=418t26CVz-w&t=830s) [A slide compares WCAG 2.2 pass and fail symbols with a proposed WCAG 3.0 bronze-to-gold gauge.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ce1922a013da15f5edbc066eb0a02642fc4f59985b61c41b25a09ec1ba54e74.webp)

ACCESSIBILITY · WHERE THE STANDARD IS HEADING

Even the standard is moving to a sliding scale

The next version, WCAG 3.0 (still a draft as of March 2026), drops pass/fail for graded scores, Bronze, Silver, Gold, across about 174 requirements. Accessibility becomes a score on a scale, not a yes/no checkbox. The same shift we're making for gameplay.

WCAG 2.2
binary criterion
pass
fail

WCAG 3.0
graded outcome
Bronze
Silver
Gold
0.34

pass/fail → graded score
final version expected around 2028 to 2030 · 2.2 is the law today

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks
XXW-Board
TODO
ViewFinder
Chapterizer
WorldCup
XXW
SlopDetection
AndroidAIClub
IntMaterials
CUNY
Mentorship
Ask Gemini
Work
Notes
23 / 32

### Player-context accessibility dial

The demo uses a draggable player-context dial to illustrate how six settings can vary with a player’s needs, from assistance to challenge.

[14:09](https://www.youtube.com/watch?v=418t26CVz-w&t=849s) [A player-context slider sits above six bars for input tolerance, step granularity, distractors, hints, time pressure and on-screen activity.](https://assets.ai.engineer/sites/aiecode2025/sha256/7dfb636a262a57a483182f44c2d696273e9728b7f325b95d010e3c7ada239aff.webp)

ACCESSIBILITY · THE DIAL · DRAG THE NODE

player context = 0.60
NEEDS ASSISTANCE
balanced
HIGH SKILL

input tolerance SC 2.5.8  0.40
hint density SC 3.3.x  0.37
step granularity SC 3.2.x  0.40
time pressure SC 2.2.1  0.60
distractor count  0.63
on-screen activity  0.60

each setting adjusted on its own · live
cool = help · warm = challenge · tags map to WCAG

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
24 / 32

### Computer vision and player context

Gaze dwell, handwriting ink, and touch jitter feed an on-device LiteRT model that supplies features to the agent. The slide connects these signals to search friction, shaky controls, and writing instead of typing.

[14:55](https://www.youtube.com/watch?v=418t26CVz-w&t=895s) [Gaze dwell, handwriting and touch jitter feed an on-device vision model that supplies features to an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/37ab282b03963c78d725d12b7ca3f2e2b2c99c54f75b3e0dd9d1756e7d030901.webp)

ACCESSIBILITY · READING REAL CONTEXT

Computer-vision agents sense the player

Vision models on the phone tell the agent real things about the player. Eyes resting on a spot can mean searching; shaky taps can mean struggling with the controls; handwriting recognition lets them fill a grid by writing instead of typing. All on the device.

gaze dwell
handwriting ink
touch jitter
LiteRT
on-device CV
~3 ms
agent
features

gaze · writing · shaky taps → signals → agent
about 3 ms · nothing sent to the cloud

Interface text:
Notes
25 / 32
Local Agentic Theory For Acc[illegible]
05:00 - Google Search
Ask Gemini
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
Work
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship

### When focus gets trapped, the agent clears it

In a simulated sign-in dialog, keyboard focus cycles without an exit. The agent can detect the loop and add an escape route on the device.

[15:19](https://www.youtube.com/watch?v=418t26CVz-w&t=919s) [A phone diagram shows focus cycling inside a sign-in dialog.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ea5d326b599d45e0e9f91bee67d53a9c035702ba621c82218bbfbda4d9426ad.webp)

ACCESSIBILITY · CLEARING A KEYBOARD TRAP

When focus gets trapped, the agent clears it

For someone using a switch or keyboard instead of touch, focus order is everything. A broken dialog can trap it: tab, and focus just cycles inside, with no way back to the screen. WCAG calls this a keyboard trap, and it stops the person cold.

The agent watches focus order on the device, notices when focus can't escape, and restores a way out, so they keep moving through the app.

Home
Search
Sign in
email
OK
Messages
Settings

agent
Tab, and it just cycles back

tab, tab, stuck · the agent gives focus a way out
WCAG 2.1.2 · no keyboard trap

Interface text:
Local Agentic Theory For Acc
05:00 - Google Search
storage.googleapis.com/projects-prototype/aiewf/presentation2.html
NYT Bookmarks
XXW-Board
TODO
ViewFinder
Chapterizer
WorldCup
XXW
SlopDetection
AndroidAIClub
IntMaterials
CUNY
Mentorship
Ask Gemini
Work
Notes
26 / 32

### When a control is too small to hit, the agent grows it

The agent measures the live layout and enlarges and spaces undersized controls. The illustrated targets are 48 px, compared with the slide’s 24 px WCAG minimum.

[15:46](https://www.youtube.com/watch?v=418t26CVz-w&t=946s) [Player controls are shown as enlarged, spaced targets labeled 48 pixels.](https://assets.ai.engineer/sites/aiecode2025/sha256/9cbf2befc74880998808210bedaeb517641ce5b205b9bc6d0b4cc6c4210e5828.webp)

ACCESSIBILITY · TAP TARGETS ON A MOVING UI

When a control is too small to hit, the agent grows it

Dynamic layouts shift and reflow, and controls can end up tiny or crammed together. For someone with a tremor, limited dexterity, or large fingers, that turns a simple tap into a miss. WCAG sets a minimum target size for exactly this reason.

The agent measures the live layout, finds targets below the minimum, and resizes and re-spaces them on the fly, growing the area you can actually hit, with no rebuild.

Player controls
48 px
WCAG 2.5.8 · 24 px minimum
[illegible] now 48 px with room to spare, easy to hit

too small to tap · the agent makes the target bigger
WCAG 2.5.8 · target size minimum

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/aiewf/presentation2.html
Ask Gemini
Work
NYT Bookmarks
XXW-Board
TODO
ViewFinder
Chapterizer
WorldCup
XXW
SlopDetection
AndroidAIClub
IntMaterials
CUNY
Mentorship
Notes
27 / 32

### Let the system tune the game as you play

The diagram presents agents pulling toward assistance and challenge as a tug-of-war, with the system moving along one dial to match the moment.

[16:11](https://www.youtube.com/watch?v=418t26CVz-w&t=971s) [A diagram shows agents pulling toward assistance and challenge on a shared dial.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf6692bcec8da62d7689f5b35a7a2ed8dd4c86a4e4a8ebb6e8bdd3749336335c.webp)

Let the system tune the game as you play

ACCESSIBILITY · THE SHIFT

Instead of fixed modes, the agents keep adjusting the game while you play. Help and challenge become two ends of one dial, and the system slides along it to match the moment.

◀ agents pulling toward assistance
agents pulling toward challenge ▶

one dial, not fixed modes
one scale, always adjusting

Interface text:
Local Agentic Theory For Acc
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
28 / 32

### What local agents still need to understand games

The radar chart places today’s capabilities inside the needed range across six dimensions. The slide calls for faster reasoning, predictive models, personal memory, shared game state, better chips and testing.

[17:19](https://www.youtube.com/watch?v=418t26CVz-w&t=1039s) [Five development needs appear beside a radar chart comparing current and needed capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/d325618a08e325e27f94d0a10e0049611f41dd3e152c1a48c97d91d15de858bf.webp)

What local agents still need to understand games

CONCLUSION · THE ROAD AHEAD

• Thinking fast enough. Small models that plan and decide within a single frame's 16-millisecond budget.
• Models that predict the game. So the agent can see what a change will do before it makes it.
• Memory that lasts, per player. Learning one person's habits and needs over time, on the device, which is what makes real accessibility possible.
• A shared way to describe a game. A common format for game state, so one agent works across many games instead of being rebuilt for each.
• Better chips, and honest testing. Faster AI chips, and real ways to measure whether the agent actually improved things.

Chart axes: in-frame reasoning; world models; player memory; state access; edge silicon; a11y evals
Legend: today; needed

the gap between today and what games need
small models · world models · memory · shared formats · testing

Interface text:
Local Agentic Theory For Acc[illegible]
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
30 / 32

### From one giant AI to billions of small ones

The crossword agent’s watch–decide–act loop becomes a model for billions of personal agents, each running locally and shaped by its user. A field of glowing points pictures that future, with one game highlighted.

[17:25](https://www.youtube.com/watch?v=418t26CVz-w&t=1045s) [A field of glowing points illustrates many small agents running on personal devices.](https://assets.ai.engineer/sites/aiecode2025/sha256/505d3322423e1ddef4158efc3ff24b4690c54a3f13ce9e4336c316bea93d6df3.webp)

CONCLUSION · WHERE INTELLIGENCE LIVES

From one giant AI to billions of small ones

We picture general AI as one enormous mind in a data center. The more capable these agents get, the less that holds. An agent that watches you, learns your habits, and acts for you has to be private and personal, which means it has to be yours, on your own device.

The crossword agent is the narrow version: one small loop, watch, decide, act, tuned to a single player. Scale it up and out, and the future of intelligence looks less like one giant brain and more like billions of small ones, each capable enough to help, each running locally, each shaped by the person it belongs to.

Diagram labels: this game; every device · every player

the basic loop stays the same as it grows
private · everywhere · personal

Interface text:
Local Agentic Theory For Acc…
05:00 - Google Search
storage.googleapis.com/projects-prototype/[illegible]/presentation2.html
NYT Bookmarks · XXW-Board · TODO · ViewFinder · Chapterizer · WorldCup · XXW · SlopDetection · AndroidAIClub · IntMaterials · CUNY · Mentorship
Ask Gemini · Work · Notes
31 / 32

## Full transcript

[0:00](https://www.youtube.com/watch?v=418t26CVz-w&t=0s) [upbeat music] Awesome. So yes, as, uh, a great introduction, uh, took us to, we are gonna be talking about local agentic theory for accessible mobile games.

[0:21](https://www.youtube.com/watch?v=418t26CVz-w&t=21s) And we realize this is the graph track, but there are graphs here in this presentation, so, um, fret not if you're missing them. Uh, so, uh, yeah. So a few disclaimers before we continue.

[0:32](https://www.youtube.com/watch?v=418t26CVz-w&t=32s) Um, since we, we do work at The New York Times, and there's a few things we have to say about, um, our puzzles, AI, et cetera. Uh, our puzzles are made by people.

[0:40](https://www.youtube.com/watch?v=418t26CVz-w&t=40s) They're not made by AI, so, um, that's just a, a thing that we... is true, and it'll always be true. Um, there's no AI in the games themselves, so the- there's no AI features.

[0:51](https://www.youtube.com/watch?v=418t26CVz-w&t=51s) Um, you might have seen WordleBot. That's not an AI feature. So all our games are, are pretty much AI free. Um, and the work the, that we're gonna be talking about now is experimental work.

[1:03](https://www.youtube.com/watch?v=418t26CVz-w&t=63s) So it's stuff that we look at in terms of solvability and other, uh, capabilities of agents in the space that we can potentially take advantage of on a local device to, to do, uh, other things and actually generate games more so things about playability and other kinds of, um, features we can, uh, utilize agents for in an

[1:22](https://www.youtube.com/watch?v=418t26CVz-w&t=82s) intelligent way. So introdu... a little bit of an introduction. So I also, um, if anyone has, uh, seen, uh, the last year's World's Fair talk that I did on Connections, there's a similar, um, vibe to this.

[1:35](https://www.youtube.com/watch?v=418t26CVz-w&t=95s) Uh, we-- I did a deeper dive into how a solver can be built for, uh, that game and, um, if you're interested, you can take a look. It's on YouTube, and, uh, it has a lot of interesting deeper dive specifics for, uh, RL solvers, which I'll go into in a bit as well.

[1:50](https://www.youtube.com/watch?v=418t26CVz-w&t=110s) So little bit of a history. Um, everybody here, and just in case you aren't familiar with AI and gaming, which, um, for, you know, it's not the most common topic here at, at the World's Fair right now.

[2:02](https://www.youtube.com/watch?v=418t26CVz-w&t=122s) But, um, if you, uh, are familiar with the history of it, uh, AI has been in games for a long time, and, uh, since this is... since the eighties, uh, it's been established as a core pattern in games like Pac-Man.

[2:15](https://www.youtube.com/watch?v=418t26CVz-w&t=135s) Um, this is a simple version of an AI called a finite state machine, and if anybody-- everyone's familiar with a finite state machine here? Probably. Or I would imagine enough people are.

[2:25](https://www.youtube.com/watch?v=418t26CVz-w&t=145s) It's a basic AI system which tells, uh, the ghosts in the game what to do when Pac-Man gets a, a, a, a dot or not. Um, but basically, it is a conditional kind of symbolic AI.

[2:37](https://www.youtube.com/watch?v=418t26CVz-w&t=157s) Um, and then moving forward, we'll talk more about RL and the advancements of where that went to. But firstly, um, the core thesis of what we're gonna really focus on is the model of running the devices, run-running AI on our local devices.

[2:54](https://www.youtube.com/watch?v=418t26CVz-w&t=174s) So, uh, most AI, um, infrastructure today runs really on the cloud, and, uh, if you're running a mobile application, um, most practical scenarios involve cloud architecture. And, uh, that, that essentially is expensive because you have AI calls and inference calls that you have to make upstream, and then they have to come to your device.

[3:13](https://www.youtube.com/watch?v=418t26CVz-w&t=193s) So there's things like latency, et cetera. So, um, in an ideal world, when we evolve to that, uh, place, um, we can, uh, offload a lot of the intelligence onto a device.

[3:24](https://www.youtube.com/watch?v=418t26CVz-w&t=204s) What that does is make a lot of the call, uh, make a lot of the calls faster when we get to that world because you're not doing, uh, a round trip to the cloud to get that information.

[3:33](https://www.youtube.com/watch?v=418t26CVz-w&t=213s) So you can, uh, actually compress that time to the time within the device frame computation. Um, the-- it's private too, so your AI computation is withon-- is, is within the device itself.

[3:46](https://www.youtube.com/watch?v=418t26CVz-w&t=226s) And if it's set up properly, it won't leave the device because there's some things that for the gameplay experience that you don't actually need it to leave the device.

[3:53](https://www.youtube.com/watch?v=418t26CVz-w&t=233s) Local computation is the way to go, and it stays within your confined security zone, uh, locally, and, uh, it's not needed upstream for telemetry or anything like that. Um, and then it should work anywhere.

[4:04](https://www.youtube.com/watch?v=418t26CVz-w&t=244s) So one thing is that games, uh, you know, some, a lot of games rely on the internet and, and many... they, they will for a, a live con- connection point.

[4:13](https://www.youtube.com/watch?v=418t26CVz-w&t=253s) But at least in, in the, in many cases where you need advanced AI compute, you can develop games that can work locally on your, uh, machine, on your mobile device rather.

[4:23](https://www.youtube.com/watch?v=418t26CVz-w&t=263s) And, uh, that's great because if you're in a subway tunnel or something like that, you can still have a, a game with a very smart AI that is, um, building, uh, doing something, uh, without being disrupted due to, um, uh, HTTP calls that are not getting through.

[4:38](https://www.youtube.com/watch?v=418t26CVz-w&t=278s) Um, and then it's, uh, again, personalization is a big thing in general with on-device models because personalization is important. Uh, and a lot of people want their game experience to be tailored to them, and in a world where we can do things, um, on device, we have that capability more so because the, the features are local.

[4:57](https://www.youtube.com/watch?v=418t26CVz-w&t=297s) So now we're gonna talk about agentic AI for games, and, um, as I'm sure everybody's familiar with AlphaGo, AlphaZero, et cetera here, um, the Alpha series models. So basically, um, uh, reinforcement learning was the way that, uh, games were in the past, um, trained in order to, uh, do intelligence, right?

[5:20](https://www.youtube.com/watch?v=418t26CVz-w&t=320s) So basically, what you would do is that you would take a, you take a model, uh, have many iterations, and then that, uh, that model would then, um, be really tuned to a particular game.

[5:31](https://www.youtube.com/watch?v=418t26CVz-w&t=331s) Um, and that's still happening, but, uh, what I'm doing is setting up the kind of history and the school of thought of that, uh, all the way up to twenty twenty-four.

[5:38](https://www.youtube.com/watch?v=418t26CVz-w&t=338s) Um, has anybody heard of EfficientZero before here? Uh, if not, it's fine. EfficientZero and EfficientZero V2 are the reinforcement models, uh, reinforcement learning models that are state-of-the-art, um, and they're called sample-efficient reinforcement learning models because they learn, uh, very fast with a lot of-- a lot less data than traditional reinforcement learning models do on games.

[6:01](https://www.youtube.com/watch?v=418t26CVz-w&t=361s) Um, and there's just-- So that was up to twenty twenty-four, and the, that, uh, mechanic is continuing, but it lives alongside, uh, our, uh, reinforcement learning and agentic game, uh, development.

[6:14](https://www.youtube.com/watch?v=418t26CVz-w&t=374s) So, uh, reinforcement learning again is, on the left is a bunch of cycles that are used to train a model in order to change its weight, so it becomes more intelligent and able to play this particular game.

[6:25](https://www.youtube.com/watch?v=418t26CVz-w&t=385s) An agentic system is one where you actually have a game that is, uh, negotiated by agents which are essentially, uh, language models, which are using in-context learning to figure out the state of the world in the game.

[6:38](https://www.youtube.com/watch?v=418t26CVz-w&t=398s) So RL changes the model itself and changes the weights in the model. Um, the agentic system reasons over, uh, space in the game, um, and there's no reward system to grind out.

[6:49](https://www.youtube.com/watch?v=418t26CVz-w&t=409s) Uh, the loop, there's an agentic loop that's at play, which then learns the system through tool calls and other things that are local to the device and using local device functionality and adapts to a new situation, so it's much more dynamic.

[7:03](https://www.youtube.com/watch?v=418t26CVz-w&t=423s) So, um, so this is, uh, basically one example of an agentic-based game using-- Ev-everybody's seen Space Invaders before, I, I presume. Okay. Well, if not, y-well, there it is.

[7:14](https://www.youtube.com/watch?v=418t26CVz-w&t=434s) So this is a agentic version of Space Invaders, which, um, is designed to use a local agentic framework, which we worked on to, um, do basically the basic things like kill the spaceships as they get close and avoid the bullets as the spaceships attack.

[7:30](https://www.youtube.com/watch?v=418t26CVz-w&t=450s) And you've probably seen simulations where our RL models have been trained to do the s-same thing, but this is specifically an agentic system that we built to actually play this game.

[7:39](https://www.youtube.com/watch?v=418t26CVz-w&t=459s) Um, and then you can see the agentic states on the right. And so this is the inside the agent itself and how the loop works. So basically, you have very few basic states where you're trying to perceive what's happening in the-- where the agent is trying to perceive what's happening in the scene, where it's trying to predict

[7:55](https://www.youtube.com/watch?v=418t26CVz-w&t=475s) the next move of the spaceships attacking it, where it's deciding what to do next. Because if the bullet is too close or if the spaceship is too close, it will take a-- it'll make some sort of action, and then it will act in that, uh, way, in the appropriate way, and then loop again to see the state.

[8:09](https://www.youtube.com/watch?v=418t26CVz-w&t=489s) So that's a very simple system, but it's, uh, it's basically a simple agent that's, um, uh, using an on-device model to actually execute its steps. So some ch-challenges in this space are space is-- itself is a challenge because on the device you have a limited amount of resources.

[8:26](https://www.youtube.com/watch?v=418t26CVz-w&t=506s) You have a device budget where you have to fit the weights, you have to compress the state history of the agent, you have to compress the working set or the planning artifacts of the agent if you even think of the way that an agent will create the way that it navigates a system.

[8:39](https://www.youtube.com/watch?v=418t26CVz-w&t=519s) And you have to have enough s-render headroom in order to actually render the game and play the game itself. So this is basically that. And then you also have a time constraint because time itself is budgeted in a particular way so that you're essentially using like a refresh rate of sixty hertz.

[8:56](https://www.youtube.com/watch?v=418t26CVz-w&t=536s) So in each frame that passes, you have to-- which you get a new frame every sixteen milliseconds about if your, uh, refresh rate is sixty hertz. You need to actually fit the agent planning within that, and if you go outside of that budget, what'll happen in the agent execution, you're going to see jank and other things on

[9:11](https://www.youtube.com/watch?v=418t26CVz-w&t=551s) mobile devices that, that are very hard to con-- uh, pull out. And then you have energy constraints, and energy constraints are basically battery life. So battery life, um, you know, the basic devi-devices these days are not optimized to, um, really handle agentic workloads.

[9:27](https://www.youtube.com/watch?v=418t26CVz-w&t=567s) Uh, yet they're getting to that point where you have NPUs, neural processing units, and AI chips that are specifically designed to handle that inference. But at this point in time, um, y- uh, the agentic, uh, design of the loop on the device has to be very curated to do as minimum energy pro-processing as possible.

[9:45](https://www.youtube.com/watch?v=418t26CVz-w&t=585s) Otherwise, your phones die fast as it is, and they'll die even faster. Um, so essentially, there's a constraint function in-- of this, for example, constraint graph that takes into account all three of these areas.

[9:57](https://www.youtube.com/watch?v=418t26CVz-w&t=597s) And so there's different ways of managing the constraints in this constraint graph, so you can actually make sure that your agent is performing well. So you can handle space constraints, time constraints, energy constraints, and th-there are some techniques that we use to optimize those constraints so that we're not, uh, we're, we're penalizing over-focus on one area, and

[10:16](https://www.youtube.com/watch?v=418t26CVz-w&t=616s) then we're using like soft constraining on another area so that we're not penalizing too much if we need a little extra space. But if we use too much time, we want to penalize that because we don't want the agent to disrupt the user experience.

[10:29](https://www.youtube.com/watch?v=418t26CVz-w&t=629s) Um, so, um, we, we look at things like gaze estimation models as well because, um, gaze is another new factor. It's been around for a while, but it's evolving to the point where it can actually, um, uh, be, uh, useful.

[10:42](https://www.youtube.com/watch?v=418t26CVz-w&t=642s) And an agent has to take into account gaze model data, which is basically a convolutional neural net which takes gaze data and then can, um, translate it to agentic behavior.

[10:53](https://www.youtube.com/watch?v=418t26CVz-w&t=653s) And so, um, I'll-- it just begs to mention that we need to consider things like the ARC-AGI score, uh, because we also have the base intelligence of the model, and the base intelligence of a lot of these models aren't, uh, capable yet of being fully, um, cognizant of all the complex decisions that an agent can make in

[11:10](https://www.youtube.com/watch?v=418t26CVz-w&t=670s) the most intelligent and possibly interesting games in the future. But we, we have to, um, essentially sur-surmount this to get to that point. And the last thing I'll show you before I hand it over to Joanne is, um, this is, uh, another agent that we built to solve the mini crosswords.

[11:25](https://www.youtube.com/watch?v=418t26CVz-w&t=685s) So if you play the mini crosswords, this is called a constraint satisfaction agent, which uses a constraint satisfaction graph to figure out how to put the words in the crossword puzzle at the right place so that, um, the-- it fills the puzzle properly.

[11:38](https://www.youtube.com/watch?v=418t26CVz-w&t=698s) And it backtracks in the puzzle and finds the right word, sometimes finds the wrong word. But the constraint graph here is optimized so that if the constraints are getting too, um, uh, too crowded or too incorrect, then it will backtrack efficiently, and the agent can use natural language from the graph to backtrack and find the correct crossword,

[11:57](https://www.youtube.com/watch?v=418t26CVz-w&t=717s) uh, uh, crossword layout. Um, so Basically where are we now? We have a lot of different models that we are considering. So we have generalist models, um, like for example, Google SIMA, which will be the agentic framework, an agentic framework that can do a lot of different things.

[12:13](https://www.youtube.com/watch?v=418t26CVz-w&t=733s) In this space, um, we have agents that do visual estimation on the phone. Uh, and then we have ins- entire generative worlds that are being built using AI, um, and that will be a whole different interesting agentic space to consider.

[12:26](https://www.youtube.com/watch?v=418t26CVz-w&t=746s) Now I'll hand it over to Joanne to talk about accessibility.

[12:30](https://www.youtube.com/watch?v=418t26CVz-w&t=750s) Oh, is this thing on? Yes. Um, let's talk about accessibility. How can we build a system that truly serves everyone? Um, like Shafik mentioned, traditionally, games live in a world of fixed state models, rigid, hand-authored.

[12:44](https://www.youtube.com/watch?v=418t26CVz-w&t=764s) A player's choice is often limited to a toggle or a static menu. The flaw here, as you could have guessed, is that it's entirely de- deterministic, predictable, and completely blind to their real-time needs.

[12:56](https://www.youtube.com/watch?v=418t26CVz-w&t=776s) But we don't need to reinvent the wheel here. To battle, to build a battle-tested accessible game, we ground our design in an exis- existing international standard, WCAG 2.2. This framework evaluates interactions across four human-centric pillars.

[13:11](https://www.youtube.com/watch?v=418t26CVz-w&t=791s) Is this experience perceivable, operable, understandable, and robust? Giving us a proven global foundation to build our dynamic player states.

[13:21](https://www.youtube.com/watch?v=418t26CVz-w&t=801s) But you can't have rules without breaking them. Some common pitfalls where games break these standards. A mobile game like Crosswords, as an example. There are motor challenges we come across, where tiny grid cells can lead to mis-taps, active cell highlights are invisible to low vision players.

[13:37](https://www.youtube.com/watch?v=418t26CVz-w&t=817s) Uh, cognitive barriers can be forcing manual logins or timers that are unforgiving and alienating to process speeding, uh, to processing speed needs. A static easy mode can't fix a crossword grid that remains fundamentally blind to our players.

[13:53](https://www.youtube.com/watch?v=418t26CVz-w&t=833s) Although it's still in draft, WCAG 3.0, spoiler alert, is following the trends of dropping the binary pass-fail to a graded bronze, silver, and gold scoring. Accessibility is developing into a more diverse scale and moving away from our standard checkbox.

[14:07](https://www.youtube.com/watch?v=418t26CVz-w&t=847s) The same shift should apply for gameplay. This scale visualizes the values that can change depending on the needs of the player. Here we have the core dials that scale depending on the player's real-time needs.

[14:24](https://www.youtube.com/watch?v=418t26CVz-w&t=864s) If we tune for a player needing more assistance, we have input tolerance that controls the flexibility around physical constraints, making interactions more forgiving. This step granularity adjusts for the number of operations in a sequence, helping users break complex tasks and tuning to players who needs more nuance.

[14:41](https://www.youtube.com/watch?v=418t26CVz-w&t=881s) The agent's time-- The agent's entire job is to dynamically calculate these dials as they rise and fall together, balancing the scale against the player's live context.

[14:58](https://www.youtube.com/watch?v=418t26CVz-w&t=898s) With on-device models, the agent can get a real-time feel for how the player is doing. For example, we can use vision models to track eye gaze and figure out when the user might be facing search friction, analyze shaky taps for someone having a hard time with controls, meet players where they're at by allow, by allowing handwriting recognition.

[15:17](https://www.youtube.com/watch?v=418t26CVz-w&t=917s) Um, focus traps. But we don't want the agent to just watch, we want it to act. We simulate on the right, um, uh, well, the simulation on the right is a broken dialog, forcing a user into a keyboard trap.

[15:28](https://www.youtube.com/watch?v=418t26CVz-w&t=928s) For a user using a switch or a keyboard instead of touch, focus order is very important. If we have a broken dialog, tab and focus can be an endless loop with no way to navigate back.

[15:39](https://www.youtube.com/watch?v=418t26CVz-w&t=939s) Here, the agent can monitor that focus path, flag the loop, and inject an exit route live on the device.

[15:48](https://www.youtube.com/watch?v=418t26CVz-w&t=948s) When you're navigating with limited dexterity or just fat fingers, tiny targets can mean a miss. Instead of using players, um, instead of making players play Operation, our agent is a live layout auditor.

[16:01](https://www.youtube.com/watch?v=418t26CVz-w&t=961s) It measures the interface on the fly, catches violations, and dynamically resizes the controls. The agent rewrites the layout live, adapting to the human and not the other way around.

[16:13](https://www.youtube.com/watch?v=418t26CVz-w&t=973s) Think of it like a real-time tug-of-war where the system is no longer fixed, but tunes the game to the player. Accessibility and challenge stop being treated separately and become two ends of one dial, constantly tuning for the moment.

[16:28](https://www.youtube.com/watch?v=418t26CVz-w&t=988s) Ultimately, this is, uh, there, uh, this is where on-device AI is heading. By building systems that can actively sense human frustrations, we can move past fixed menus and turn our devices into responsive, empathetic partners.

[16:42](https://www.youtube.com/watch?v=418t26CVz-w&t=1002s) But there's still groundwork to be laid. Looking ahead for local agents to truly understand games, there is still a lot we need to build. First, they have to be faster.

[16:51](https://www.youtube.com/watch?v=418t26CVz-w&t=1011s) We need a plan, uh, we need plans and decisions within a sixteen millisecond frame to prevent stuttering for games. We need models to predict the game so that we can see exactly what a layout change will do before they actually make it.

[17:03](https://www.youtube.com/watch?v=418t26CVz-w&t=1023s) And the real magic, though, will be the long-term memory, learning one specific person's unique habits and needs over time. We need a shared game state language so one agent can work across multiple games instead of being rebuilt from scratch for different releases, releases.

[17:18](https://www.youtube.com/watch?v=418t26CVz-w&t=1038s) And finally, we need better chips and honest testing. Faster chips paired with real benchmarks to prove that the agent makes things better.

[17:28](https://www.youtube.com/watch?v=418t26CVz-w&t=1048s) And the future of AI doesn't have to be one giant centralized brain. It can be billions of small local brains, each running on a personal device, each shaped entirely by the individual it serves.

[17:41](https://www.youtube.com/watch?v=418t26CVz-w&t=1061s) Thank you. [audience applauding] [upbeat music]
