How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI

A skills file can accumulate two hundred instructions surprisingly quickly. Conditional behavior, required sections, forbidden phrases, tone rules and edge cases each add another obligation. If an agent quietly loses track of them, the file’s apparent sophistication exceeds what the model actually does. Laurie Voss, head of developer relations at Arize AI and co-founder of npm, begins with that practical concern: how many instructions can fit before compliance starts to fall?

The investigation starts with an aside in a talk by Dexter Horthy at AI Engineer Miami: an agent could follow about two hundred instructions before forgetting some, based on a 2025 result. That would make a substantial skills file a risky engineering choice. A polished response offers little reassurance. It might follow the full specification, or merely resemble the answer the developer expected.

IFScale’s task is simple: write a business report containing a list of exact words. The opening examples are customer and revenue. Each required word becomes one separately checkable instruction. After generation, the benchmark counts how many appear.

Two quantities describe the experiment:

• Density, N: The number of required words supplied together in one request.
• Accuracy: The percentage of those requirements satisfied by the generated report.

The report gives the model somewhere to use the words. Its business insight is not what the score measures.

Follow the revenue requirement through generation. It stays on the input list whether or not the answer sounds convincing. If the report omits that exact word, the requirement fails. Adding more words increases the obligations the model must carry into its answer; counting their appearances makes the loss visible. This answers a question that casual reading struggles with: did each named constraint survive?

The connection to skills is a proxy. Requiring revenue resembles requiring a pricing section because both specify an identifiable obligation. Real instructions can demand reasoning, conditional behavior or resolving conflicting requirements. Voss therefore treats keyword capacity as an optimistic ceiling for more complicated work. Passing this task demonstrates constraint tracking; it does not prove that an equally large real skills file will work.

Before testing newer models, Voss repeats the original experiment. Only three of the paper’s ten models remained accessible through APIs when he ran the replication: GPT-4.1, Claude Sonnet 4 and Gemini 2.5 Pro. Availability determined the comparison set. By the talk, another had been retired—hence his warning, “don’t get attached to your models.”

The replicated curves matched the original paper within what Voss describes as its noise boundary. Accuracy began deteriorating around two hundred to three hundred rules. At five hundred, the models were missing roughly thirty to fifty percent of the required words. The old ceiling had a measurable basis.

The same prompts and words then went to GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro and DeepSeek V4 Pro. All four scored one hundred percent on the original test. Opus 4.8 appeared a week after testing, so these are results for the named tested versions. The benchmark’s maximum of five hundred required words no longer reached their failure region.

The report example now changes in one crucial way: the list grows. Five hundred requirements become one thousand, then two thousand, with testing extended to a ten-thousand-word vocabulary. Each expansion adds words that must survive into the answer. The experiment can finally expose where accuracy bends instead of stopping at a perfect score.

How far did the ceiling move? The results discussion at 7:12 compares deterioration around two hundred to three hundred requirements with newer curves extending nearer two thousand, and the strongest reaching five thousand. Voss describes this as close to a tenfold gain over about twelve months. The horizontal axis is logarithmic: equal distances represent multiplicative changes in rule count. A steep-looking decline can span many additional constraints.

This is a gain in a specific capability: keeping many named requirements in play during generation. It can be much larger than the improvement a developer feels in ordinary conversation. Architecture built around a small instruction budget may now preserve fragmentation that its chosen model no longer needs.

Extending the list changes more than the score. Older models mostly returned reports with missing words. The newer tested versions fail at different points between receiving the request and finishing the answer. Their behavior matters because a single keyword-accuracy curve can hide very different causes.

• DeepSeek V4 Pro: missing requirements. Forgetting starts around seven hundred and fifty rules. By two thousand, nearly half are dropped. A report still arrives, and keyword counting measures the damage directly. Voss prefers this failure’s predictability.

• Claude Opus 4.7: API-level refusal. Combinations in the random vocabulary trigger refusals. Voss attributes them to safety classification, citing anthrax and cyanide appearing together. A large random list can resemble a dangerous request despite its benchmark purpose.

• Gemini 3.1 Pro: reasoning consumes the answer budget. Performance remains strong to about five thousand instructions. Beyond that, Voss describes the model spending its token budget checking requirements, leaving little or no useful report. The request incurs work without delivering the answer.

• GPT-5.5: a plausible report followed by abandonment. The strongest tested result reaches ninety-nine percent keyword accuracy at five thousand rules. When pushed far enough, the model begins writing, objects to the task and stops. Failure becomes apparent at the ending.

Claude required an input change. Voss passed the vocabulary through OpenAI’s safety filter and removed words that looked problematic; the filtered task then performed well. Claude’s successful completion therefore depended on vocabulary filtering. A refusal caused by the list’s content is a different limit from losing track of its length.

GPT’s partial report makes the opening anxiety observable. The model accepts the assignment and writes substantial prose, then rejects the premise of producing a coherent business report containing thousands of unrelated words. The unfinished answer lacks many required keywords. Its objection may be reasonable; the deliverable is still incomplete.

Where can the report disappear or become incomplete? The diagram separates refusal before writing, budget exhaustion during reasoning and defects in returned text. It makes the detection problem visible: an API refusal announces itself, while omissions and late abandonment require inspecting the answer. Receiving fluent prose does not establish completion.

A two-hundred-instruction budget encourages compression and fragmentation: keep the main file short, send the agent to sub-skills and coordinate additional files or specialized agents. Voss calls the result a “Byzantine labyrinth.” Greater capacity makes it plausible to put a hundred or three hundred straightforward requirements together without that machinery.

A style guide illustrates the potential change. Brand rules and legal disclaimers can amount to thousands of named constraints. Keeping more of them in one prompt reduces the need to distribute instructions solely to accommodate the old ceiling. The practical question then becomes whether each addition earns its cost: very large prompts take more time and money to process. Shortening or splitting instructions can still serve those purposes.

There is no universal breaking point. Voss gives a range from roughly seven hundred and fifty to more than nine thousand requirements across the tested models. Selection matters even for this narrow task. A separate question remains: can a model reason clearly over a giant prompt, especially when its instructions conflict?

Voss cites subsequent long-input research across eighteen models reporting accuracy declines of thirty to fifty percent before the context-window limit. He describes a counterintuitive result in which coherent, structured text suffered more than shuffled material, while leaving its cause unexplained. Fitting text into the window, retaining named constraints and reasoning over that material are distinct capabilities.

The unfinished report creates a practical warning. An API refusal announces failure. A confident beginning followed by abandonment makes failure harder to notice. Monitoring must inspect what the application actually returned, rather than treat acceptance of a large prompt or polished opening paragraphs as success.

Voss reports approximately two thousand three hundred calls across seven models costing $29. A simple experiment made a consequential engineering assumption affordable to retest. For tricky production tasks, he recommends monitoring outputs with another LLM. That is his proposed semantic evaluation method; IFScale’s exact-word requirements show why mechanical obligations can instead be checked by counting.

The later reliability discussion introduces Revisiting the Reliability of Language Models in Instruction-following, a paper testing forty-six models. Voss describes sensitivity to rewording and instruction order: the same intended requirements can produce substantially different compliance when their presentation changes. High capacity on one formulation does not guarantee stable behavior across equivalent prompts. A universally correct ordering remains an open research question.

The research direction also moves toward real, messy constraints. Voss names FireBench, CCR Bench and GuideBench as efforts to measure how well models follow many such requirements together. They address the gap between a report containing random words and an application balancing meaningful obligations.

The ending separates two engineering jobs. Compression fits requirements into usable model capacity. Verification checks whether the answer satisfies them. Voss’s declaration that the compression problem is gone expresses the scale of improvement in the tested task. The new hard part is knowing whether the model did what was requested.

Revisit prompt-size assumptions made six months earlier, then test returned outputs against the requirements that matter. A better prompt requests complete work. An eval determines whether complete work arrived. For readers who want to investigate the experiment itself, Voss closes with an invitation to obtain its code and data at the repository shown in the recording around 21:44.