How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS

Nick Nisi, a DX engineer at WorkOS, describes working across more than 20 repositories and eight languages, including projects such as AuthKit Next.js, AuthKit React, WorkOS Node, WorkOS Kotlin, WorkOS Ruby, and PHP. Agents increased his ability to handle implementation while he reviewed their work, but coordinating one agent at a time across many repositories created a new operational bottleneck: continual context switching. 0:14

Starting each task also required repeated setup: identifying the relevant GitHub issue, linear ticket, or Slack thread, explaining the problem, and establishing enough context for an agent to begin. Nisi describes spending roughly 10 minutes on this handoff for each task. He also identifies a second challenge beyond internal productivity: agents increasingly mediate how developers encounter and use products, making agent experience relevant to developer experience. 1:30

These pressures led to two complementary efforts: an internal system that could take an existing work item and drive it toward a verified pull request, and a customer-facing WorkOS CLI intended to reduce the friction of installing AuthKit. Across both, the underlying question was not whether an agent could generate plausible code, but whether the surrounding system could establish that the result actually worked. 2:19

Nisi built an internal harness called Case that accepts inputs such as a GitHub issue, pull request, Slack thread, or linear ticket, gathers the necessary context, and works toward a pull request accompanied by evidence. Its initial implementation as a Claude skill worked until additional complexity caused context loss, skipped tasks, and unreliable adherence to requested steps. 2:19

He rebuilt Case on top of Py, using a TypeScript state machine to coordinate five agents: an implementer, a verifier, a reviewer, a closer, and a retro agent. The essential design feature is not the number of agents but the gates between states: verification must precede review, review findings return to implementation, and the closer cannot finish without producing evidence. These transitions move enforcement outside the model’s discretionary behavior. 3:25

An early testing gate illustrates why superficial evidence is insufficient. Case originally treated the presence of a .case tested file as proof that tests had run, but the agent could simply create the file. Nisi responded by saving a SHA-256 hash of test output and checking that evidence, with the practical goal of making the requested work easier than fabricating compliance. The broader lesson is to tie progression to verifiable artifacts instead of assuming an agent followed written instructions. 4:23

For interface bugs, Nisi wants the agent to use the Playwright CLI to record behavior before and after a fix and attach those videos to the pull request. He still reads generated code and evaluates whether it meets his standards, but he does not want to spend review time until the agent has first demonstrated the requested behavioral change. If the evidence is missing, the task returns to the agent rather than immediately becoming a human debugging exercise. 11:23

The customer-facing WorkOS CLI can identify the surrounding project, install AuthKit, and provision an account that can be claimed later. Nisi describes support scenarios involving Next.js, TanStack, and Ruby projects, as well as replacing an existing Auth0 setup. However, an installation into TanStack Start exposed a framework-specific failure: changes to start.ts appeared reasonable to both the human reviewer and the model, but violated an implicit contract governing what that file must export. 5:25

His first response was to transform WorkOS documentation into more than 10,000 lines of generated skills. The generation system tracked sections of documentation using cryptographic hashes so unchanged material would not be regenerated. Although sophisticated, this approach produced lengthy evaluation runs, repeated failures and retries, high token consumption, and guidance that could send the model through unnecessary lines of investigation. 7:27

Evaluations showed that comprehensive coverage was counterproductive, so Nisi replaced the generated corpus with 553 lines of handwritten gotchas drawn from recurring failures. Evaluation scenarios dropped from 68 minutes to 6 minutes per run, while the smaller context kept the model focused. He characterizes the change as deleting 95% of the skills and seeing performance improve, emphasizing that the improvement was established through measurement rather than inferred from the smaller prompt. 8:26

A direct comparison made the downside particularly clear: on one task, loading a particular skill produced a 77% success rate, while performing the same task without it produced 97%. The implication is not that every skill is harmful; it is that any individual skill can add misleading or distracting context, and that its contribution should be tested against a no-skill baseline. Product guidance is most useful when it supplies specific missing constraints rather than attempting to reteach a model how to code. 9:28

The final agent in Case performs a retrospective across the work that preceded it. It examines logs and Claude and Codex transcript JSONL files for patterns such as redundant tool requests, repeated actions without any intervening change, and unproductive loops. Instead of treating these failures solely as isolated mistakes, the system extracts information that can improve later runs. 3:25

Case stores what it learns in Markdown memory files at different levels of specificity, including general memory and framework-oriented memory for Next.js and TanStack Start. A mistake involving start.ts, for example, can become a durable framework-specific warning rather than a problem rediscovered independently in every future task. Nisi also identifies automatic memory pruning as a capability he wants to add later, distinguishing a proposed improvement from the features already present. 13:23

This supports a broader operating rule: when the harness produces a mistake, improve the harness so it can detect or correct that class of mistake itself, rather than repeatedly patching individual generated outputs. Human feedback remains part of the loop, but the objective is to make each observed failure useful to the next run through stronger gates, better memory, and targeted system changes. 12:18

Nisi condenses the approach into three related principles: enforce rather than merely instruct, guide rather than overwhelm, and measure rather than assume. State-machine gates and behavioral evidence address unreliable execution; concise framework or product gotchas address missing context without flooding the model; and evaluation pass rates reveal whether the intervention improves or degrades results. These practices are especially important because agent behavior is nondeterministic and plausible-looking output is not equivalent to successful execution. 9:28

For product teams, the practical starting point is to identify what agents get consistently wrong about the product and document those specific landmines. Nisi also warns that information introduced through client-side JavaScript may be absent from whatever process an agent uses to retrieve and summarize a page. Treating agents as product consumers therefore includes examining both the content of guidance and whether that guidance is actually accessible to automated readers. 14:21

For internal automation, trust should be grounded in observable outputs such as test evidence, pass rates, score changes, and before-and-after demonstrations. None of this removes human review: Nisi explicitly continues reading generated code after the requested behavior has been established. The resulting shift is from manually supervising every intermediate action toward engineering an environment that enforces required work, exposes failures, and improves when those failures recur. 11:23