Your Agent Hates Your CLI - Ayush Bhardwaj, Allos AI

The agent reviewing Ayush Bhardwaj’s talk needed one setting from a documentation page. It downloaded the page, filtered it and capped the result at 40 lines. In another session, Codex wanted a count of vulnerable dependencies. It requested the whole JSON report and kept the last 40 lines, where that report put its totals. Cursor ran a test suite with one failure and kept the last 80 lines. All three cuts worked. Each depended on a guess about where the answer would be.

From the tool author’s side, a successful call can conceal the work it took to use the answer. Sanity’s MCP server had served more than 20,000 agents and handled about 3 million tool calls without a single bug report. Complaints started arriving once Sanity added a tool for giving feedback. Bhardwaj had another way to hear from the users of his CLI. An agent that repeatedly writes the same little program around a command has already left a record of what it wanted.

Bhardwaj extracted roughly 20,000 agent calls from his transcripts, covering 14 CLIs including Git, GitHub CLI, Cargo, curl, npm and Docker. A quarter of the calls fed output into filters such as head, grep, tail, Python, sort or wc. For git show and cargo test, the share was 62% apiece. The environment running the agent matters to that count. Codex, for example, lets an agent limit output on an individual call, so it has another way to keep a large response out of context.

What succeeded when cargo test | head came back green? In Bash with pipefail disabled, the pipeline takes the exit status of its last command. head can successfully return the requested lines even when cargo test fails. In Bhardwaj’s example, a green indicator appeared alongside a compile error. The indicator reported the success of head. The diagnostic still told the agent what had happened to the build.

In a second case, the diagnostic itself disappeared. A user reported that a script threw a ReferenceError and exited with an error code. tail returned three harmless lines. The error never reached the model, and the agent reported success. Bhardwaj’s advice for CLI authors is to put a short failure summary and the path to the full log in the very last lines. An agent reading only the tail can then recognize the failure and retrieve the details.

Bhardwaj’s chat-history CLI lets coding agents search earlier sessions across tools. It ships with a skill, an instruction file that explains when and how to call the CLI. An October audit covered about 1,000 calls to this tool. Four out of five times an agent opened an old session, it sent the output straight through a filter or a Python program. That kept irrelevant messages out of context. It also meant writing the filter by hand and guessing again where the relevant message would be.

There’s a good case for leaving a one-off request to grep or head. The shell already lets the caller choose how much to read. To find candidates for a feature, Bhardwaj grouped pipes across the broader CLI sample by CLI subcommand and downstream tools. Arguments could vary. On that definition, 83% of pipes belonged to patterns seen at least ten times. The search term or line count could change on every call while the agent kept asking for the same operation.

Opening one long session in full produced over a million characters. Reading the part around a search match took about 3,000. A harness can cap either response at a fixed size. Asking for the messages around a match tells the CLI which part of the session deserves that space. Two recurring filters suggested changes to what chat-history returned.

• Select messages with view. Agents searched and sliced a session after opening it. The view command gained options to return the messages around a match, only the matching messages, or the last few messages under a size cap. An agent looking for an earlier decision could request the surrounding discussion directly.

• Select fields with compact search. Agents ran Python over JSON search results to extract the few fields they wanted. Compact search returned one line per hit. It preserved the same set of hits in about one-sixth the space of the JSON output.

What did a for loop around inspect reveal? Each invocation summarized one session, and about a third of inspect calls came from agents looping over several sessions. The summary was already the operation they wanted. The loop supplied the missing ability to ask about several sessions at once. Bhardwaj changed inspect to accept them in one call.

A cloud-cost CLI team found agents writing jq and Python pipelines over its JSON dumps. The team moved filtering into the CLI, added a compact format and wrote a skill to teach both. In its benchmark, the agent went from answering zero of six hard questions correctly to answering all six. Cost across the benchmark fell 41%. The team tested those changes together, so the result applies to the combination. One of the authors likened reading the logs to watching usability tests.

A change to chat-history made its deep-search flag a no-op. Bhardwaj deliberately left the instruction to use it in the skill. All 95 searches passed the flag. Then two sessions loaded a rewritten skill that dropped the instruction. Across those two sessions, no search passed the obsolete flag, and both agents tried the new options on their first read.

For an agent choosing its next command, SKILL.md, help text and error messages are all parts of the interface. Stale instructions can keep generating obsolete calls long after the implementation changes. Once agents learned the new flags, nearly every search used compact output and nearly every read used the new options. The share of session reads that went through a pipe fell from 82% to about a quarter.

A recorded command supplies an input for a replay test. The next action supplies a rough answer key. Suppose an agent searched old sessions and then opened one of the hits. Replay that search against the old and new builds. If the change affects ranking, track where the session the agent actually used appears in each result. The search can complete successfully in both builds while the useful hit slips down the list.

For a format change, check whether the new output still contains the information used downstream. For a new default, replay the same arguments and identify which calls change behavior. An immediate retry with different arguments is a candidate failure to investigate. It gives a clue to the agent’s intent, though the agent may have other reasons for changing the query. Keep a few cases out of the design work and use them to score the fix.

Starting over, Bhardwaj would build ranges and limits first. In his broader audit, 55% of the measured pipelines used only simple downstream tools for slicing output, such as head and tail. Adoption varied even where a CLI already offered a shortcut. Agents used the limit flag on 81% of the GitHub CLI list commands he observed. Cargo’s quiet test mode prints a character per test and still reports failures, yet agents requested it on only about one run in ten and often piped the output themselves.

Teach the flag in the skill and help text, and let the output itself teach the next call. A default page can tell the caller it is seeing 20 of 300 results and can request the next page with an offset of 20. The count tells the agent its view is partial. The offset gives it the argument it needs to continue.

An agent mode can bundle several of these defaults. Some coding agents set a shell environment variable that lets a CLI recognize its caller. In Bhardwaj’s database CLI example, a person at a terminal gets a formatted table. An agent gets a plain table that says how many rows were left out and how to retrieve them. A very large query stops early and reports that it stopped. Errors come back as JSON. The omitted-row count and early-stop message tell the agent how complete a result it has.

The database CLI cut the output the model read from the tool by 59%. Across everything the model read, the saving amounted to about 0.5% of input tokens. Cost and runtime showed no improvement, and the agent was already getting the answers right. The cloud-cost benchmark had room to improve on correctness, moving from zero to six correct answers on its hard questions. To tell what changed for the user, count turns and correct answers alongside tokens, cost and elapsed time.

Why not ask the agent to design its own CLI in one shot? In Bhardwaj’s experience, an open-ended request tends to produce a familiar human-facing tool with agent support added afterward. He cited a study in which most official MCP servers wrapped an existing API, and a separate check that flagged issues in 97% of the tool descriptions it examined. Generating more instructions has costs of its own. In a study of agent-written context files, the files increased the cost of running tasks without improving task success and performed worse than files people wrote.

Give the agent a transcript and a test it can run, and the assignment becomes concrete. A tool author can decide that a session reader ought to return the discussion around a match. The agent can replay the view call, check whether the returned window includes the message used next, and try a correction if the useful passage falls outside it.

Start by backing up the transcripts. Bhardwaj noted that inactive sessions in one coding tool could be deleted after 30 days. For a CLI, search for its name in commands that feed a pipe and read what the downstream program keeps. For an MCP server, look for two calls to the same tool in quick succession and compare their arguments. The changed argument may be the clearest account you have of what the first response left unresolved. To find those calls across past coding sessions, Bhardwaj uses chat-history.