# Your agent hates your CLI Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [01:46:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6406s) Hi, I am Ayush. [01:46:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6407s) I built a CLI that lets your agent reuse context from your past session across tools. [01:46:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6412s) While improving it, I realized why my agent secretly hated my CLI and what you can do about yours. [01:47:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6421s) CloudCode wanted one setting from a docs page, so it downloaded the whole page, [01:47:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6426s) grabbed it, and kept the result at 40 lines. Codex wanted to know how many dependencies [01:47:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6432s) were vulnerable. It asked the whole JSON report and kept the last 40 lines [01:47:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6437s) where this report puts the totals. [01:47:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6440s) Cursor ran a test suite, found one failure, and kept the last 80 lines. [01:47:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6445s) All three cuts kept the information shown here, each one encoded in an assumption about [01:47:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6450s) where the answer sits in the output. [01:47:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6452s) That can hide friction from the people who build these tools. [01:47:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6455s) One pipe tells you very little, the same pipe written again and again can point to something [01:47:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6460s) your tool is missing. [01:47:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6462s) These are real calls from my own transcripts and the cloud code one was the agent reviewing [01:47:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6466s) this talk and it still couldn't help itself. Agents can work around friction without filing [01:47:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6473s) a bug report. They write a little program around your program. In June, Sanity, the [01:48:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6480s) content platform, wrote up their MCP server. More than 20,000 agents, about 3 million tool [01:48:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6486s) calls since launch, and not a single bug report. So they shipped a GIF feedback tool. Once [01:48:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6491s) the agent had somewhere to complain, they started expressing what's wrong. But you [01:48:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6495s) You don't need a feedback tool to start. [01:48:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6498s) Feedback is already sitting somewhere in your transcripts. [01:48:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6503s) I extracted about 20,000 agent calls from my same transcripts. [01:48:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6507s) 14 CLIs appear in that sample, git, github-cli, cargo, curl, npm-docker, and others. [01:48:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6513s) A quarter fed output into filters such as head, grab, tail, python, sort, or wc. [01:48:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6519s) For read-heavy commands, it's more than half. [01:48:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6522s) Git show and cargo test 62% each. [01:48:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6524s) Some harnesses pipe far less, codex for example, lets the agent set an output limit on each call. [01:48:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6533s) And pipes don't only cut noise. In bash, without pipe fail, a pipeline returns the exit code of its last command. [01:49:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6541s) So let's say you pipe cargo test into head. It can report success even when the test failed. [01:49:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6547s) Here's one example on the screen, a compile error and a green dot. [01:49:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6553s) The status was wrong, but the agent could still read the error. [01:49:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6558s) A pipe can also cut the error itself, which is worse. [01:49:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6562s) In a cloud code issue on GitHub, a user reported it a script through a reference error and [01:49:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6567s) exited with an error code. [01:49:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6568s) Tails showed three harmless lines and Cloud reported success. [01:49:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6572s) The error never reached the model. [01:49:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6574s) So my advice would be to fail loudly. [01:49:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6577s) when a tool fails, make the very last lines a short summary and a part to the full log. [01:49:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6585s) I learned all this while tinkering with the CLI I built. It's a small tool that lets coding agents [01:49:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6591s) search their own past sessions across cloud code, codex cursor. It shifts with a skill so the agent [01:49:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6597s) can decide when and how to call it. [01:50:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6600s) Agents are the users that I'm studying. My October audit recorded about 1,000 calls and that's my small case study. [01:50:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6608s) The method works on anything your agent calls. Just look at what they do right after the call. [01:50:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6615s) 80% [01:50:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6617s) 4 out of 5 times an agent opened an old session with my tool. The output went straight into Crap, Sad, Head, Tail or Python. [01:50:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6625s) I call that the pipe text. [01:50:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6627s) A pipe keeps junk out of context, and that's fine. [01:50:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6630s) The text is that the agent writing that same cutdown command by hand every single time. [01:50:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6636s) It's small each time, and it mostly works, but each one is another guess about where the answer is. [01:50:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6643s) The fair objection comes from a hacker news thread about designing CLS for agent that I was reading. [01:50:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6649s) The command says to leave it to the agents to choose how much output to read. [01:50:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6654s) agent can pipe through grab or head perfectly well. Well, they can and for one-off they should. [01:51:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6661s) I'm not against that but across those everyday CLIs I found 83% of pipes fell into the pattern [01:51:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6667s) seen at least 10 times the same CLI subcommand and pipe tools. The arguments past could differ [01:51:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6673s) but the agent keep asking for the same things. I understand the output they're cutting is [01:51:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6680s) probably big. I opened one long session with my tool two ways. The whole thing, 1.2 million [01:51:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6687s) characters and that's why the agent spy. Just the part around the search match was about 3,000 [01:51:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6692s) characters. Harness is do-cap output but a size gap does not identify the relevant messages. [01:51:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6699s) Selection flags on the other hand let the agent tell your tool which part it wants. [01:51:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6705s) So based on my experience and improvements I made to my own tool, I'll give you three [01:51:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6710s) examples. [01:51:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6711s) Read the left column as the bug reports. [01:51:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6714s) So starting with the first one, filtering with RG and said means let me pick the part [01:51:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6718s) I want. [01:51:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6719s) So my read command view can now written just the part around a match, only the matching [01:52:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6725s) messages or the last few with a size gap. [01:52:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6729s) Second example would be Python over JSON. [01:52:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6732s) So Python over JSON means give me fewer fields. So search got a compact mode one line per hit about six times smaller than JSON with the same results. [01:52:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6743s) Inspect command in chat history summarizes a session. [01:52:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6746s) About a third of inspect calls came from agent looping over several sessions by hand as you can see with the for loop. [01:52:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6753s) That means let me ask several about several at once and now it's one call. [01:52:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6758s) As you can see, my roadmap was in the commands around my command and I had to learn from that. [01:52:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6767s) InfraCost, the CloudCost tool, wrote up the same discovery in May. [01:52:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6770s) Their agents were writing JQ and Python pipelines over JSON dumps, so they moved the filtering into the CLI and added a compact format and a skill to teach it. [01:52:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6779s) Plain Cloud with none of that got 0 of 6 hard questions right in the testing and with it, [01:53:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6785s) 6 of 6 and cost across the benchmark also fell 41%. [01:53:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6789s) On Hacker News, the person who wrote that post said that reading those agent logs felt [01:53:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6794s) like watching usability tests and I couldn't agree more. [01:53:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6799s) This part might sound trivial and obvious. [01:53:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6802s) Once it's in the CLI, teach your agent it's there with the skill file or the help flag. [01:53:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6806s) But agents take their teaching literally. I made some changes that turned my deep flag [01:53:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6812s) into a no-op. I left the skill telling the agents to use it to see what they do, [01:53:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6817s) would they find out or not. 95 searches out of 95 still passed it. After that, [01:53:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6824s) two sessions loaded a rewritten skill and that no longer told them to use it. None of the searches [01:53:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6830s) passed deep flag and both tried new flags on their first read. So your skill.md, your help [01:53:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6836s) text, your error messages are the interface. That also means stale dogs are bugs because [01:54:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6841s) your agent will keep following them. Once the new flags reach agents, nearly every search used [01:54:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6849s) compact output and nearly every reduced the new flags. Piped read fell from 82% to about a quarter. [01:54:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6862s) You can get real levels and test cases for your tool easily. [01:54:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6865s) If you use it every day, they are already in your transcripts. [01:54:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6868s) The commands agent really ran are the test cases and what agent did immediately next [01:54:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6872s) is a rough answer key. [01:54:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6874s) Replay those commands through your old build and your new one. [01:54:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6877s) If you change the ranking, check that result the agent went on to use still comes out on [01:54:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6884s) the top or not. [01:54:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6884s) If you change an output format, check that nothing went missing. [01:54:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6888s) If you change the default, see which commands now behave differently. [01:54:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6892s) And when an agent read a call with different arguments, you have probably found a failing [01:54:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6897s) case. [01:54:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6897s) The answer key is only a rough guess at what the agent wanted. [01:55:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6901s) Keep a few cases back and use those to score your fix. [01:55:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6906s) So after analyzing 20,000 invocation, if I were to start all over again, this is what [01:55:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6912s) I would build first. [01:55:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6913s) 55% of the measured pipelines used only head, tail, or set downstream. [01:55:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6919s) That points to demand for smaller slices. [01:55:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6922s) So ship ranges and limits before anything fancy. [01:55:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6925s) A flag only helps if agents reach for it. [01:55:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6929s) GitHub CLI's list command have a limit flag. [01:55:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6931s) An agent used it 81% of the time. [01:55:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6934s) The counter example would be cargo. [01:55:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6936s) For cargo test, quiet mode, [01:55:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6938s) prints a character per test instead of a line and still reports failures. [01:55:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6943s) Agent passed it on about one cargo run in 10 and piped instead. [01:55:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6948s) So teach the flag in your help test and your skill. [01:55:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6952s) And make big outputs print less by default and say how to get more. [01:55:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6957s) For example, end with something like showing 20 of 300, add offset flag 20 for the next wage. [01:56:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6965s) One way to ship several of these at once is an agent mode. [01:56:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6969s) Your CLI notices an agent is calling it and prints differently. [01:56:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6972s) Agents like Claudecodes set an environment variable in their cells so your tool can spot them. [01:56:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6978s) DuckDB shipped one. At a terminal you get a nicely drawn table but an agent gets a plain one that says how many rows were left out and how to get them. [01:56:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6987s) A huge query stops early and says so and errors come back as JSON because agents read them better. [01:56:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6994s) It's the same list as we have been just through and it cut the output model read by 59%. [01:56:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=6999s) However, the tokens saved were nearly 0.5% of all the tokens the model read. [01:56:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7005s) So cost and runtime didn't improve. [01:56:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7008s) If you remember, InfraCost agents started getting answers right. [01:56:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7012s) DougDBs were already there. [01:56:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7014s) So measure turns and write answers, not just tokens. [01:57:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7020s) might be thinking, I'll ask my agent to do all of this. [01:57:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7024s) They are not that good at this yet. [01:57:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7026s) At least not one-shotting it. [01:57:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7028s) The design is where it goes wrong. [01:57:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7030s) Ask an agent to build a tool for agent. [01:57:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7032s) It probably pictures a person and then bolts the agent on. [01:57:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7036s) It copies most common patterns it has seen, and most of those are poor. [01:57:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7040s) A study of official MCP servers found most just rep an existing API. [01:57:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7046s) Another checked tool descriptions from real MCP servers and flagged issues in 97% of them. [01:57:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7053s) Docs have the same gap. [01:57:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7054s) researchers at each year Zurich tested context files written by agents. They didn't help tasks [01:57:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7060s) succeed, cost more to run, and did worse than files written by people. Agents are already good [01:57:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7066s) at other half of this talk. Give one transcript and a test it can run, and it can find what's [01:57:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7072s) wrong and improve the tool, sometimes past what people wrote by hand. And Tropic reported that [01:57:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7077s) on its own internal tools as well. So you decide what the tool should be and let the agent check [01:58:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7083s) against what really happened. You've seen the evidence for each of these so I [01:58:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7088s) won't read them. Take a moment to absorb these or screenshot. [01:58:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7098s) To find your own list, backup your transcript first because Cloud Code can [01:58:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7103s) delete inactive once after 30 days. Then search for them for a tool's name [01:58:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7109s) followed by a pipe and other similar patterns. [01:58:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7111s) If you ship an MCP server, look for agent calling your tool again, [01:58:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7116s) straight away with different arguments. [01:58:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7120s) If you remember one thing from this talk, [01:58:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7123s) if you are building a tool for your agent, [01:58:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7125s) your agent transcripts are the product analytics you probably aren't reading. [01:58:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7131s) The workaround is the bug report, so go read your transcripts. [01:58:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7135s) And if you need any help doing that, check out Chat History on Cargo or GitHub. [01:59:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7140s) It can do this for you as well. [01:59:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=7142s) Thank you.