1 01:46:46,220 --> 01:46:47,840 Hi, I am Ayush. 2 01:46:47,840 --> 01:46:52,260 I built a CLI that lets your agent reuse context from your past session across tools. 3 01:46:52,700 --> 01:46:58,940 While improving it, I realized why my agent secretly hated my CLI and what you can do about yours. 4 01:47:01,760 --> 01:47:06,020 CloudCode wanted one setting from a docs page, so it downloaded the whole page, 5 01:47:06,200 --> 01:47:12,820 grabbed it, and kept the result at 40 lines. Codex wanted to know how many dependencies 6 01:47:12,820 --> 01:47:17,000 were vulnerable. It asked the whole JSON report and kept the last 40 lines 7 01:47:17,000 --> 01:47:19,240 where this report puts the totals. 8 01:47:20,540 --> 01:47:24,820 Cursor ran a test suite, found one failure, and kept the last 80 lines. 9 01:47:25,400 --> 01:47:30,300 All three cuts kept the information shown here, each one encoded in an assumption about 10 01:47:30,300 --> 01:47:32,060 where the answer sits in the output. 11 01:47:32,240 --> 01:47:34,820 That can hide friction from the people who build these tools. 12 01:47:35,400 --> 01:47:40,800 One pipe tells you very little, the same pipe written again and again can point to something 13 01:47:40,800 --> 01:47:41,720 your tool is missing. 14 01:47:42,000 --> 01:47:46,640 These are real calls from my own transcripts and the cloud code one was the agent reviewing 15 01:47:46,640 --> 01:47:53,700 this talk and it still couldn't help itself. Agents can work around friction without filing 16 01:47:53,700 --> 01:48:00,640 a bug report. They write a little program around your program. In June, Sanity, the 17 01:48:00,640 --> 01:48:06,480 content platform, wrote up their MCP server. More than 20,000 agents, about 3 million tool 18 01:48:06,480 --> 01:48:11,880 calls since launch, and not a single bug report. So they shipped a GIF feedback tool. Once 19 01:48:11,880 --> 01:48:15,980 the agent had somewhere to complain, they started expressing what's wrong. But you 20 01:48:15,980 --> 01:48:17,580 You don't need a feedback tool to start. 21 01:48:18,020 --> 01:48:20,880 Feedback is already sitting somewhere in your transcripts. 22 01:48:23,420 --> 01:48:27,500 I extracted about 20,000 agent calls from my same transcripts. 23 01:48:27,580 --> 01:48:33,420 14 CLIs appear in that sample, git, github-cli, cargo, curl, npm-docker, and others. 24 01:48:33,920 --> 01:48:39,120 A quarter fed output into filters such as head, grab, tail, python, sort, or wc. 25 01:48:39,560 --> 01:48:42,400 For read-heavy commands, it's more than half. 26 01:48:42,620 --> 01:48:44,940 Git show and cargo test 62% each. 27 01:48:44,940 --> 01:48:51,980 Some harnesses pipe far less, codex for example, lets the agent set an output limit on each call. 28 01:48:53,620 --> 01:49:01,020 And pipes don't only cut noise. In bash, without pipe fail, a pipeline returns the exit code of its last command. 29 01:49:01,320 --> 01:49:06,700 So let's say you pipe cargo test into head. It can report success even when the test failed. 30 01:49:07,320 --> 01:49:10,980 Here's one example on the screen, a compile error and a green dot. 31 01:49:13,120 --> 01:49:16,900 The status was wrong, but the agent could still read the error. 32 01:49:18,360 --> 01:49:21,880 A pipe can also cut the error itself, which is worse. 33 01:49:22,160 --> 01:49:27,140 In a cloud code issue on GitHub, a user reported it a script through a reference error and 34 01:49:27,140 --> 01:49:28,480 exited with an error code. 35 01:49:28,900 --> 01:49:31,800 Tails showed three harmless lines and Cloud reported success. 36 01:49:32,220 --> 01:49:34,400 The error never reached the model. 37 01:49:34,780 --> 01:49:37,220 So my advice would be to fail loudly. 38 01:49:37,220 --> 01:49:43,020 when a tool fails, make the very last lines a short summary and a part to the full log. 39 01:49:45,340 --> 01:49:51,700 I learned all this while tinkering with the CLI I built. It's a small tool that lets coding agents 40 01:49:51,700 --> 01:49:57,920 search their own past sessions across cloud code, codex cursor. It shifts with a skill so the agent 41 01:49:57,920 --> 01:50:00,000 can decide when and how to call it. 42 01:50:00,760 --> 01:50:08,080 Agents are the users that I'm studying. My October audit recorded about 1,000 calls and that's my small case study. 43 01:50:08,320 --> 01:50:13,360 The method works on anything your agent calls. Just look at what they do right after the call. 44 01:50:15,780 --> 01:50:16,680 80% 45 01:50:17,280 --> 01:50:24,720 4 out of 5 times an agent opened an old session with my tool. The output went straight into Crap, Sad, Head, Tail or Python. 46 01:50:25,000 --> 01:50:26,820 I call that the pipe text. 47 01:50:27,160 --> 01:50:30,300 A pipe keeps junk out of context, and that's fine. 48 01:50:30,660 --> 01:50:35,760 The text is that the agent writing that same cutdown command by hand every single time. 49 01:50:36,000 --> 01:50:41,340 It's small each time, and it mostly works, but each one is another guess about where the answer is. 50 01:50:43,720 --> 01:50:49,340 The fair objection comes from a hacker news thread about designing CLS for agent that I was reading. 51 01:50:49,700 --> 01:50:54,420 The command says to leave it to the agents to choose how much output to read. 52 01:50:54,420 --> 01:51:00,960 agent can pipe through grab or head perfectly well. Well, they can and for one-off they should. 53 01:51:01,220 --> 01:51:07,600 I'm not against that but across those everyday CLIs I found 83% of pipes fell into the pattern 54 01:51:07,600 --> 01:51:13,340 seen at least 10 times the same CLI subcommand and pipe tools. The arguments past could differ 55 01:51:13,340 --> 01:51:20,760 but the agent keep asking for the same things. I understand the output they're cutting is 56 01:51:20,760 --> 01:51:27,160 probably big. I opened one long session with my tool two ways. The whole thing, 1.2 million 57 01:51:27,160 --> 01:51:32,600 characters and that's why the agent spy. Just the part around the search match was about 3,000 58 01:51:32,600 --> 01:51:38,540 characters. Harness is do-cap output but a size gap does not identify the relevant messages. 59 01:51:39,140 --> 01:51:43,800 Selection flags on the other hand let the agent tell your tool which part it wants. 60 01:51:45,580 --> 01:51:50,560 So based on my experience and improvements I made to my own tool, I'll give you three 61 01:51:50,560 --> 01:51:51,160 examples. 62 01:51:51,460 --> 01:51:53,800 Read the left column as the bug reports. 63 01:51:54,200 --> 01:51:58,920 So starting with the first one, filtering with RG and said means let me pick the part 64 01:51:58,920 --> 01:51:59,420 I want. 65 01:51:59,740 --> 01:52:05,980 So my read command view can now written just the part around a match, only the matching 66 01:52:05,980 --> 01:52:08,260 messages or the last few with a size gap. 67 01:52:09,360 --> 01:52:12,060 Second example would be Python over JSON. 68 01:52:12,060 --> 01:52:21,680 So Python over JSON means give me fewer fields. So search got a compact mode one line per hit about six times smaller than JSON with the same results. 69 01:52:23,420 --> 01:52:26,260 Inspect command in chat history summarizes a session. 70 01:52:26,540 --> 01:52:33,220 About a third of inspect calls came from agent looping over several sessions by hand as you can see with the for loop. 71 01:52:33,480 --> 01:52:38,500 That means let me ask several about several at once and now it's one call. 72 01:52:38,500 --> 01:52:45,300 As you can see, my roadmap was in the commands around my command and I had to learn from that. 73 01:52:47,080 --> 01:52:50,640 InfraCost, the CloudCost tool, wrote up the same discovery in May. 74 01:52:50,880 --> 01:52:59,300 Their agents were writing JQ and Python pipelines over JSON dumps, so they moved the filtering into the CLI and added a compact format and a skill to teach it. 75 01:52:59,300 --> 01:53:05,100 Plain Cloud with none of that got 0 of 6 hard questions right in the testing and with it, 76 01:53:05,600 --> 01:53:09,580 6 of 6 and cost across the benchmark also fell 41%. 77 01:53:09,580 --> 01:53:14,900 On Hacker News, the person who wrote that post said that reading those agent logs felt 78 01:53:14,900 --> 01:53:18,000 like watching usability tests and I couldn't agree more. 79 01:53:19,420 --> 01:53:21,680 This part might sound trivial and obvious. 80 01:53:22,000 --> 01:53:26,660 Once it's in the CLI, teach your agent it's there with the skill file or the help flag. 81 01:53:26,660 --> 01:53:32,060 But agents take their teaching literally. I made some changes that turned my deep flag 82 01:53:32,060 --> 01:53:37,240 into a no-op. I left the skill telling the agents to use it to see what they do, 83 01:53:37,540 --> 01:53:44,540 would they find out or not. 95 searches out of 95 still passed it. After that, 84 01:53:44,840 --> 01:53:50,620 two sessions loaded a rewritten skill and that no longer told them to use it. None of the searches 85 01:53:50,620 --> 01:53:56,640 passed deep flag and both tried new flags on their first read. So your skill.md, your help 86 01:53:56,640 --> 01:54:01,440 text, your error messages are the interface. That also means stale dogs are bugs because 87 01:54:01,440 --> 01:54:09,560 your agent will keep following them. Once the new flags reach agents, nearly every search used 88 01:54:09,560 --> 01:54:17,000 compact output and nearly every reduced the new flags. Piped read fell from 82% to about a quarter. 89 01:54:22,160 --> 01:54:25,240 You can get real levels and test cases for your tool easily. 90 01:54:25,560 --> 01:54:28,000 If you use it every day, they are already in your transcripts. 91 01:54:28,200 --> 01:54:32,560 The commands agent really ran are the test cases and what agent did immediately next 92 01:54:32,560 --> 01:54:33,780 is a rough answer key. 93 01:54:34,360 --> 01:54:37,100 Replay those commands through your old build and your new one. 94 01:54:37,340 --> 01:54:44,020 If you change the ranking, check that result the agent went on to use still comes out on 95 01:54:44,020 --> 01:54:44,740 the top or not. 96 01:54:44,740 --> 01:54:48,380 If you change an output format, check that nothing went missing. 97 01:54:48,620 --> 01:54:51,840 If you change the default, see which commands now behave differently. 98 01:54:52,680 --> 01:54:57,000 And when an agent read a call with different arguments, you have probably found a failing 99 01:54:57,000 --> 01:54:57,360 case. 100 01:54:57,880 --> 01:55:00,700 The answer key is only a rough guess at what the agent wanted. 101 01:55:01,000 --> 01:55:03,820 Keep a few cases back and use those to score your fix. 102 01:55:06,620 --> 01:55:12,680 So after analyzing 20,000 invocation, if I were to start all over again, this is what 103 01:55:12,680 --> 01:55:13,580 I would build first. 104 01:55:13,580 --> 01:55:19,420 55% of the measured pipelines used only head, tail, or set downstream. 105 01:55:19,740 --> 01:55:21,880 That points to demand for smaller slices. 106 01:55:22,100 --> 01:55:24,780 So ship ranges and limits before anything fancy. 107 01:55:25,440 --> 01:55:28,880 A flag only helps if agents reach for it. 108 01:55:29,020 --> 01:55:31,340 GitHub CLI's list command have a limit flag. 109 01:55:31,600 --> 01:55:34,100 An agent used it 81% of the time. 110 01:55:34,580 --> 01:55:36,260 The counter example would be cargo. 111 01:55:36,620 --> 01:55:38,400 For cargo test, quiet mode, 112 01:55:38,620 --> 01:55:42,700 prints a character per test instead of a line and still reports failures. 113 01:55:43,340 --> 01:55:47,280 Agent passed it on about one cargo run in 10 and piped instead. 114 01:55:48,200 --> 01:55:51,840 So teach the flag in your help test and your skill. 115 01:55:52,560 --> 01:55:57,060 And make big outputs print less by default and say how to get more. 116 01:55:57,400 --> 01:56:03,520 For example, end with something like showing 20 of 300, add offset flag 20 for the next wage. 117 01:56:05,640 --> 01:56:08,920 One way to ship several of these at once is an agent mode. 118 01:56:09,040 --> 01:56:12,160 Your CLI notices an agent is calling it and prints differently. 119 01:56:12,160 --> 01:56:17,660 Agents like Claudecodes set an environment variable in their cells so your tool can spot them. 120 01:56:18,360 --> 01:56:26,760 DuckDB shipped one. At a terminal you get a nicely drawn table but an agent gets a plain one that says how many rows were left out and how to get them. 121 01:56:27,080 --> 01:56:33,060 A huge query stops early and says so and errors come back as JSON because agents read them better. 122 01:56:34,000 --> 01:56:39,260 It's the same list as we have been just through and it cut the output model read by 59%. 123 01:56:39,260 --> 01:56:45,700 However, the tokens saved were nearly 0.5% of all the tokens the model read. 124 01:56:45,940 --> 01:56:47,600 So cost and runtime didn't improve. 125 01:56:48,000 --> 01:56:52,240 If you remember, InfraCost agents started getting answers right. 126 01:56:52,740 --> 01:56:54,500 DougDBs were already there. 127 01:56:54,760 --> 01:56:57,840 So measure turns and write answers, not just tokens. 128 01:57:00,500 --> 01:57:03,620 might be thinking, I'll ask my agent to do all of this. 129 01:57:04,120 --> 01:57:06,740 They are not that good at this yet. 130 01:57:06,940 --> 01:57:08,480 At least not one-shotting it. 131 01:57:08,480 --> 01:57:10,560 The design is where it goes wrong. 132 01:57:10,940 --> 01:57:12,940 Ask an agent to build a tool for agent. 133 01:57:12,960 --> 01:57:15,920 It probably pictures a person and then bolts the agent on. 134 01:57:16,120 --> 01:57:19,680 It copies most common patterns it has seen, and most of those are poor. 135 01:57:20,020 --> 01:57:25,980 A study of official MCP servers found most just rep an existing API. 136 01:57:26,680 --> 01:57:32,320 Another checked tool descriptions from real MCP servers and flagged issues in 97% of them. 137 01:57:33,160 --> 01:57:34,400 Docs have the same gap. 138 01:57:34,400 --> 01:57:40,360 researchers at each year Zurich tested context files written by agents. They didn't help tasks 139 01:57:40,360 --> 01:57:46,660 succeed, cost more to run, and did worse than files written by people. Agents are already good 140 01:57:46,660 --> 01:57:52,180 at other half of this talk. Give one transcript and a test it can run, and it can find what's 141 01:57:52,180 --> 01:57:57,520 wrong and improve the tool, sometimes past what people wrote by hand. And Tropic reported that 142 01:57:57,520 --> 01:58:03,000 on its own internal tools as well. So you decide what the tool should be and let the agent check 143 01:58:03,000 --> 01:58:08,660 against what really happened. You've seen the evidence for each of these so I 144 01:58:08,660 --> 01:58:12,060 won't read them. Take a moment to absorb these or screenshot. 145 01:58:18,920 --> 01:58:23,240 To find your own list, backup your transcript first because Cloud Code can 146 01:58:23,240 --> 01:58:29,320 delete inactive once after 30 days. Then search for them for a tool's name 147 01:58:29,320 --> 01:58:31,540 followed by a pipe and other similar patterns. 148 01:58:31,800 --> 01:58:36,360 If you ship an MCP server, look for agent calling your tool again, 149 01:58:36,840 --> 01:58:38,180 straight away with different arguments. 150 01:58:40,580 --> 01:58:42,680 If you remember one thing from this talk, 151 01:58:43,000 --> 01:58:45,380 if you are building a tool for your agent, 152 01:58:45,660 --> 01:58:49,420 your agent transcripts are the product analytics you probably aren't reading. 153 01:58:51,700 --> 01:58:55,640 The workaround is the bug report, so go read your transcripts. 154 01:58:55,640 --> 01:58:59,860 And if you need any help doing that, check out Chat History on Cargo or GitHub. 155 01:59:00,100 --> 01:59:01,800 It can do this for you as well. 156 01:59:02,120 --> 01:59:02,520 Thank you.