Pick Problems That Survive When AI Solves Coding - Ara Khan, Cline

An engineer used to be able to put a weekend project on GitHub and let the working software speak for itself. What does the same repository tell a hiring manager when a coding agent can build it in an afternoon? Ara Khan, a founding engineer at Cline, takes that question seriously. A project can be useful to its users and still say very little about the judgment it took to build.

He starts with a piece of his own work that a stronger coding model made more interesting. Running Kimi K3 inside Cline gave his team 69 passes on 89 Terminal-Bench 2.1 tasks. The model developer had reported a higher score. Cline could read the failures and change the software around K3. Could another coding agent do the investigating and the editing, and leave the team with a harness worth keeping?

The jump from a chatbot to a coding colleague required tools that could change things outside a conversation. A system that can list a directory, read files, edit them and run commands can try its own fix. Khan remembers building early multi-agent conversations in which much of the apparent collaboration came from the developer wiring messages between classes. Giving agents more people to talk to still left the developer with the hard question of whether any work had been done.

Take a request to set up an MCP server. An agent might have to find the documentation, edit a configuration file, install a dependency and start a process. A confident final message tells you little about whether the client can now use the server. An independent test can open the client and exercise the tool. The transcript then becomes a way to explain a pass or a failure.

Cline used the same separation at benchmark scale. Each of the 89 terminal tasks got an isolated container and a grader. The pass count gave the team a starting point. Tool calls, token use, elapsed time and command output let it distinguish a broken file reader from a working session that exhausted its time allowance. The next decision depended on which explanation recurred.

Khan put GPT-5.6 Sol in charge of the outer loop. K3 attempted the tasks through Cline. Sol read the traces, grouped similar failures, proposed a harness change and ran the affected cases again. With a full evaluation taking roughly an hour, a useful lead agent had to remember which hypotheses it had tried and when a change had earned another full run.

The loop detector gave it a particularly good clue. A shell command could appear unchanged across several turns because the agent was polling a background job. The job’s output kept changing. Killing the session for repeating a command threw away evidence of progress that the harness already had. Checking the output as well let healthy tasks continue. A different group of failures came from a provider rate limit ending otherwise viable sessions. Those failures called for waiting and retrying.

The results slide puts both runs on the same denominator. The stock harness passed 69 of 89 tasks (77.5%). The confirmation run passed 79 of 89 (88.8%). In Khan’s published account, that final run cost $49.80. The full 17-hour experiment, including the repeated evaluations and the agent doing the editing, cost roughly $680. Those numbers answer two different questions an engineering lead would ask about doing this again.

The team kept the grader separate, ruled out task-specific shortcuts and reviewed the resulting pull request. Khan also cautions against adapting the harness to every mistake as it happens. A tool change that helps on several unrelated tasks has a better claim to belong in the product.

Khan widens the lens from the harness to the rest of the stack. Training-data selection, post-training, inference kernels and evaluation all offer different ways to make the next run better. Faster inference matters twice in a fixed-budget experiment. It improves the experience of the person waiting for an answer and may buy the team another round of trials.

His database example starts with an ordinary request. Add a column, populate it and put a cache in front. The agent completes the migration but chooses the wrong cache lifetime. A developer tells it which TTL to use, watches it fix the problem and moves on. That short exchange identifies a boundary of the system’s competence much more precisely than a thumbs-down on the whole task.

An engineer can turn that observation into a reproducible case with a check for what happens when the cached value expires. Then it becomes possible to compare a better tool, different instructions or a different model on the behavior that actually failed. A collection of such cases also gives the engineer somewhere to look for regressions after the next change.

In a personal project, Khan had a small device with a camera, speaker and microphone. It responded to a question about what it could see, but spoke back too slowly. He connected it to his computer and had a coding agent send requests to the device and time the replies. The device supplied something a second chatbot could not: an end-to-end measurement taken on the hardware that would actually be used.

The overnight experiments could change the firmware, the tool calls, the inference model or the speech-to-text component. Once the computer could put a repeatable question to the device, a change in any one component could be judged by what happened at the speaker. Khan reports the device got faster. The transferable part of the experiment is how he made latency observable across the whole path.

Khan’s career advice follows from the difference between the weekend repository and the hardware on his desk. A new model release might build the old demo in one shot. That same release gives the engineer with the device a better candidate to test. The apparatus, the latency measurement and the knowledge of what a good interaction feels like all carry forward.

He suggests asking two questions before committing to a long project. If the next model gets much better, which part of the work becomes more valuable? Which part could disappear entirely? Data pipelines that expose real failures, hardware that can test a change and networks that make experiments cheaper all give concrete answers to the first question. Sometimes the honest answer to the second is that it makes sense to wait one release.

The time horizon is a judgment call, and Khan invites disagreement about it. His practical test works on a shorter clock. When an agent can already finish most of an engineer’s daily tasks, look for the correction that still requires that engineer. Can someone capture the failure, tell when it is fixed and give the next attempt better conditions? That is a piece of work a more capable model can help compound.