I love you so jevving much — Diogo Almeida, TypeSafe AI
Loading slides…
Read the talk
I love you so jevving much
A model can write a research report that Diogo Almeida would struggle to match and still give an answer he would hesitate to use. His case for Jev starts at the moment a program has to act on that answer, with consequences for agent memory, security checks and type systems.
From a talk by Diogo Almeida
At a glance
Ideas worth remembering
For an automated decision, ask how the caller can tell when an answer is safe to use. Occasional brilliance helps much more when a person can inspect the result.
Judge a training task by the operation it makes possible afterward. Repeated next-token prediction generates strings. Human preference data made a small model better at following instructions. Jev pursues the corresponding training problem for decisions consumed by code.
Keeping a coding agent’s history solves only the storage problem. A later call needs a way to retrieve what matters, concurrent agents need a way to share discoveries, and the cost of processing their context can determine whether dividing the work pays off.
Why is accounting harder to automate than a research report?
If difficulty alone explained what AI could do, routine bookkeeping ought to come well before prize-level mathematics. Yet someone watching the benchmarks sees extraordinary performance on hard reasoning problems, while someone looking at everyday work sees people still doing the accounting. Diogo Almeida, cofounder of TypeSafe AI and a contributor to ChatGPT, enjoys coding agents and chatbots. What bothers him is how readily success on the first kind of task gets treated as a forecast for the second.
Who gets to inspect the answer before anyone depends on it? Take deep research. Almeida thinks the system can produce a better report than he can. Now suppose you have to pick whose answer to use, in advance, and the only thing that counts is whether it is correct. He thinks you would pick him. A human reader can take what is useful from an impressive report and investigate a doubtful claim. A program handed a decision needs to know what to do next.
A drive-through makes the distinction concrete. The interaction can please the customer and still yield the wrong order. Automation requires a decision the ordering software can use, after which ordinary code can carry on. Almeida traces the mismatch to a technique he helped develop, reinforcement learning from human feedback (RLHF). Human preferences supply a training signal for the kind of response a person wants to receive. That signal works remarkably well for assistance. It rewards operational correctness only to the extent that the person giving feedback can recognize it.
Tasks with cheaply verifiable answers are a partial exception. A system can run on its own and defer inspection until the end. Training with verifiable rewards also gives a model a signal tied to success on the task, although Almeida finds models trained that way difficult to interact with until they receive human-feedback training. Background work poses its own test. The decision has to hold up after the person leaves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What should a machine-native model practice?
Jev is TypeSafe AI’s attempt to train a model whose decisions another program can use. Its interface is only part of that bet. Almeida starts with a loose reading of Sutton’s bitter lesson, that methods able to use more compute tend to win over clever algorithms. He puts two decisions ahead of buying compute. Where will the training data come from? And what, exactly, will the model practice doing with it?
Chess hides both decisions. The rules tell you what counts as a win. Spending compute to play out games produces more experience under those same rules. It is easy in that setting to credit compute for the improvement, because the game supplies a well-defined task and turns computation into data. With language pretraining, researchers had to choose what a model should learn from a finite supply of text.
Masked language modeling asks the model to recover a missing piece of text. Next-token prediction asks it to continue a prefix. The latter has a useful consequence at inference time. Append the prediction to the prefix and the model has another instance of the task it practiced. It can keep generating a string. Getting better at filling in a blank serves a different purpose, even when both objectives use the same text.
How much can changing the task buy? In the early RLHF work Almeida recalls, researchers first needed data about which responses people preferred. With that feedback, even a small model using simple training algorithms made a striking gain on instruction following. Almeida expresses the size of the gain by estimating how far researchers would have had to scale pretraining alone to match it. The extrapolation concerns one capability. The experimental result was that a small model got markedly better at following a request once its training signal rewarded that behavior.
Jev takes its cue from that experiment. If another piece of software consumes the answer, dependable decisions on a narrow task are worth training for directly. Almeida credits the choice of task for Jev’s behavior. Its particular training objective and data recipe remain unspecified.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The call an application can depend on
Imagine getting Einstein 95% of the time and Mr. Bean the other 5%. In Almeida’s analogy, the trouble is that the caller cannot tell who answered this time. A person who occasionally needs a brilliant idea can check it afterward. Software that acts on every answer inherits every mistake. Almeida attributes developers’ enthusiasm for Jev to its dependability on a focused task. Speed, price and a convenient interface alone would leave the caller with the same predicament.
A coding agent can generate software just in time and make it cheaper to write. Once delivered, the application can still behave much as it did a decade ago. A chatbot alongside it goes a step further by interacting with the running application, while leaving costly decisions to the user. What Almeida calls smart software puts judgment inside the application’s own execution loop. The program encounters a case, calls the model for the part that needs judgment, then uses the result to continue. At that handoff, a more dependable model changes which decisions the product can take on.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the blue LED suggests about a fourth logic gate
Red and green LEDs made good indicators and clocks. Adding blue sounds like a 50% increase in the choice of colors. The interesting change came from what engineers could do with that third color. White light and full-color displays became possible, putting LEDs to work in products that had little in common with an indicator lamp.
Software already builds elaborate behavior out of binary operations and comparisons such as greater than, less than and equal. Almeida imagines a conceptual fourth logic gate that lets code ask a question about meaning. Consider an email address. A programmer can spell out a pattern for a Gmail address in a regular expression. Asking whether an address looks real expresses a different condition, one a person can often judge without being able to enumerate the rules. If a model could return that judgment cheaply, quickly and dependably, existing code could branch on it. A new kind of question becomes available wherever the program already makes decisions.
Which combinations would matter is hard to predict from the primitive alone. Almeida calls the work of trying them software research. His precedent is the early web, where people could build a strange site simply because the tools allowed it. Enough people experimenting independently found uses that would have been hard to justify in advance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agent memory beyond one KV cache
A program with a model in its inner loop still needs decomposition. Break the task into stages and give each stage a result that can be checked. Then the next stage can use that result without reconstructing everything the previous stage did. State needs the same care. If every stage relies on whatever happens to be in a single model context, the program has effectively acquired global state.
There is an efficiency incentive to build agents that way. The key-value (KV) cache retains attention keys and values for the context the model has already processed. Extending the same history lets it reuse that work. Rearranging the context or sending work to an agent with a different history can sacrifice some of that reuse. An agent architecture can look sensible as software and prove expensive as a sequence of model calls.
Consider two kinds of lost work in a coding agent. Across sessions, the system may have the entire history on hand yet lack a good way to select which earlier discovery matters to a new call. Across concurrent agents, several workers may each spend time rediscovering the same pattern in a codebase. The first case calls for selective lookup over stored experience. The second calls for shared state that lets one worker benefit from another’s discovery.
Routing work to subagents raises both questions at once. Which agent should get the task, and what must it know to get started? A clean division of responsibilities can still duplicate expensive context processing. Cache-aware routing, shared discoveries and efficient retrieval are open engineering problems in Almeida’s account. Their payoff would be a division of labor that saves work in the running system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where a program needs to know what someone meant
The most speculative direction begins with the principal–agent problem. Someone delegates a job to a representative whose incentives may differ from their own. Citizens and elected officials are Almeida’s example. Writing explicit rules can make enforcement fairer, but a representative can satisfy the letter of a rule while defeating its purpose. Software runs into a related difficulty when the condition a programmer can encode captures only part of what they meant. Could an application check the intended property where a decision actually happens?
- Real-time security. Recognizing that an action is suspicious may take less intelligence than carrying it out. A defender still needs the judgment early enough to intervene, at a cost it can afford on routine actions, with enough reliability to act on the result. A model that spots the problem on retrospective inspection meets only part of that requirement.
- Semantic types. A developer wants to declare locally that a property must hold. Convincing yourself that it holds can require reading down a chain of dependencies. Almeida imagines a Jev-like model in the lint checker that evaluates the semantic property where it is declared. How dependable such a check could be remains open.
- Voice interfaces. The old ambition was to tell software what you wanted and have it infer the intended action. Earlier attempts ran into the sheer number of rules someone had to hard-code. A spoken request becomes a different sort of interface if the software can interpret what the user means after it hears the words.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Who pays for slop, and who gets to play?
Unreliable AI already has a use wherever somebody else bears the cost of a mistake. A company may hesitate to let a model make a decision that could lose it money, yet feel comfortable sending out material nobody at the company has read. The recipient inherits the job of sorting, checking and discarding it. That asymmetry is Almeida’s explanation for slop. Reading and filtering before sending puts the work back on the producer. A reliable filter on incoming material could also give the recipient some relief.
How much play can you get out of an account-deletion prompt? Almeida describes a Bad UI Battles entry that turns that familiar prompt into a game. The joke works because everyone knows how straightforward the interaction ordinarily ought to be. He likes these exercises because the maker cares about the result, even when the intended result is a horrible interface. Give people room to pursue enough odd ideas and one of them may uncover a possibility worth keeping.
Engineers carry a mental list of things they do by hand because they cannot see how to express the decision in code. Almeida’s closing invitation is to revisit that list. If a program could use a judgment about meaning as readily as it uses a comparison, the question for some of those jobs would become how much engineering the automation takes. His parting objection to benchmarks is on behalf of the engineer with time to consult only a score. That engineer needs to know whether the next call is dependable enough to build on. Making a semantic decision as reliable as a logic gate is still the ambition.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…