The New Primitives: Building AI-Native Software

What should we build when a new computing capability arrives, but its useful forms are still unclear? Voice agents make that question immediate. Kwindla Hultman Kramer works at Daily on infrastructure for real-time audio, video, and AI, and on Pipecat, its open-source, vendor-neutral agent framework. Kramer describes Pipecat as the most widely used voice-agent framework, naming AWS, NVIDIA, Anthropic, and thousands of other organizations as users. That experience supplies the starting point: building agents is already practical; deciding what comes after them is harder.

An earlier version of the problem appears in Vannevar Bush’s 1945 essay As We May Think. Bush brought together experience with analog computers, photography, radio, and radar to imagine capabilities that the emerging computing industry could eventually deliver. His example matters because he was designing toward a future whose infrastructure barely existed.

Read through modern categories, Bush’s essay anticipates a remarkable collection of technologies. Kramer identifies documents displayed on screens, document scanning and OCR, speech-to-text and text-to-speech, programming languages, hypertext, search engines, and data networks. He also sees precursors to a GoPro-like camera, a Kindle Store-like service, voice interfaces, and brain-computer interfaces. These are retrospective analogies to Bush’s proposals, rather than the vocabulary of the original essay. The useful exercise is to connect technical possibilities into activities people might actually want to perform.

Today’s immediate engineering task is to assemble a coherent software stack for agents. In the No Priors × Latent Space conversation with Satya Nadella, the organizing abstraction is the harness: it defines the models, data, and tools, then maintains a loop across all three. Nadella points to GitHub Copilot, the security system MDASH, and Discovery for Science as examples of products taking this direction. Their harnesses support multiple models, supply rich context, and progressively disclose tools so that every tool definition does not have to consume tokens at every step.

Loops, tool calls, and context engineering lead toward what Kramer calls Agents++: multi-model harnesses, copilots embedded throughout software, and harnesses operating at the organization level. The unit of design grows from an individual agent to the surrounding system that determines what it knows, what it can do, and how its work continues.

The early web offers a useful comparison. In 1995, Kramer wrote HTML by hand, built web servers and indexing and search software in C, and developed authoring and management infrastructure in Perl. Web pages were the object around which this work revolved, much as agents organize AI engineering today. His enthusiasm for HTML then was comparable to his enthusiasm for agents now.

Web pages remain useful, but the applications built on that foundation became much more expansive: web applications and native mobile applications coordinate services, data, and interaction in ways an early page did not. An agent can be a durable primitive without being the final form of AI-native software. The proposed progression is agents → Agents++ → applications whose behavior is designed around AI from the beginning.

To understand how a primitive becomes a platform, return to Bush’s longer timeline. His account treats the abacus as both an everyday calculating device and a conceptual tool associated with place value and zero. From there, he moves to the electromechanical keyboard calculators available in 1945, anticipates an arithmetical machine, and then imagines the Memex. The sequence moves beyond faster calculation toward a different relationship between a person and stored knowledge.

With hindsight, Kramer redraws that timeline around the stored-program computer, the personal computer roughly forty years later, and the agent era roughly another forty years after that. The intervening decades were spent building ways to express intent and interact with machines:

Languages combined mathematical formalisms with expressions closer to natural language. Interactivity then made computing a dialogue rather than only a sequence of submitted instructions. Ivan Sutherland’s Sketchpad extended that dialogue into graphical programming.

The popular imagination developed alongside those technical foundations, even when few people had access to a computer. In the Star Trek excerpt, a captain dictates a log, the computer records it, and then repeatedly addresses him with an unwanted term of endearment despite his correction. It is a compact vision of conversational computing with personality—and of a user struggling to control that personality. Kramer also draws attention to the punch-card-reader sounds in the background: the machine makes its work audible even while speaking about it.

HAL 9000 in Stanley Kubrick’s adaptation of Arthur C. Clarke’s 2001: A Space Odyssey supplies a much darker talking-computer vision. These fictional systems made interaction imaginable before it was broadly available. Meanwhile, growing computing power and data volumes created a practical problem for the next decade: software needed abstractions that could scale.

The 1970s supplied foundations for manipulating larger amounts of data and organizing more complex software. Relational databases introduced a theoretical basis for data manipulation; declarative languages gave programmers ways to use it. Smalltalk and object-oriented programming supplied another set of abstractions. These developments helped set the stage for personal computing, including the Macintosh in 1984 and Windows 1.0 in 1985.

Microsoft’s ambition was “A computer on every desk and in every home.” The reason to put one there was the tangible work it enabled. Kramer’s central example is VisiCalc, which he introduces as the first spreadsheet program: a two-dimensional, interactive abstraction for numerical computation. Its descendants, including Excel and Google Sheets, preserve that basic way of working because it remains useful.

The comparison between a spreadsheet in 1957 and one in 1985 is therefore about access to a capability. Work that once required a roomful of people with specialized knowledge could become something a person manipulated on a screen. The interface compressed a complex computational practice into a form that many more people could use.

Kramer offers VisiCalc as a possible counterargument to fears of mass unemployment from AI. In his account, spreadsheets did not eliminate accountants; they expanded the amount of accounting-like work people could perform and enabled categories of work that had been difficult even to imagine. This is an analogy for how access can expand demand, rather than a demonstrated forecast of AI’s employment effects.

Imagine looking ahead from Macintosh System 2 and Windows 1.0 in 1985. What would computing look like ten, twenty, or forty years later? Apple’s 1987 Knowledge Navigator concept video provides one answer. In the excerpt, an assistant reads messages from a research team in Guatemala, a student asking for another term-paper extension, and the user’s mother. The user interrupts the last message by supplying the birthday reminder himself. Personal information and conversational turn-taking are already part of the same interface.

The larger concept integrates several kinds of capability:

• Device and interaction: a foldable tablet, touch input, and a conversational voice assistant with a strong personality.
• Information and perception: access to global and personal information, real-time computer vision, and real-time video generation.
• Communication and agency: seamless video calls, delegation of complex tasks for autonomous execution, and what Kramer describes as continual learning.

The significance is their coexistence. The assistant does not inhabit an isolated voice interface; it participates in a workspace containing information, visual media, communication, and ongoing tasks.

Knowledge Navigator extends Bush’s vision with another forty years of technical experience. It anticipates the agent era more directly than the Memex could, because it can draw on personal computers, graphical interaction, and a much richer understanding of software. The vision changes as the available primitives change.

Knowledge Navigator falls roughly halfway through Kramer’s computing timeline. The next major layer was networking: local area networks and dial-up connections, followed by the internet and the web. Looking back, Kramer identifies the web’s multimodal character as its most important property. Text, audio, video, and data could belong together instead of being treated as separate things requiring separate programs. For many web builders, making something like Knowledge Navigator real was a conscious ambition.

The following decade made that networked, multimodal computer mobile and continually connected. Putting it in a pocket changed where interaction could happen. Portable computers, cameras everywhere, and increasingly inexpensive screens also gave designers and filmmakers a new set of near-feasible possibilities. The hardware was approaching capabilities that could be imagined coherently before they could quite be assembled into a working product.

John Underkoffler’s work on Minority Report and Iron Man supplied two influential cinematic interfaces from this period. In the Minority Report excerpt, gestural investigative work occurs alongside spoken requests to check neighbors and relations. Information handling is embedded in a larger activity involving people, instructions, and a shared environment.

The gestures shown in Minority Report were rendered with special effects, but Kramer explains that they were grounded in Underkoffler’s PhD research at the MIT Media Lab. The underlying move was to bring the interface out of a small screen and into the world around its users. Iron Man adopts a more playful future than Minority Report’s dystopia, while retaining recognizable elements of the same spatial-interface era.

The Iron Man excerpt adds the assistant’s social presence: a personalized welcome, references to recent public events, and sarcasm aimed at its user. As with Star Trek, personality is part of the imagined interface, not merely a voice wrapped around a command parser.

Kramer and Underkoffler then tried to move that vision into commercial software, co-founding a startup in 2006. The 2012 demo reel shown in the talk represents six years of that effort. It places the cinematic vision beside the much longer engineering process of turning an interaction concept into a product.

The continuing project is a computer that is multimodal, spans devices and screens, supports multiple people, and stays connected. The cloud buildout of the 2010s added another essential layer: the infrastructure, data centers, and data capacity needed to scale AI training and inference. That substrate brings the historical sequence back to agents and to the possibility of software beyond them.

Kramer condenses the progression to calculator → computer → personal computer → global cloud computer. His assessment is that this foundation now makes fully working versions of the Memex, JARVIS, and Knowledge Navigator possible. The broader opportunity is to build those systems and learn, through using them, which new forms of software become desirable.

A recent Tavus reinterpretation of Knowledge Navigator makes the transition from concept to working interaction concrete. Kramer introduces it as built entirely with real, available technology. In the excerpt, the assistant greets the participant, comments on his houndstooth jacket, and offers to review tomorrow’s schedule. The interaction combines an observation about the present scene with access to personal information.

The participant asks to review the schedule. The assistant says it is opening the calendar and gives a short summary because it is late: an investor meeting at 9:00, followed by one-on-ones and internal meetings until 1:00. The response does more than read calendar entries aloud; it frames them as a busy morning and adjusts the amount of detail to the conversational situation.

Kramer describes the full four-minute Tavus video as a single take using real technology. The producer also describes the demonstration as single-take, while acknowledging that its assistant does not yet reproduce every capability of the original Knowledge Navigator. The example makes the old concept feel recognizable and newly practical without establishing that the entire vision is complete.

Gradient Bang, a multiplayer game Kramer is developing with friends, provides a different testbed. LLMs sit at the core of every interaction rather than appearing as an optional assistant beside an otherwise conventional product. Kramer reports hundreds of inference calls happening at every moment in the game. He also says a comparable system could not have been built a year earlier. The inference count is his description of the system’s activity; no measurement window, player count, or model configuration accompanies it.

The demonstration begins with asynchronous, non-blocking context compression. A player asks the assistant to remember an intended future elimination of another in-game character, and the assistant acknowledges the note. The requested action is deferred: recording an intention is distinct from carrying it out. The pattern lets ongoing interaction coexist with work on the context needed for later turns.

Next come long-running subagents that share context. The assistant reports Eagle working through five trade loops, while Hawk and Raptor each work through five exploration loops. The player can ask about the fleet while those activities remain in progress; conversation is an interface to ongoing work rather than a requirement that every task finish within one exchange.

The remaining examples connect orchestration to the interface:

• Progressive skills loading: a question about average ship cost introduces the pattern. The assistant begins explaining that ship prices vary; the excerpt supplies no numeric price.
• Dynamic UI generation: the player requests task history, then asks to hide the map. The interaction can change what the interface presents as well as produce spoken responses.
• Conversational voice: the player corrects the intended transaction from exchanging a ship to selling it for cash. The assistant must follow the revised intent while still respecting the game’s rules.

The last request reaches a constraint: the assistant refuses a sale that would leave the player without a ship.

That final interaction separates understanding an action from permitting it. A small Python rule illustrates the boundary for the requested sale:

def validate_ship_sale(owned_ship_count: int) -> str | None:
    if owned_ship_count <= 1:
        return "Sale refused: you must retain a ship."
    return None

requested_action = "sell ship for cash"
refusal = validate_ship_sale(owned_ship_count=1)

if refusal is not None:
    print(refusal)
else:
    print(f"Permitted for further processing: {requested_action}")

Here, passing validation would only permit further processing; it would not execute a sale. In the demonstration, the corrected request is understood but refused because the galaxy does not allow the player to become shipless and hitchhike. The conversational interface can be flexible while the world it operates on retains firm constraints.