AI Engineer
← All AI Engineer talks

AI Engineer Europe 2026

Building Conversational Agents

Thor Schaeff· Developer Relations Engineer, Google DeepMindPhilipp Schmid· Google DeepMind1:47:34

Read the talk

Building Conversational Agents: From Stateful Tool Loops to Live Audio

Build a small coding agent, give a conversational model tools and vision, and follow the deployment and reliability tradeoffs exposed by the live demonstrations.

From a talk by Thor Schaeff and Philipp Schmid

Before you start: You should be comfortable reading Python, using environment variables, and following JSON tool schemas and WebSocket connections.

011:43 ↗

Start with a key—and a device that can use it

What does it take to connect an application to Gemini? The first practical step in Thor Schaeff and Philipp Schmid’s workshop is a secret API key. Both work on developer experience at Google DeepMind, spanning the Gemini API and Google AI Studio. AI Studio provides a place to try models and create credentials; using Antigravity itself does not require this same key setup.

The setup proceeds through AI Studio, reached through ai.dev:

  1. Open Get API key, also available through ai.dev/api-key.
  2. Create or import a project.
  3. Select the project and create an API key.

Philipp describes the core hands-on exercises as accessible with a Google account and the free tier, without a credit card. The selected frame shows the project-created notification and the key-creation dialog, before a credential is exposed.

Google AI Studio shows a project-created notification and a Create new key dialog with the AIE workshop project selected.
Creating an API key after selecting the workshop project.

An existing key works too. The workshop mentions shell configuration such as .bashrc or .zshrc, as well as inline configuration for the demo. The essential constraint is that the key remains secret: do not share it or commit it to GitHub. Philipp deletes the key he exposes during setup.

While credentials are being created, Wolfram Rebenwolf from Thursday AI describes using Gemma and Gemini for coding and agents, including an experiment on glasses. A brief Chrome tip—right-click the tab bar to move tabs vertically—recovers some laptop screen space. The glasses discussion then becomes concrete: VisionClaw connects Gemini with OpenClaw tools. The glasses connect to a phone, and the phone maintains the Gemini Live WebSocket connection. The Meta Ray-Ban SDK makes that integration possible; the attendee describes his setup as experimental. Another attendee, Michael from GetYourGuide in Zurich, is building an initial AI customer-support agent. These are two distinct destinations for the same ingredients: conversational input, retained context, and tools that act outside the model.

1:431:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:43 · section reference included

028:39 ↗

One interaction surface for models and agents

The first build is a small coding agent that can read files, write files, and run shell commands. Live audio comes afterward. Philipp introduces the Interactions API, launched in public beta in December 2025, as the intended direction beyond generateContent. That is a direction of travel, not a claim that the established API had already been retired. The new interface follows conventions familiar from Chat Completions, Anthropic, and OpenAI.

A request selects either a model, such as Gemini 3 Flash, or an agent, such as Deep Research. The latter can plan and conduct an extended research task rather than return a single immediate model response. Philipp describes runs lasting roughly 10–15 minutes and visiting hundreds of sites. Bringing or defining custom agents is presented as future work. The broader goal is to chain outputs across models: Nano Banana for images, Flash for further processing, Lyria for audio, and potentially Veo for video.

The structural simplification is typed content blocks. Inputs and outputs use a common pattern with an explicit type, covering text, images, audio, video, function calls, and thought signatures. Compared with a protobuf-style structure in which fields such as text or inline data imply the content type, this gives application code a more direct way to inspect and dispatch content.

8:398:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:39 · section reference included

0311:24 ↗

Keep history on the server without surrendering context control

With server-side state, the client sends new input together with a previous interaction ID. The server attaches that input to the retained conversation. Long-running agent work can use background: true and polling instead of holding a connection open; webhooks are described as forthcoming. Streaming uses server-sent events. Built-in tools, remote MCP, and the recently introduced combination of Google Search with custom functions extend the same surface.

ConcernServer-managed historyClient-managed history
Follow-up requestNew input plus previous IDExplicit conversation input
Context editingContinue retained historyRemove or rewrite selected content
ReusePreserve existing contentPreserve it carefully when rebuilding

Server-side state is optional. If an application needs to remove parts of its context, it can still manage and submit the history itself. Background execution addresses a different concern: Philipp’s examples involve agent tasks taking minutes, for which polling or notification avoids tying completion to a long-lived HTTP request.

Retaining history also helps implicit caching. A model must process the input tokens before generating a response; cached processing can be reused on a follow-up request. Philipp estimates that cached input tokens are 90% cheaper and reports two to three times better cache-hit rates among startups using Interactions API. His explanation is that rebuilding history on the client invites small changes—removed line breaks, stripped whitespace, or deleted content—that can disrupt reuse. The reported improvement concerns those startups, not a controlled comparison supplied with the workshop. The ensuing code comparison reinforces the API’s other simplification: explicit typed blocks instead of oneof-style input fields.

11:2411:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:24 · section reference included

0414:31 ↗

The loop connects reasoning to action

An agent has four working parts: a model chooses what to do; tools interact with its environment; context describes the task, capabilities, preferences, and constraints; and a loop keeps those parts working until the model stops requesting tools. Server-side state simplifies history management, but the client still runs the tool loop.

For ordinary chat, interactions.create receives a model and input—for example, a question about the capital of France. A follow-up supplies the returned ID, so the server already has the earlier user input and model response. The same continuation mechanism can cross model boundaries: the slide example researches AI agents in 2026 with Deep Research, then asks Nano Banana to generate a visual from that research. Applications that need manual control can instead provide an array of turns with user and model roles.

A function declaration supplies its type, name, description, and parameter schema. When an interaction requires action, the application must do the work:

  1. Inspect the interaction’s outputs.
  2. Find function-call blocks.
  3. Execute the corresponding local functions.
  4. Submit their results in another interaction.
  5. Repeat until the model produces its final response.

A function call is a request to act; the function result records what the application actually did. The slide places that distinction inside the complete request–execute–respond cycle.

The Agent Loop slide shows five workflow steps, notes about history and thought signatures, and a Python code example.
The agent loop: request, check outputs, execute tools, return results, and repeat.
14:3114:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:31 · section reference included

0517:29 ↗

Give the coding agent documentation it can use

The workshop uses a preferred IDE agent or CLI to write the implementation. Before asking it to build anything, install the relevant Gemini API skill. The documentation’s coding-agent setup distinguishes general Gemini development, Live API, and Interactions API skills. For this first exercise, choose Interactions. A project-local install is sufficient; the demonstrated location is .agents/skills/gemini-interactions-api.

Philipp has tested discovery with Cursor, Antigravity, and Gemini CLI. The discussion of Claude Code’s .claude versus .agents discovery is less settled, so the useful check is behavioral: ask the coding agent which skills it can use. Antigravity identifies the installed Interactions skill. Context7 from Upstash and Vercel’s skills.sh provide installation routes backed by the same GitHub repository. The deliverable itself is a Python script, which Philipp expects can also run inside an attendee’s sandbox-based workflow.

The skill is deliberately smaller than a copied API manual. It addresses things the model may do unreliably, such as choosing an obsolete model, and workflow preferences, such as using Bun for tests. Available-model guidance prevents stale Gemini 1.5 defaults. For detailed API behavior, the skill points to Markdown documentation that the agent can fetch. When a feature changes, the documentation can be updated without requiring every user to reinstall a large skill. Asked about the extra fetch, Philipp compares it with reading a local reference file: either approach requires retrieving the material before using it.

17:2917:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:29 · section reference included

0625:09 ↗

Build the smallest stateful agent

The implementation request specifies an Agent class in workshop.py, rather than simply asking for an agent. Its constructor creates a GenAI client, selects a model, and initializes the retained previous interaction ID. Its run method sends input and updates that ID. A main function exercises the class. During generation, Philipp corrects the coding agent to use the workspace’s uv environment; the required package is google-genai, installable with pip or uv pip install.

This Python version follows the recording’s beta-era outputs response shape. The current Interactions documentation uses execution steps and model_output instead; it also specifies that tools, system instructions, and generation configuration must be supplied again on subsequent requests. A previous ID retains conversation history, not those request settings.

python

import os
from google import genai


class Agent:
    def __init__(self, model: str):
        self.client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
        self.model = model
        self.previous_interaction_id = None

    def run(self, text: str) -> str:
        request = {"model": self.model, "input": text}
        if self.previous_interaction_id is not None:
            request["previous_interaction_id"] = self.previous_interaction_id

        interaction = self.client.interactions.create(**request)
        self.previous_interaction_id = interaction.id
        return "\n".join(
            output.text
            for output in interaction.outputs
            if output.type == "text"
        )


if __name__ == "__main__":
    agent = Agent(model=os.environ["GEMINI_MODEL"])
    print(agent.run("My name is Phil."))
    print(agent.run("What is my name?"))

In the generated workshop code, the model defaults to Gemini 3 Flash. The important state transition is the assignment of interaction.id: the next call points to the interaction that just completed. The screenshot shows that update and the name-recall test awaiting execution.

Code editor showing the interaction ID update and a main function that introduces Phil, then asks “What is my name?” A run command awaits approval.
The generated code stores the latest interaction ID and defines a two-turn name-recall example.

The first run produces an experimental-API warning. A moment of confusion follows because one prompt asks for the model’s name rather than the user’s; when the question asks for the user’s name, the response recalls it. Philipp credits precise instructions, the installed skill, useful context, and actually running the script for the coding workflow’s success. Participants also report working TypeScript implementations and even phone-based participation.

25:0925:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:09 · section reference included

0730:11 ↗

Add file tools, then close the execution loop

The next request adds readFile and writeFile. Each tool needs two things: a JSON schema that tells the model which arguments to generate, and a Python function that executes those arguments locally. A tools map associates the name with both. During generation, the coding agent discovers an existing solution folder and uses its implementation as guidance—an instructive reminder to inspect what context an agent actually consulted.

The basic file implementation is small enough to see the entire model-to-code contract:

python

from pathlib import Path


def read_file(file_path: str) -> str:
    return Path(file_path).read_text(encoding="utf-8")


def write_file(file_path: str, content: str) -> str:
    Path(file_path).write_text(content, encoding="utf-8")
    return content


TOOLS = {
    "readFile": {
        "function": read_file,
        "schema": {
            "type": "function",
            "name": "readFile",
            "description": "Read a UTF-8 file and return its contents.",
            "parameters": {
                "type": "object",
                "properties": {"file_path": {"type": "string"}},
                "required": ["file_path"],
            },
        },
    },
    "writeFile": {
        "function": write_file,
        "schema": {
            "type": "function",
            "name": "writeFile",
            "description": "Write UTF-8 text to a file.",
            "parameters": {
                "type": "object",
                "properties": {
                    "file_path": {"type": "string"},
                    "content": {"type": "string"},
                },
                "required": ["file_path", "content"],
            },
        },
    },
}


def execute_tool(name: str, arguments: dict) -> str:
    if name not in TOOLS:
        raise ValueError(f"Unknown tool: {name}")
    return TOOLS[name]["function"](**arguments)

The model receives the schemas; the application retains the Python callables. These file functions directly access the process’s filesystem, so the environment in which the exercise runs determines what they can read or overwrite.

The agent’s input now accepts either text or a list of content blocks, because a follow-up may contain function results. Each model request includes the tool schemas. The client inspects interaction.outputs, which can include reasoning-related content and thought signatures as well as function calls. With server-managed state, those signatures do not have to be reconstructed and sent back manually. Before dispatching a call, the application checks that the requested tool exists: a request for an unsupported edit-file tool must not become an arbitrary invocation.

Each executed function produces a corresponding function result. The generated implementation recursively calls self.run when it has tool results to submit; otherwise it returns the interaction’s text. The test asks the agent to write Hello from the agent to hello.txt and read it back. The demonstrated trace contains a writeFile call, its result, a readFile call, and the final response. That trace is stronger evidence of filesystem work than a sentence claiming that a file was created.

30:1130:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:11 · section reference included

0835:01 ↗

Use conversation to expose missing instructions

An outer stdin loop turns the one-shot example into an interactive application. It waits for user input, calls agent.run, and prints the result. Inside that call, the tool loop continues until the model finishes. A greeting correctly produces a conversational response without a file operation. A request for a thumbs-up SVG is more revealing: the model initially supplies content but hesitates to write it to disk, even though the tools are available.

Philipp proposes making the agent’s role explicit: it is a coding assistant that can use tools to interact with the local filesystem. A write call eventually appears in the existing conversation, and he then asks the coding agent to add system instructions to the implementation. The skill remains useful here because it tells the coding agent how this API accepts those instructions. Philipp says Gemini 3 Flash’s training predates the Interactions API, attributing the implementation to supplied documentation and coding infrastructure rather than memorized API examples.

The generated configuration permits instructions at construction or when running a request. After restarting with a coding persona and explicit filesystem access, the same thumbs-up SVG request produces a writeFile call, and the resulting icon is displayed. Tool availability and instructions about when to use tools are separate parts of the application. This exercise improves the latter without establishing the cause of every earlier hesitation.

The final coding capability is a shell-command tool implemented with Python’s subprocess, returning standard output. An attendee reports hallucinated tools during their own testing, while Philipp explicitly sets security aside for this shell example. The generated change also updates the system prompt. A proposed Nano Banana image test is set aside because the agent has no corresponding skill or guidance; instead, asking for the time invokes date and returns Wednesday, April 8. A suggestion to delete all files is rejected, not executed.

35:0135:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

35:01 · section reference included

0942:55 ↗

Branch history, but account for expiry and context limits

Keeping interaction IDs on the client enables branching. Instead of always continuing from the latest turn, select an earlier ID and start a different continuation. Philipp suggests using an initial web-research result as the common base for several parallel requests. To export history, interactions.get retrieves an interaction and its previous ID; following those links walks backward through the conversation.

Philipp states default interaction retention of one day on the free tier and 55 days for paid usage. Storage duration is distinct from context capacity: he describes a million-token Gemini context limit and says exceeding it produces an error. Compaction remains the client’s responsibility in this demonstration. An eight-hour pause can still leave a free-tier interaction available; after its retention period expires, the old ID can no longer be found. This is expiry of stored interactions, not a global daily conversation reset. Vertex support and additional flexibility are prospective, and no dedicated context-reworking API is offered in the discussion. An audience suggestion to speak an API joke through eSpeak prompts a mention of Gemini text-to-speech before the handoff to live audio.

A follow-up caching question sharpens the earlier explanation. Philipp describes reuse at an object level rather than treating an entire interaction as indivisible: a 4,000-token PDF might be cached while a ten-token accompanying text input is not; later, the PDF and some follow-up turns might both be reusable. The example is conditional. Whitespace changes can still disrupt reuse, and request routing and the timing of follow-ups affect whether a cache hit occurs. Preserving the conversation exactly improves the opportunity for reuse without guaranteeing it.

42:5542:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

42:55 · section reference included

1049:02 ↗

A live DJ with a music-generation tool

Thor opens the audio portion with Gemini 3.1 Flash Live, a newly released native-audio model. He describes architectural work aimed at latency and scalability. The Live API remains a separate interface in the workshop; bringing it into Interactions is work in progress.

The first application combines Live conversation with Lyria 3. Thor describes its clip model as generating 30-second songs, including lyrics, alongside a separate full-song model. Live Jukebox gives the conversational model a DJ persona and a music-generation tool. Its inspiration is the BBC Radio 1 phone-in experience: talk to the host, request a genre and mood, and hear the result.

Thor requests high-energy German techno schlager about the UK AI scene. The DJ asks for more detail, accepts a request to be surprised, and introduces the generated track. The lyrics bring together London, robots in bowler hats, builder’s tea, and German flourishes. A second audience request asks for a nursing song in Swahili; Thor adds hardcore techno. Music plays, but he acknowledges that its vibe is wrong. The demonstration exercises conversational clarification and tool use while also exposing a mismatch between the requested style and the result.

Live Jukebox was built and published in AI Studio. Unlike the core free-tier exercises, Thor says this demo requires a paid API key with billing enabled because it generates music.

49:0249:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

49:02 · section reference included

1154:49 ↗

What travels over the Live connection

Thor treats live-audio benchmarks as an incomplete guide to application behavior and encourages trying the interaction itself. He identifies the model as powering Gemini Live on phones and is less certain about Search Live. The Live API provides a stateful WebSocket connection accepting text, streamed audio chunks, and video frames. The demonstrated Live API accepts video at a maximum of one frame per second. A frame can come from a camera, a canvas, or screen sharing. Thor’s Shopify Sidekick example uses screen context to guide a merchant through setting up a custom domain.

The return stream contains audio buffers and can include transcription. Tool calling adds external capabilities, including Google Search grounding. Underneath the conversation, the model processes sound tokens directly rather than requiring a speech-to-text → text-model → text-to-speech cascade. Thinking levels adjust how much reasoning precedes a response. Thor estimates support for 97 languages in preview and highlights mixed-language conversation, such as German and English within the same exchange. Automatic voice activity detection enables barge-in: the user can interrupt while the model speaks.

The direct transport is WebSocket-only in this demonstration. Thor contrasts that with GPT Realtime’s direct WebRTC infrastructure. Applications that want WebRTC can use integrations from LiveKit, Pipecat, Fishjam by Software Mansion, Vision Agents, or Voximplant. That choice determines how much media transport infrastructure the application must assemble around the model connection.

54:4955:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

54:49 · section reference included

1259:24 ↗

Prompt the voice, then test its access to facts

In AI Studio’s Live view, Thor enables the webcam and types a question about his outfit. The response identifies a green jacket, blue T-shirt, and black cap. He then changes the delivery: the base voice is Puck, and the system instruction asks for a friendly Irish accent. Thor describes roughly 30 base voices, with prompts supplying additional style. The subsequent outfit conversation uses the requested Irish manner of speaking.

Current weather requires more than audio understanding. Thor turns to Google Search grounding and explains the thinking control: the interface’s no-thinking setting means minimal thinking, while additional reasoning increases response latency. He then asks for London’s current weather. The model first claims it is about nine degrees and cloudy. Challenged to identify the date and look again, it identifies April 8, 2026, then gives a forecast of five to thirteen degrees with rain. Thor disputes the answers and suspects a grounding or UI problem. No repair is established in this sequence. A fluent spoken answer does not establish that the intended information source was used.

59:2459:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

59:24 · section reference included

131:04:00 ↗

Choose a generated app or a working example

AI Studio’s real-time voice option supplies one starting point. Thor asks it to build a multilingual interview-practice assistant for languages including German, English, and Spanish. He describes this builder as using Gemini 3.0 Flash Preview and supporting JavaScript full-stack environments such as Next.js and Angular, with XR building blocks for glasses and web VR experiences. The generation starts in the background; the recording does not return to a completed interview app.

The alternative is to start from the Live API documentation, which links AI Studio, coding-agent skills, and GitHub example apps. Thor clones the examples into a directory named AINGEU and opens them in Antigravity with agy. The repository presents two architectures worth comparing: a server that proxies media, and a server that issues credentials for a direct client connection.

Gemini API documentation with an App-to-Live-API connection diagram and buttons for Google AI Studio, GitHub example apps, and coding agent skills.
The Live API documentation offers AI Studio, GitHub examples, and coding agent skills as entry points.
1:04:001:04:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:04:00 · section reference included

141:07:28 ↗

Route browser media through a Python server

The Python SDK example uses a browser → application server → Gemini path. The browser captures microphone and camera input. A FastAPI WebSocket receives audio, video, and text queues, and the backend forwards them into a server-side Live session. A separate Gemini Live module contains the model integration, keeping it apart from the browser-facing transport.

LiveConnectConfig holds system instructions, guardrails, and tool definitions. After setting up the session, the server forwards the client’s queued media through it. To launch the example, Thor creates and activates a uv virtual environment, installs dependencies, copies .env.example to .env, and supplies GEMINI_API_KEY. Running main.py starts the demo at localhost:8000.

The running application opens with an Irish-accented greeting and responds to the camera view. Thor requests German, then Chinese, and asks about eating. The exchange appears to be understood, but he points out missing transcription. He also notices worse latency and attributes it to the extra server hop. That observation motivates removing the application server from the media path.

1:07:281:07:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:07:28 · section reference included

151:14:09 ↗

Connect directly using an ephemeral token

A direct browser connection must not expose the long-lived secret API key. The second example keeps that key on a small backend, which creates an ephemeral token and returns it to the browser or phone. The client uses that short-lived credential to establish its own WebSocket connection to the Live API. The token server still needs its virtual environment, dependencies, and API-key configuration; it no longer forwards the media stream.

In the demonstrated version, token creation and the bidirectional WebSocket endpoint use v1alpha. An expiry limits how long a leaked token remains useful. The frontend uses raw WebSockets without an SDK, passes the token as its access credential, and handles the incoming events itself.

ArchitectureBackend responsibilityMedia path
Server proxyHold key and run Live sessionBrowser → backend → Live API
Ephemeral tokenHold key and mint expiring tokenBrowser → Live API

The network inspector makes the direct sequence visible: first a response containing a token and its expiry, then the WebSocket connection, then a setup message with the model, real-time input configuration, and tools. The Live model’s published identifier is gemini-3.1-flash-live-preview. Connection establishment and session configuration are separate steps; having a socket open does not by itself show that the desired tools were configured correctly.

1:14:091:14:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:14:09 · section reference included

161:19:49 ↗

Separate successful actions from unresolved failures

The direct-connection demo does not fix the weather problem. Asked whether an umbrella is needed in London, the model says it cannot obtain the current weather and suggests checking elsewhere. Explicitly asking it to use Google Search does not recover the result. Thor moves on to test custom tools.

Those tools produce visible effects: a request displays a Hello World alert, and another changes the page background to green. After further configuration changes, however, a request to restore white receives a refusal—the model says it cannot control display or page settings. Thor revisits the grounding and custom-tool toggles without establishing a diagnosis. Another attendee also reports that Google Search grounding is not working. The successful browser actions remain useful demonstrations, but neither the search failure nor the later loss of tool behavior receives a confirmed fix.

The implementation entry points remain the same after the failures: the Live documentation links AI Studio, the example repository, and coding skills. The useful artifact is a connection and event-handling structure to inspect and extend, not an assumption that every configured capability has already passed an end-to-end test.

1:19:491:19:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:19:49 · section reference included

171:25:07 ↗

Longer sessions require a policy for forgetting

Asked about Live context, Thor states uncompressed session limits of 15 minutes for audio-only input and two minutes for audio-video input. He describes a go-away notification around session termination, then introduces context-window compression as a way to extend conversation.

Context-window compression trades older context for continued interaction. In Thor’s explanation, a sliding window retains a configured amount of recent material and drops what precedes it. The quantity of video matters: adding frames consumes context that would otherwise remain available for conversation. Audio-only sessions can therefore retain more conversational history than sessions continually ingesting images. Extending a session does not mean preserving everything said or seen within it.

1:25:071:25:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:25:07 · section reference included

181:26:57 ↗

Choose conversational flow and control deliberately

The business-use question brings several applications into view. Thor describes Shopify Sidekick as already in production. Stitch supports voice-directed design, though he points out that it is a Google-built product. Waymo integration is prospective. He then shows Ato, an elder-companion example combining voice interaction with an app for caregivers or family members. He describes the startup as Argentina-based and emphasizes multilingual conversation; the product page shows a fabric-covered companion device beside an AI care-companion headline. A brief case-study playback introduces a grandmother named Maria.

Ato website shows a fabric-covered companion device on a side table beside the heading The AI Care Companion that keeps aging adults sharp.
Ato’s AI care companion product page.
ApproachPrincipal benefitIntervention point
Native audioNatural conversational flowLess fine-grained control before speech
Cascading pipelineObservable intermediate stagesInspect or rewrite text before synthesis

Thor suggests that a cascading pipeline may fit some current business requirements better. Its separate transcription, model, and speech stages make it easier to see what happened and modify a response before it is spoken. Native audio offers a more immediate conversational experience, but that does not make it the right fit for every workflow.

Transcripts are another application responsibility. Thor says the Live session does not provide a transcript archive that can simply be retrieved afterward; the application must store the events it needs. He points to LiveKit and Pipecat for surrounding capabilities such as audio and transcript storage and additional observability.

For recruitment, he suggests selected interview stages or screening rather than replacing the entire interview pipeline. The audience raises criteria, conversational guardrails, and Google Meet integration with existing transcripts. The answer supplies no concrete Meet integration. Thor instead emphasizes evaluating the use case and building the required surrounding controls, including requirements such as SOC 2. He characterizes the preview as ready for experimentation, with production suitability depending on that work.

1:26:571:27:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:26:57 · section reference included

191:33:22 ↗

Recognizing speech is not authorizing its speaker

An attendee describes a practical hazard: during a voice-agent demo, someone nearby could tell a coding agent to delete a file. Thor does not claim a reliable capability that authorizes only one enrolled speaker. Proactive audio can filter out irrelevant conversation, but he says it is not a dependable way to accept only the owner’s voice. The attendee reports experimenting with NVIDIA Parakeet and a short voice sample for speaker differentiation; that remains an attendee-reported approach, not a demonstrated Live API feature.

A separate question asks what the model’s thinking looks like. Thor describes opt-in thinking information as text events on the WebSocket. It is not spoken aloud as part of the audio response.

For coding dictation, the relevant problem may be grounding rather than continuous conversation: the transcription should use the class names visible in the editor. If fully real-time interaction is unnecessary, Thor recommends Gemini 3 Flash with contextual inputs. The discussion gives a concrete multimodal request: supply main.py and util.py as text alongside the audio recording. The model can use both the spoken request and the exact source identifiers. Stitch’s spoken edits to a visual mockup illustrate the same principle: visual context disambiguates which button the user means.

Live interaction can also ingest text or images, but those inputs consume context. For a static editor view, the suggested sequence is to send one image, then several seconds of audio, without resending the unchanged image every second. Philipp estimates roughly 1,200 tokens per image. The optimization is driven by information change: send a fresh frame when its contents or relevance change, rather than treating continuous image streaming as mandatory.

1:33:221:33:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:33:22 · section reference included

201:39:23 ↗

Personalization, evaluation, and storage have separate boundaries

A civil-engineering example initially sounds like a question about supplying specialist knowledge. Thor suggests putting that knowledge into the initial context or providing function calls that retrieve it during conversation. The questioner then clarifies: can the agent gradually adapt to what it learns about the user’s expertise? Thor’s answer is yes within the active context window. The model can remember names and information from the conversation while that material remains in context. That conversational adaptation is different from durable memory or trusted-speaker authorization.

The evaluation question receives a requirements-oriented answer rather than a testing protocol. Thor names HIPAA, SOC 2, function-call behavior, and guardrails as factors that change what a business application needs. He recommends looking at partner infrastructure such as LiveKit for the surrounding voice-agent capabilities. The workshop’s enjoyable demos are therefore starting points for evaluation, not substitutes for defining what the application must reliably do.

Returning to Interactions, an attendee asks when it will reach Vertex. No date is given; the timing is outside the presenters’ control. They expect the same API surface, so experimentation can begin on the Gemini API, with requests for higher rate limits handled separately. Applications that require Vertex-specific enterprise features may need to wait.

A question about PII and data sovereignty exposes the cost of opting out of interaction storage. store=false disables storage of the interaction resource, which removes server-side continuation and background-execution features. It should not be read as a universal guarantee about every aspect of processing or retention. Philipp expects future Vertex data-sovereignty behavior to resemble generateContent, but presents that as an expectation rather than an established capability.

1:39:231:39:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:39:23 · section reference included

211:45:16 ↗

Return to the failed weather answers

The final substantive question returns directly to the weather demonstrations: entertaining failures are one thing, but how should an application handle hallucinations in production? Thor acknowledges that the demos lack best-practice system instructions. He opens guidance on defining the agent’s persona, conversational rules, tool behavior, and examples.

His proposed next step is to structure the system prompt with explicit guardrails, guidelines, and tool definitions so the agent better follows its intended operating boundaries. That is a concrete implementation task, but the recording does not demonstrate a repaired weather lookup or elimination of hallucinations. The remaining work is to make the instructions and tool configuration explicit, then establish through the application’s behavior that they are being followed.

Gemini API documentation lists guidance on agent persona, conversational rules, tool calls, and examples under Design clear system instructions.
Live API best practices for clear system instructions.
1:45:161:45:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:45:16 · section reference included

Resources

From the talk

Further reading

  • Interactions API launchArticle

    Philipp Schmid's coauthored introduction to the public beta, server-managed state, background execution, and Deep Research.

  • Building with Gemini 3.1 Flash LiveArticle

    Thor Schaeff's coauthored launch overview, including voice-agent examples and integration partners.

Updates since the talk

Read the complete timestamped transcript
  1. 0:00

    [upbeat music] Hello, everyone.

  2. 0:15

    Hello.

  3. 0:17

    Uh, heute machen wir alles auf Deutsch. Is that-

  4. 0:20

    Super. [laughing] [laughing] Perfect.

  5. 0:25

    It's just that Philipp and I were both [REDACTED:origin], so we thought it was funny. Maybe we, we can do it in [REDACTED:origin]. Actually, it looks like there's a, there's a [REDACTED:origin] crew there, which is nice.

  6. 0:34

    Uh, no, no worries. We'll, we'll, we'll do it in English. We'll do it in a couple different languages maybe. We'll find out. Do we have other languages in the room?

  7. 0:43

    Other nationalities? Yeah, what do we have? Sh- shout it out.

  8. 0:47

    Serbian. Italian.

  9. 0:49

    Serbian. Italian. Spanish.

  10. 0:51

    Spanish.

  11. 0:51

    Any Icelandic? [laughing] No? Dane- Okay, close. Close.

  12. 0:58

    Romanian.

  13. 0:59

    Romanian. Nice.

  14. 1:00

    Dutch.

  15. 1:01

    Dutch.

  16. 1:01

    [REDACTED:origin].

  17. 1:02

    Okay.

  18. 1:03

    [REDACTED:origin].

  19. 1:04

    Uh, which?

  20. 1:05

    Delhi. [REDACTED:religion].

  21. 1:06

    Yeah, but [REDACTED:religion]. Okay. Canada. No?

  22. 1:10

    No, no.

  23. 1:10

    Okay. Bangalore. No? Uh-

  24. 1:13

    Farsi.

  25. 1:14

    Farsi. All right, nice.

  26. 1:16

    Czech.

  27. 1:17

    Czech, yeah. Brilliant. Okay, this is, uh, fantastic. [laughing] Well, thanks everyone for making your way over here with, with all your languages. Really appreciate it. We can, we can really put the model to the test today, which is great.

  28. 1:30

    Um, yeah. Hi, I'm Thor, or Torsten for the [REDACTED:origin] speakers amongst you. Um-

  29. 1:36

    Hi, I'm Philipp. Um, only Philipp. [laughing] So [laughing] for [REDACTED:origin] and English speakers, so. [laughing]

  30. 1:43

    Yeah. It's nice. Uh, we... So we work on the developer experience at Google DeepMind, broadly covering kind of Gemini API, and also, uh, working Google AI Studio as a tool for-

  31. 1:55

    Yeah

  32. 1:55

    ... you know, developers to try out the models quickly, uh, and then also the, the API, um, interfaces. To use the API, you do need an API key. Uh, actually, who has an API key already?

  33. 2:08

    Gemini API. Okay, a couple folks. Uh, who, who has used AI Studio before? Google AI Studio. Okay, a couple folks. Great. Um, maybe we can quickly-

  34. 2:19

    Antigravity.

  35. 2:19

    Sorry?

  36. 2:19

    Antigravity.

  37. 2:20

    Yeah, that's good. Uh, you actually don't need an API key for Antigravity. But, um, so if you don't have an API key yet, uh, if you have your machine on you, I hope you do, uh, it's, it's, it's a hands-on workshop.

  38. 2:36

    I do apologize, they took away your tables.

  39. 2:38

    Yes.

  40. 2:39

    Um, so it's a very laptop situation. Uh, like literally, uh, laptop. Um, I hope you didn't bring your Mac Mini or, you know, whatever. [laughing] Uh, but so yes, if you just go to ai.dev.

  41. 2:55

    Um, and I mean, there's... You can also go to ai.studio. You can go to aistudio.com, but we paid a lot of money for ai.dev, so please use that. [laughing] Uh, it just, it just redirects to AI Studio.

  42. 3:08

    But, um-

  43. 3:10

    Yes. Let me-

  44. 3:10

    So ai.dev/api-key.

  45. 3:13

    Uh, yes. Or on the-

  46. 3:15

    Or

  47. 3:15

    ... corner left side, there's a Get API key, and then api-keys. It's where you can-

  48. 3:21

    Or in [REDACTED:origin], Schlüssel.

  49. 3:22

    Uh, yeah. It's, it's my personal account, and we cannot change language, so [laughing] Uh, you need only a Google account, so no credit card, no nothing. All of the things we are going to do is part of the free tier.

  50. 3:35

    So if you go to AI Studio or ai.dev, and there's some, like, sign-in form, you can, like, just use your Gmail, no worries. We will not charge you. Uh, and then on api.keys, uh, you find at the top right corner normally something called API key create or Create API key.

  51. 3:53

    Uh, and when you do it for the first time, you, uh, might need... So similar for me, uh, I can import my projects or I can create my projects.

  52. 4:00

    I can just click Create, give it any name. I mean, we can call it, uh, IEI

  53. 4:09

    Workshop, uh, and then Create Project. Maybe I can slide it to English.

  54. 4:19

    Okay. Takes a few second, and then you should be able to create key, and this key will be used for the demos or the hands-on things we are going to do use later.

  55. 4:32

    Wait, it shows the API key?

  56. 4:33

    Yeah, I will delete it, so don't, don't copy it. [laughing] [laughing] But you can like-

  57. 4:37

    Copy very fast.

  58. 4:38

    Yeah.

  59. 4:39

    Take photo.

  60. 4:39

    You can use that key. You can, like, uh, put it into your Bash RC or Zsh RC file, or later you can, like, directly inline it it, whatever you prefer.

  61. 4:47

    If you have one, you, you can use just one. And since we have a few more minutes time, we really want to make sure everyone who wants to follow along, take your time, create your API key.

  62. 4:58

    If there's an error appearing or something else is not working, feel free to raise your hand. Thor will come to you, and we'll help you. And-

  63. 5:07

    And yeah, just a reminder, it is a secret API key.

  64. 5:10

    Don't show it.

  65. 5:11

    So don't do what Philipp is doing there. Uh-

  66. 5:13

    That's why I delete my key again.

  67. 5:14

    Yes. [laughing] So we do get that a lot, that people, um, leak their API keys. Mostly it's clock code that is, like, pushing the API keys to- [laughing] ... to, to GitHub.

  68. 5:25

    So I recommend you don't do that. No, just, just kidding. Just remember, it is a secret API key. Um, so treat it like a secret. Uh, don't share it with your neighbor.

  69. 5:36

    Uh-

  70. 5:36

    Yes

  71. 5:36

    ... um, but yeah, do create that API key now, and we'll give you a couple minutes to do that. And while you do that, or, you know, once you've finished creating your API key, I'd love for you, you know, to just briefly introduce yourself if you want to, and just sort of, you know, let us know what

  72. 5:55

    you would like to get out of the workshop. Maybe there's a specific use case you're working on. Um- Yeah. We'd love to kind of get to know you all a little bit as well.

  73. 6:04

    So while you create your API keys, uh, if you want to, feel free to just, you know, shout out, uh, your name or nickname, uh, and sort of what you're working on, what you would like to get out of the workshop.

  74. 6:17

    Yeah. Wolfram Rebenwolf from the Thursday AI podcast, working for Corby for Weights & Biases, and a big fan of Gemma, uh, and, you know, Gemma Four and Gemini. I'm one of the few who are using it for coding and stuff as well.

  75. 6:30

    Nice.

  76. 6:30

    For agent use. And, um, I'm also running it on the glasses, actually.

  77. 6:34

    Oh, nice. Mm.

  78. 6:34

    Um, and I want to see more of this, you know?

  79. 6:37

    Okay.

  80. 6:37

    And I want Google Glasses.

  81. 6:40

    Okay. Yeah. We'll, we'll see if we can get to the glasses by the end of the workshop.

  82. 6:45

    Thank you.

  83. 6:45

    Sorry. Yeah.

  84. 6:46

    If you have a seat to your right, can you jump in so he can, uh, come up on the side here? Makes it easier to be able, um, to find one.

  85. 6:54

    And while we are waiting, uh, we launched something cool on Chrome, which I haven't activated it, but when you go on your tab bar, click right, you can now move tabs to the side, and we have vertical tabs now, so... [clapping] [laughs]

  86. 7:09

    Yay. So more screen. That's good.

  87. 7:14

    Nice. Okay, cool. So glasses.

  88. 7:18

    Gemma.

  89. 7:18

    Are you, are you, are you running... What are you running on the glass? Well, technically on the phone, right?

  90. 7:24

    Vision Claw is the software where you use Gemini-

  91. 7:26

    Yeah

  92. 7:26

    ... which calls OpenClaw-

  93. 7:28

    Nice

  94. 7:28

    ... customization for tool.

  95. 7:30

    And then that's just using Gemini Life kind of through the WebSocket on the phone, right? Yeah, nice. Yeah, that's cool. Yeah, Vision Claw, if you haven't heard of that, a pretty, pretty fun open source project.

  96. 7:40

    Um, and I think you can run it, yeah, sort of on the... Well, now that, uh, Meta has opened up the SDK for the, the Meta Ray-Ban, uh, you can actually hook in something like Gemini Life API, uh, into the glasses.

  97. 7:56

    Well, the glasses connect to your phone, and then the phone actually connects to Gemini Life, which is cool.

  98. 8:01

    An experiment, but, uh, it shows what will be possible.

  99. 8:04

    Yeah, nice.

  100. 8:05

    Any problems creating API keys? Everyone has an API key?

  101. 8:11

    Should we check? [laughing] Okay.

  102. 8:16

    Anyone else? Anything specific that... Yeah?

  103. 8:19

    Uh, Michael. I work for GetYourGuide in Zurich. Um, we're building, like, our first kind of AI customer support agent. So-

  104. 8:26

    Cool

  105. 8:26

    ... just interested, I think, in the general how you guys are doing it, comparing.

  106. 8:30

    Okay, cool. Nice. Last chance. Anyone else? Okay. All right.

  107. 8:39

    So we can start. Before we go into the hands-on session, we... I have, like, ten, 15 minutes slides, um, to give a bit of a background what we are going to use to build.

  108. 8:50

    The first session, which I'm going to do, is more on, like, the building an agent without any live au-audio input. That's where Thor is going to take over later, and then we are going to building some very nice conversational agents.

  109. 9:03

    So, um, who of you has used the Gemini API to make an API call to Gemini before? Good. That's a few. Have any one of you used the Interactions API?

  110. 9:15

    One, okay. Two, okay. At least some person. So the Interactions API is a new API we launched in December in beta, which hopefully will succeed generateContent soon. It's a unified API to use with models, uh, with agents, and is much more aligned with, I would say, the industry.

  111. 9:31

    So much closer to what you are familiar with from open models, with the Chat Completions API, or from Anthropic, or from OpenAI. Um, and what we are going to do, the slides, then we build a coding agent, like a small little Claude code with reading, writing, uh, reading files, writing files, running bash commands.

  112. 9:49

    And then we have some time at the end if you have questions around it. And then we do a short break, toilet, drinks, and then Thor will continue. We all have an API key.

  113. 9:57

    So as I said, we want to build an API which works for both models and agents. And when we launched the Interactions API, we also launched Deep Research. So maybe you have used Deep Research in ChatGPT or in the Gemini app, where you basically start off a query, and then you get back a plan.

  114. 10:12

    And then the model goes on and, like, does deep research for ten, 15 minutes, visiting hundreds of sites. And, um, the API, um, supports both models and agents, and it's very simple to switch between those.

  115. 10:24

    You basically either define a model, which could be Gemini 3 Flash, or you define your agent, which could be Deep Research. And we are working on it to, for you to bring your own agent or you to define your own agent that you can customize all of these behaviors.

  116. 10:36

    And it's the same surface. So we'll see later, but when you send a request to NanoBanana to generate an image, you can, like, basically chain those interactions to a Flash model to do something else, to Luria to have you, like, generate, uh, audio and, uh, or even to hopefully soon, um, Veo to generate you, uh, video.

  117. 10:56

    And the interface is very similar to what you see from OpenAI. So we now have, like, those, uh, content blocks, basically. So every input you provide and output you provide is the same type.

  118. 11:07

    It has a type field, which could be a function call, a thought signature, a text, audio, video, image, which hopefully makes it for you easier to build with the Gemini API.

  119. 11:16

    And it is less, I would say, Google branded, less proto specific, less gRPC to make it easier for developers to build.

  120. 11:24

    Um, the core primitives of the Interactions API, in addition to making it easier, we also introduced state on the server, which will be, which we will use for building our agents, so we don't need to manage our loop and always send back the whole history.

  121. 11:37

    Very similar to, uh, Responses API. You now have a previousInteractionId you can provide, which basically attaches to the existing history, so you can, like, just send a new input.

  122. 11:47

    Um, as mentioned, we have an agent with Deep Research and background true, so you can start your research, can poll it, or, um, soon use webhooks to get notified when your research is done, so you don't need to keep the connection open.

  123. 12:00

    We have the type blocks, and then also we have the same, like, streaming pattern, very typical for web development using SSE, which also makes it hopefully easier for you to build.

  124. 12:09

    We all support the built-in tools, but we also now have support for remote MCP. And I think two weeks ago we launched tool combination, so you can now combine Google search with your own custom function, which was one of the big features people were asking for years, I think, and now you can do this.

  125. 12:24

    Um, and to summarize again the difference between... So generateContent is what we have today. Interactions API is what we will have tomorrow. We will have, um, server-side state management, but you don't need to use it.

  126. 12:36

    So if you say, "Hey, I want to manage my turns, I want to manage my context. I don't trust you," or, "I need to do, like, context engineering. I will need to remove certain parts," you can do this.

  127. 12:46

    Um, you send... Can... It's way easier to send new input, basically. We have the built-in agents. We have also background support, so you will get asynchronous execution. We all see with agents when you send the prompt, it might take one, two, three, four minutes to complete, and keeping HTTP requests or connections open for, I would say, more

  128. 13:06

    than, like, 10 seconds is not a very good practice. So you want to, like, use asynchronous calls to either get notified or, like, do polling, uh, when it is done.

  129. 13:15

    And it is less proto-oriented, which I really prefer. Um, it's much closer to what people know from, like, the, the developer ecosystem. And also a side effect of the state management is that the implicit caching for the API is much better.

  130. 13:28

    So for the one of you who don't know what is implicit caching, so when you send a request to the model, the model needs to encode all of your input tokens.

  131. 13:37

    And you can cache those encodings for a follow-up request to save cost. And cached requests, I think, are 90% cheaper for the input tokens. And when you have to manage your state or context yourself, you maybe strip out, uh, line breaks, remove, um, certain parts.

  132. 13:54

    This breaks the cache. And using the server-side state, the server keeps the context, so the chances for your cache hit rate is much higher. And we see, like, two to three times better cache rates from, like, the, the startups using Interactions API today.

  133. 14:10

    Quick code example on what we mean with making it simpler. So on the left side, you have, like, the very proto specific one-off input parts with inline data or text where the, the field basically describes the type.

  134. 14:21

    And on the right side, almost looks similar to other APIs, I would say. So should be much easier if you decide after the workshop to give it a try.

  135. 14:31

    Um, then I guess all of you know roughly what an agent is. We have a brain or our model which decides what it wants to do, calling a tool, generating text, doing something else.

  136. 14:42

    We have tools which basically gives our brain hands and eyes to interact with the environment where it is in. We have the context. It's basically all of the... All the model knows, uh, what it has to do, what it can do, if there are certain preferences, if there are certain constraints.

  137. 14:57

    And then we have the loop, which basically combines our model with hands and tools and runs it until the model no longer calls the tools and generates a text.

  138. 15:06

    So some quick examples on how to use the API, and then we go into, like, the, the nice hands-on part. So basic chat usage with server-side state becomes very easy because we have our interactions.create call.

  139. 15:20

    So we define our model, we define our input, what's the capital of France. We get back the output, and then we can just continue providing the previous ID. And the model behind the scenes or, like, the server behind the scenes basically has our user input, our model output, and then appends the new user input that you don't

  140. 15:37

    need to have, like, a client-side history object where you append the user turns, the model turns, and then the user turn again. This becomes very helpful when you build agents where you have the loop and always need to append new user input.

  141. 15:52

    And, um, as mentioned before, that also works for agents and models. So we really want to build this unified interface where you can continue your conversations no matter what model you use.

  142. 16:05

    And in this example, we basically run a deep research request on research AI agents in 2026, and then we take the research output and just continue with the NanoBanana model to generate a visual for it.

  143. 16:19

    And it's, like, four lines of code basically without for you to, what's the context, how do I provide the input, and hopefully makes it a lot easier. But as said, you don't have to do it.

  144. 16:30

    So, like, the input field also accepts the same array with role user, role model, and all of the inputs.

  145. 16:38

    Um, for tool use, it's also hopefully now much easier. We have a type function, our name for our function which we want to use, describe, and then we have the parameters the model needs to generate.

  146. 16:51

    And then here's roughly what we are going to build. So we make our API call with the tools, and then we check what the, the output of the interaction is.

  147. 17:02

    Requires action basically means you as a client, you need to do something. So the output generated some function call or some object which you need to react to. We iterate over our, um, output types, check for the function call, execute the function, append the result, send a new interaction until the model no longer decides it wants to

  148. 17:23

    call a function and generates a text or something else. Okay.

  149. 17:29

    Um, the last time I did a workshop, we were all still coding manually. So that's my first time doing a hands-on workshop where at least I don't code much manually anymore.

  150. 17:43

    I'm not sure how you do it. So what we are going to do today is we don't code manually. We are going to use your preferred IDE agent CLI of choice.

  151. 17:54

    I'm not sure how many of you are following, but, um, to make it easier, we created agent skills for our agents to use. So if you go to the Gemini, you can search for Gemini API docs coding agents.

  152. 18:11

    Can you zoom in it?

  153. 18:12

    Ah, yes. Sorry, of course. Um I can also... Can you see it?

  154. 18:20

    Yeah, I think you can also just Google the- Yeah ... you can also Google, uh- I mean, I can- Gemini coding agent skills. Gemini

  155. 18:30

    agent skills. Okay. Um, yes, this one it should be. The set up your coding agent with Gemini MCP and skills. I think that should bring us here.

  156. 18:48

    And the- Very different results Really? Yeah. [laughs] Yes. Or if not, you can go to the documentation, and then on the left side in the Getting Started section, there's a coding agent setup.

  157. 19:07

    Are you-- Are we successful? Okay. Um.

  158. 19:13

    So one question.

  159. 19:13

    Yeah.

  160. 19:14

    Should we install this skill globally or just for the project?

  161. 19:16

    You can-- You don't need to install it globally.

  162. 19:19

    Okay.

  163. 19:19

    And that's a good question. So how do you install? So we have multiple skills. So we have the Gemini API dev skill. We are going to use this one.

  164. 19:26

    That's for the generateContent API. If you want to use that API or if you are familiar with it, go for it. But we have the, the Gemini Live API, and that's what Thor is going to use later.

  165. 19:36

    So we also don't install this. But then we have the Gemini Interactions API. And here you can either pick the, the first command or the second command, depending on what you want, and then, like, just copy it, uh, open your workspace, um, where you're working in.

  166. 19:53

    And then I already did it, but you can, like, add the command npx install, and then you should get a, a wizard asking you to install it. There are many preselected.

  167. 20:06

    Don't get confused by it. It just means that all of those agents are, are compatible with the .agents folder. So you might have a .agents/skills/gemini-interactions-api, and in there we have our skill.

  168. 20:23

    Yeah. I-- Let me also, uh, wait. So I tested with Cursor and in Antigravity and also Gemini CLI. So if you are using one of those three, you are good.

  169. 20:33

    If you are using Claude Code, I think it should work as well. I'm not sure if they follow .agents best practice. If not, I mean, I'm sure you can- They invent- invented it, right?

  170. 20:43

    Mm-hmm. Like the skills and then the- Yeah, the skills, but I think they only look at .claude. Uh, I think... Yeah, they don't look at agents. They don't look at agents.

  171. 20:52

    But I think it installs into skills, no? I don't... Okay. So [coughs] - It should, it should work. Yeah, if you install npx skill with Claude, it doesn't really- Okay.

  172. 21:01

    Yeah. Question? Uh, you showed two different commands. One said skills.sh and the other one Context7 or- Yeah. They both, both work the same. What-- But what's the-- what is Context7 mean?

  173. 21:10

    Uh, so Context7 is, I think, a product based out of Upstash. So Context7 is-- has an MCP server in our skills CLI, which you can use to get access to, like, skills.

  174. 21:22

    It's like a public repository. So- Okay ... and skills.sh is the same from the Vercel team. Okay. So it's like-- And it, it works-- both works on GitHub. So the google-gemini-gemini-skills is a GitHub repository, and the GitHub repository includes all of the skills we, we documented.

  175. 21:45

    So you can also go there and find it there. There are also the deals and install commands, and it makes it much easier than cloning it and making sure you have it in the right directory.

  176. 21:56

    And then what we can do is, um, to make sure our skill works, we can just, like, ask our agent, uh, "What skills can you

  177. 22:07

    use?" And then it should-- If it-- If you have installed it correctly, you should see, yes, we have our specialized interactions API skill. So Antigravity here picked up our skill in the .agents/skill folder.

  178. 22:24

    Okay. How are we doing? Can you zoom in just a bit? Zoom in more? One more. Yeah. I think just in the back. It's- Okay. Yeah. This, this should be.

  179. 22:34

    Better? Yeah.

  180. 22:35

    Um, I'm-- I was using Telegram with my agent.

  181. 22:38

    Mm-hmm.

  182. 22:38

    Is the result of this being run on the system, or would I have to have it run on-

  183. 22:43

    No, you can-- Like, you should be able to do it on, like, your OpenClau kind of stuff if you tell it install the interactions API and then-

  184. 22:49

    Just the final product of, of the workshop. Will it run?

  185. 22:52

    Yeah, it should run. It's a Python script we are going to execute later.

  186. 22:55

    So because I'm only running in a sandbox.

  187. 22:57

    Yeah. No, that should work. Okay. Any, any difficulties installing the skill? Any questions what a skill is? Everyone familiar with skills? Okay. What we can do maybe, uh, to give you some, uh, insights on what skill we created or what it contains, right?

  188. 23:17

    The importance when creating skills is it should be either something the model cannot do reliably, or if you have some personal preferences on, like, how to do a certain workflow, or I don't know, you always need to run tests using BAN or something like this.

  189. 23:32

    And what we did with our skill is we made sure that the agent is aware of which Gemini models are available. A common issue we saw before that is, like, Gemini always use Gemini 1.5, which is no longer the latest model.

  190. 23:45

    Um, we also included the agents here. Uh, we have some, like, very high-level information on how it works, but we did not include, like, all of the documentation. What we did instead is you should see, yes, a, a link to our documentation, which is available as Markdown.

  191. 24:04

    So instead of the need to always update our skill with, like, for example, we added a new feature to Interactions API to combine tools, we would have needed to update our skill, and then every one of you also need to update your skill to be able to use it.

  192. 24:18

    And then we are not making a lot of progress in terms of, like, knowledge cutoff. Instead, we provide the, the information as part of the skill. So all of the agents now have, like, web fetch tools, so they can query the information based on the skill.

  193. 24:31

    And then, like, we only need to maintain, like, the documentation, which is mostly up to date. Yeah.

  194. 24:36

    How do you find that works with, like, efficiencies? Sorry. How do you find that works with, like, tool calling efficiency and all that sort of thing? Do you find it, like, it having to go and fetch the page to then I mean, it- Add it ...

  195. 24:45

    it normally you would provide it as a reference on your local file, right? Yeah. So it either needs to do a read file call or a web file ch- call, which is the same, I would say, in terms of, like, cost.

  196. 24:56

    And it works very well. Okay. So what we are going to do as a first example, since we are not wipe coding, uh, we want to build something more substantial.

  197. 25:09

    We not just say, "Build an agent." We want to be more specific, so we want to build, um, create

  198. 25:17

    an agent class with a constructor and a run method.

  199. 25:28

    The constructor creates a GenAI client. So the GenAI client is what we are going to use to call our model. We also need, uh, defines our model.

  200. 25:46

    And we also need a global previous interaction

  201. 25:58

    ID. And then add the main method to run an example.

  202. 26:06

    Uh, do all of that in, uh, workshop.py. Okay.

  203. 26:21

    So then as we can see, the model in this case, or the agent as a first step, read our skill,

  204. 26:28

    analyzed our main file. It's implementing the skill. Okay. It checks. It should probably fail. Yes, because I'm using uv, so I stop, and I tell it, "Use uv, uh, from the

  205. 26:42

    workspace env." And then we let it generate uv. Okay. It checks if we have installed the library.

  206. 26:56

    We have. So maybe in your case, the agent still tries to install Google GenAI. If not, you can do it yourself with, like, pip install or uv pip install, um, google-genai.

  207. 27:09

    And then- Just for this ... make sure to... Okay. So

  208. 27:15

    we have our starting agent class. We have our GenAI client. We have our model ID. It defaults to Gemini Free Flash. It would never have defaulted to Gemini Free Flash if it would, uh, it had hadn't read the skill, right?

  209. 27:30

    Because then we were stuck with Gemini 1.5. We have our run method, which calls, uh, makes our interactions.create, uh, call. We have the input text. We have our, um, previous interaction ID.

  210. 27:43

    We set our new previous interaction ID, and then we return the text. And then nice as a main example, um, which we will run in a bit. We create our agent.

  211. 27:53

    We have turn one, so my name is Phil. And then the agent, uh, runs it. And then what is my name? To... Now we can check if w- our

  212. 28:02

    Gemini created Gemini agent, uh, works with, uh, multi-turn in our interactions API. You might see a warning similar to the one I got here. Uh, interaction usage is experimental.

  213. 28:15

    That's... As we are still in beta, we really work hard to get the API out of beta to make sure you can use it in production. And our call was successful.

  214. 28:25

    "Hi, Phil. Nice to meet you." Okay, maybe we weren't successful. I don't actually have a name. I s- "What is my name?" Hmm.

  215. 28:34

    Maybe we should check. Did we do it correctly?

  216. 28:39

    I mean, my name is maybe... Ah, okay, that's still the call. Sorry. I asked it what's your name. I mean, it's a language model for trained by Google. It makes sense.

  217. 28:50

    And what is my name? "Your name is Philipp. How can I help you?" So yeah. You are using Gemini 3 Flash as a coding model? Yes. Respect. I mean, it, it, it works really well if you are providing good instructions with good skills and good context, and don't expect it to, I don't know, like, cure cancer.

  218. 29:10

    Like, if you have a very good understanding of what you are trying to build, Gemini Free Flash is, like, very fast. I mean, it didn't take much long. It didn't consume credits, right?

  219. 29:19

    Every one of us is somewhat token constraints at the moment. And it, it works really well. Okay. Use it for agentic use, yes. But for planning and coding, I'm not- Yeah.

  220. 29:28

    No, it, it works. I mean, we will going to use free w- I mean, like, it also asks, like, if it wants to run. So we are closing the loop.

  221. 29:34

    Even, like, with Gemini Free Flash, it tries to run our script to make sure it works. And then we will continue in a bit. How are we doing? Anyone making successful calls?

  222. 29:46

    Yes? Yeah. Okay. Perfect. Back around. Okay. Any, any issues, any errors, any questions?

  223. 29:54

    Yeah. I'm using TypeScript. Okay. It, it works? Yeah. Nice. Great. Works for me too. Mm-hmm. Awesome. Even coding on a phone, so... Okay. So normally the next step for our agent, right, we now have our, like, very basic run.

  224. 30:11

    We can chat with the model. Now we need to add tools to it. And we want to build some kind of a coding agent. So, uh, first tools are, we are going to add is a read and write file tool.

  225. 30:21

    And we just continue in our, um, main agent thread as like, okay,

  226. 30:27

    uh, next we need to add a, a read file and a write file tool. Create the,

  227. 30:39

    create a basic Python implementation. And also the

  228. 30:49

    JSON schema definition. So when you use function calling or tool use, right, we need to create a JSON schema which we provide to the model so the model understands what it needs to generate.

  229. 31:01

    Once it generates that schema, we also need to have some kind of a code implementation which we can then run on the client. So we ask it to create a Python implementation and also the JSON schema, uh, and, uh, map to...

  230. 31:17

    for the key and key function comma schema.

  231. 31:26

    Okay. Let's see what it will come up with.

  232. 31:31

    Okay. It's cheating a little bit.

  233. 31:51

    So I have like a, a solution folder, and it looked up the- [laughs] ... implementation, implemented. Yeah. I mean, that's why it's still important to, like, check your way to...

  234. 32:00

    I can like... It, it, it found, like, the solution, and then it was like, "Oh, I got this solid Python example and solution.agent to guide me. The task is implement readFile and writeFile." [laughs]

  235. 32:14

    Smart model for sure.

  236. 32:15

    Yes. Uh, so what we got back is we have two new, like, very basic, very, very basic file implementations. So we have a readFile tool with a file path which uses, um, Python syntax to open it, to read it, and we have a writeFile tool with file path and the content and writes it.

  237. 32:33

    And then we have our readFileSchema, um, reads a file and returns the content. writeFile writes the file and returns the content. And it made some updates to our

  238. 32:43

    agent. So what did it change? Okay, our input is now a text, uh, string and a list. Makes sense since we now need to return a function call. We check.

  239. 32:55

    What do we do? Okay, we create a tool definition for our model, which is the

  240. 33:01

    t schema. So that's our tools map schema.

  241. 33:07

    And then we have our loop, which you might be familiar with from the slide I showed. So after we run our request, we check the interactions.outputs. So the interactions.outputs include all of the events generated by the model.

  242. 33:23

    And since Gemini is a reasoning model, it also includes, for example, the thoughts and the thought signatures, which we need to return since we are using the previous interaction ID and the server-side state.

  243. 33:34

    That's done by us, and we only need to check, okay, do we have a function call? We have a nice debug, so we can check it. And for our outputs or our function call, we check our tools.

  244. 33:45

    Do we have our tool or not? I mean, maybe the model wants to edit the file, but we don't have edit a file tool. We would catch it here.

  245. 33:53

    It calls our model, and then it creates the tool results. So for function call, we have a function result. That's also part of the change. We really want to make it easy.

  246. 34:04

    Um, we will see later when we use Google Search, there will be a Google Search call and a Google Search result to have very the, the same schema. And then we use recursion.

  247. 34:13

    So if we have tool results, we basically call ourself the self run method again. If we don't have tool results, we return the interaction text. And it also updated our example.

  248. 34:26

    Write Hello from the agent to a file named hello text and return it back. So we-- I mean, the changes roughly look good to me. We can try and run it.

  249. 34:38

    So you will run Python workshop main. What do we get? We get our tool call with writeFile. We get our tool result. We get a readFile call again, and then the final agent response.

  250. 34:54

    The file hello agent text was successfully created with the content. Okay.

  251. 35:01

    Agents are not only like singleton, right? Maybe we wanted to respond. So next step is basically very ambitious, telling it, "I want to have a

  252. 35:13

    continuous," I don't know, "stood in implementation to test."

  253. 35:23

    Let's see it. Let's see where it will continue.

  254. 35:32

    Um, I hope at least it should update our main function where we have an input, a while loop, basically waiting for the input using our agent. So yep. So we have our user input, uh, and then we have-- we always continue with our agent.run method, which then inside the agent runs it in a loops until there are

  255. 35:52

    no tool calls anymore. And if there's a result, we basically get back our response. Okay. Let's quickly wait. Is it too fast, or are we still on track roughly?

  256. 36:05

    Yeah. It's fast.

  257. 36:06

    I think it's fast.

  258. 36:06

    It's fast?

  259. 36:06

    Yeah.

  260. 36:06

    Okay. I mean, we will share the code later. Um, also very happy to answer questions. Uh, we have many, like, blog posts and examples of that online, so you can...

  261. 36:16

    if you are interested in, like, rebuilding it later with less fast. Um, but I'll-- I try to be a little bit slower, but I also want to give Thor enough time to have you speak.

  262. 36:26

    So what we can do now since we or, like, our agent implemented our, like, while loop to, um,

  263. 36:34

    to, to provide input, we can say something like hello, right? Normally, we should now not do any tool calls because hello is like, yeah, our agent correctly kinda understood, or Gemini in this case understood, "Hey, hello is nothing I need to solve with a read or write file tool," right?

  264. 36:50

    So I can say, "Hello. How can I help you? Can you write- Or maybe, can you create a CSV with a thumbs up?

  265. 37:07

    Might take a while a bit. So it does the thinking, and it does the function call. And whoop.

  266. 37:15

    Okay. Um, certainly, here's a CSV file, blah, blah, blah. Um, okay. Can you write it to disk?

  267. 37:31

    Uh, yes I can if you allow... Wait. Maybe

  268. 37:36

    our model is not having our tools. Let's check.

  269. 37:40

    There's the tools map. It does. Why is it not using our tools? What tools can you use?

  270. 37:50

    Okay, it has a read file. Yes. Maybe we are, we're not explicit enough.

  271. 38:07

    And what we can do to improve this in one second is we can add a system instruction to tell the model, "Hey, you are a coding agent. You can use tools to write and interact with the..."

  272. 38:20

    Okay. There we got our tool call write file with our CSV. Cool. Since we saw a mistake, we can now tell the model, "Hey,

  273. 38:29

    add system instructions for the interactions API call, and add an example

  274. 38:41

    prompt for a coding agent." Okay. And what's really nice now, since we, uh, loaded the interactions API skill in the beginning, the model still has the awareness of, okay, how do I add system instructions to the interactions API?

  275. 39:03

    And what I can guarantee you is that, uh, Gemini 3.0 Flash has never seen any code of the interactions API, because the model was trained be- before we even released the API.

  276. 39:13

    So all of the work we were doing so far is based on, like, the skills and, like, the decoding infrastructure and hasn't been part of the, the training. Okay.

  277. 39:23

    So what did we get? Um, okay. We can provide it on the run command, or we have one when we create it. That's good. And okay. Coding persona. You are an expert software engineer and helpful coding assistant.

  278. 39:38

    You have access to the local file system. Okay. Let's accept it. Let's start our agent again. Let's say hello.

  279. 40:00

    Okay. Hello, how can I help you with your software engineering and coding tasks? I mean, definitely better than what we had so far. And what did we send before?

  280. 40:08

    We said, "Can you create a SVG with a thumbs up?" So can you create an SVG with a thumbs up? And now let's see if it calls. Yes. It...

  281. 40:21

    Now at this time, we got our write file tool call, and then also we got a, hey, I have created a thumbs up SVG file with a simple line art thumbs icon.

  282. 40:31

    So, um, can I...

  283. 40:37

    I think you should.

  284. 40:38

    Yes.

  285. 40:38

    Yeah.

  286. 40:38

    There we go. We have got a thumbs up icon. Cool.

  287. 40:43

    Um, and of course what's missing for a coding agent, right, we need to get some bash tools. That's now not part of the solutions folder. So let's see how we will get our bash tool.

  288. 40:55

    Now add a similar run command tool that allows the model to execute

  289. 41:07

    bash commands. Okay. Creates an implementation plan. [coughs]

  290. 41:21

    It does hold its weight. You're asking to

  291. 41:25

    do something.

  292. 41:27

    Oh, Philipp, I think there's a question. Yep. Can you-

  293. 41:29

    Yeah, I'd say, like I was just testing it, it does hold its weight tools a little bit. Like-

  294. 41:33

    It does

  295. 41:33

    ... if you try to do like with character and then-

  296. 41:35

    Yeah

  297. 41:35

    ... even with the

  298. 41:36

    Okay. So we have our run command, which uses a subprocess, which in this case I guess is okay. We don't care too much about security for this example. And then our output is a std out.

  299. 41:51

    Um, works. Edit. We have our run command tool. Yeah. [coughs] It even updated our system prompt. And now let's stop that. Let's clear that.

  300. 42:06

    Let's run it again. Um, any suggestion on what we should test?

  301. 42:15

    Use nana banana to create an image.

  302. 42:17

    Uh, I'm not sure that will work because we don't have any skills or any information for it.

  303. 42:22

    What's time?

  304. 42:23

    It's time. Time.

  305. 42:24

    Time. Yeah, I guess so.

  306. 42:25

    Oh.

  307. 42:26

    From bash.

  308. 42:28

    Get the time. Tool call, run command, date, Wednesday, April 8.

  309. 42:36

    It looks good.

  310. 42:38

    It works.

  311. 42:38

    Um, yeah. Cool. That's our small little coding agent. Delete all files. No, I mean, let's not do this. [laughs]

  312. 42:49

    Try. Try. Any more questions? Any more ideas? We have, like, roughly five to seven minutes. Yes.

  313. 42:55

    General question. So because the state is kept in the cyber-

  314. 42:58

    Yeah

  315. 42:58

    ... is, is it possible to, like, fork the history of-

  316. 43:01

    Yes

  317. 43:01

    ... different parts?

  318. 43:02

    Yes. So- Um, what you always can do... So what we are doing here is, right, we always use the previous interaction ID from the previous term. So we basically, uh, stack it, and you can always go back to any index in the stack and branch from there.

  319. 43:23

    So if you would keep the interaction IDs on your client side, you can always use those to, I don't know, branch out and, like, I don't know, have, like, a, a first, um, prompt on, like, do basic web search and then, like, use this as, like, a base for, like, five parallel requests doing some other work.

  320. 43:41

    And you can always get the context. So we have an interactions.get method

  321. 43:48

    which you can use to retrieve the interaction and then also get the previous interaction ID. So you can basically go back until the beginning and get all of your state if you want to save it for later.

  322. 44:00

    And the default for those interactions being stored on the server for free tier is one day. For paid usage is fifty-five days at the moment.

  323. 44:09

    Thanks.

  324. 44:09

    Yeah, you had a question.

  325. 44:10

    You answered my question. [laughs]

  326. 44:11

    Sorry?

  327. 44:12

    You answered my question. [laughs]

  328. 44:12

    Okay. [laughs] Perfect. More questions. Yeah?

  329. 44:14

    Does it mean then it can have infinitely long context window?

  330. 44:19

    Uh, no. So once... So Gemini models have a million, uh, token context. What would happen now if you reach that, you will get an error. But we are working on context compaction techniques, but it's easier said than done, and, um, still something you currently need to maintain on, on your own client side.

  331. 44:38

    So when you say that it will retain, the server will retain for one day, so that means every day it resets to zero and then the-

  332. 44:44

    No, no. So when you send a request, you get an ID, the interaction ID. Uh, and the interaction ID stores your input and the output of the model. In the free tier, the input and output and the ID is stored for one day.

  333. 44:58

    So meaning if you send a request now and you continue eight hours later from that point, the state or, like, the context is still available. If you would send a request tomorrow, it would basically say cannot find, um, request with the old interaction ID because it basically is pruned after a day.

  334. 45:15

    But if you use a paid API key, it's stored for fifty-five days. And, um, the interactions API is also coming to Vertex, and I think there might be a little bit more flexibility in terms of, um, how long you want to store or customize it.

  335. 45:30

    Yes?

  336. 45:31

    Is there a specific API to rework the context class if you wanna do complex context or cache them on the client side?

  337. 45:37

    We don't have one yet, but hopefully soon. Yes?

  338. 45:43

    Maybe an idea, you could have the API joke being spoken out with the eSpeak tool.

  339. 45:48

    Sorry?

  340. 45:49

    The eSpeak tool. It could use the eSpeak tool to actually speak the joke about APIs mentioned in the chat.

  341. 45:54

    Ah. And we can also use Gemini, which has a TTS model, which can speak to... But speaking and listening, I mean, tool will show many, many cool things. Any questions regarding the interactions API and the small little agent?

  342. 46:09

    No? Okay. Then you get eight minutes.

  343. 46:14

    And so five minutes.

  344. 46:15

    Okay. Five minutes.

  345. 46:16

    Five minutes bio break, and then we'll be back here-

  346. 46:20

    With-

  347. 46:20

    To make your agent talk.

  348. 46:22

    Yes. Cool.

  349. 46:24

    Cool. Thanks. [clapping]

  350. 46:32

    That worked really well. [laughs]

  351. 46:33

    Nice. Yeah.

  352. 46:39

    I used the old one and super happy that now there's a new-

  353. 46:42

    Mm-hmm

  354. 46:42

    ... one version of it. It was two point five before.

  355. 46:45

    Yes. Yeah. It's, it's been a while.

  356. 46:47

    It's been a long time.

  357. 46:50

    How much better is two point one?

  358. 46:51

    How much better?

  359. 46:51

    Yeah.

  360. 46:51

    Much better.

  361. 46:52

    Much... [laughs] We didn't even pay him for it.

  362. 46:55

    So-

  363. 46:55

    There you go

  364. 46:56

    ... big upgrade.

  365. 46:57

    Big upgrades, yes. Um, yeah, maybe, uh, I know there's a couple more minutes, but-

  366. 47:06

    Can I, can I ask a question while we wait about caching?

  367. 47:09

    Caching?

  368. 47:10

    Yeah.

  369. 47:10

    Yes.

  370. 47:11

    The input tokens.

  371. 47:12

    Yeah.

  372. 47:12

    How is that, um, are those cached, um, in the, uh, context array element level or the whole context is cached? So like when adding, uh, the elements to the interaction, is each one of those individually cached or-

  373. 47:26

    That's a good question. We probably need to find Philipp to answer that. I actually, I actually don't know. Philipp, caching question. Yep. Uh, the input tokens. Yep. What was it?

  374. 47:40

    Um, is the context, uh, cached at the individual interaction level or is the whole context, uh-

  375. 47:46

    It is not on an interaction level. It's more on an, like, object level. So it's, it's- For example, when you provide an input, like, no interaction ID, first input PDF, four thousand tokens, and the text input with ten tokens, and you do an follow-up interaction call, maybe only the PDF will be cached and not the other, like

  376. 48:07

    the short text. And then if you do another one, maybe the PDF and, like, the follow-up turns will be cached. So it's, like, more on, like, an object level.

  377. 48:15

    But since you... I mean, how was it? It's very easy to make a mistake in caching if you, like, even the slightest change in your prompt, removing white space, line breaks will break it.

  378. 48:29

    So, like, having this rely on the server to keep it, it's more guaranteed that it's secure, and it could be as easy as, hey, my user says there's an empty line break at the end.

  379. 48:40

    Okay, I remove it, and then I use that history again, and then it falls apart.

  380. 48:44

    To do it right, ideally every conversation, the previous part of the conversation is cached.

  381. 48:49

    Yes. Uh, optimally. But I mean, it depends on where your request kind of hits it and, like, how fast you follow up, but the cache rate should be pretty high.

  382. 49:02

    Cool. Maybe, uh, to kick things off, we can sort of look at one of the examples that, that we can build, um- With the, the Live API, the new model.

  383. 49:14

    So this is now, uh, Gemini 3.1 Flash Live, which came out two weeks ago, I think. Um, very exciting. It's been a while. I think the previous 2.5 native audio model was December, so it's been a while.

  384. 49:31

    The reason being, we did kind of major underlying architecture rework to, you know, ideally

  385. 49:39

    lower latency, you know, better scalability in the, in the future as this ramps up. So, uh, that's very exciting. And then, you know, in parallel, we're working on bringing, uh,

  386. 49:49

    the Live API to the Interactions API as well. Um, so that's something that Philipp and I are working on at the moment. Not there yet, so we still have to use the, uh, Live API for now.

  387. 50:00

    But let's do a little demo. And so this demo uses the Live API in combination with Lyria 3. Uh, so on the Gemini API... Have, have people played with Lyria?

  388. 50:13

    So the new Lyria model is actually a music generation model, and so Lyria 3 can now generate songs. Uh, you know, including songs with lyrics. Uh, so this is the clip model, which is kind of a 30-second, uh, clip generation.

  389. 50:28

    Uh, but then also we have a full song model now, where you can generate kind of full, full songs. So the idea is that we have our conversational agent, which is kind of the DJ that you're interacting with, and then that DJ has a tool called To Generate Music, right?

  390. 50:44

    Um, and so you know, I... So I'm originally [REDACTED:origin], but, you know, growing up I learned kinda English through Harry Potter and BB- BBC Radio 1. And if you remember back in the day, you could like phone in and like wish for a song, and so that's sort of the, the, the idea of the experience here.

  391. 51:04

    Right then, you're on Live Jukebox. What absolute banger are we cooking up today? Tell me, what genre are we leaning into, and what sort of vibe are you going for?

  392. 51:14

    Let's make-

  393. 51:14

    Can, can we get some sort of high energy [REDACTED:origin], um, techno schlager- [laughs] ... about the AI scene in the UK?

  394. 51:26

    [REDACTED:origin] techno schlager about UK AI? That's absolutely brilliant, mate. Proper Radio 1 style. Any specific lyrics or themes you want crammed in there? Or sh-

  395. 51:36

    Nah, just surprise me. Let's go.

  396. 51:41

    Right, you beauty. Let's get those speakers rattling. Prepare for a burst of machine mayhem. Here comes [REDACTED:origin] techno schlager about the UK AI scene. Let's go. [techno music]

  397. 51:53

    In London town, the future's bright and keen. The cleverest robots that you've ever seen. The computers wear a proper bowler hat, and the androids all enjoy a lovely chat.

  398. 52:07

    They're drinking builder's tea with digital grace. A happy British smile on every metal face. Oh, wunderbar, the AI is super gorgeous. Ja, super gorgeous. [techno music] [laughs] [claps]

  399. 52:25

    Yeah. Um- [claps] The, the TPUs enjoy when you clap for them. [laughs] So, um, appreciate that. Uh, any- anyone have any idea for another one? We c- we can do one more.

  400. 52:39

    Just shout it out. Any languages? Any genres? Can we have a nursing,

  401. 52:48

    uh, uh, song in Swahili? A nur- nursing song? Song, yeah. In- Swahili ... Swahili? Okay.

  402. 52:58

    Right then, welcome to the Live Jukebox. You're on air with me.

  403. 53:01

    Can we get a nursing song in Swahili, please?

  404. 53:07

    A nursing song in Swahili. Brilliant. That's something we don't hear every day. [laughs] To make it an absolute banger, what sort of vibe are we going for? Upbeat and celebratory or something more relaxing and gentle?

  405. 53:19

    And what specific bits about nursing should we squeeze in? Let me know, and I'll get this track buzzing.

  406. 53:25

    Yeah, tech- techno hardcore- [laughs] ... about... Oh. [laughs]

  407. 53:35

    Techno hardcore in Swahili about nursing. Wicked. Prepare to blast those speakers. I'm dialing that in now. Get ready for a total banger, mate. [techno music] [singing]

  408. 53:49

    Okay. The vibe is wrong. That's true. [singing] I never know.

  409. 54:11

    Well, we, we can feed it into Gemini to, uh, you know, guess the audio out. Yeah. Okay. [laughs]

  410. 54:19

    That is the Live Jukebox DJ, uh, DJ. You, uh, if you want to, it's, it's on... So this was built fully with Google AI Studio. Uh, so it's, it's kind of a little vibe codus, um, demo.

  411. 54:31

    I've published it in AI Studio as well, so you can try it out. Uh, you will need a paid API key for this one, because the music generation actually requires a billing account.

  412. 54:41

    Um, so if you have, you know, an account with credits, uh, you can use that, uh, for example.

  413. 54:49

    Uh, cool. Yeah, Gemini, uh, 3.1 Flash Live. You know, benchmarks, obviously benchmarks don't really tell you the truth, uh, as much. They are great for benchmarking things. Uh, the real world, especially in kinda live audio, does often look a bit different.

  414. 55:08

    So, um, you know, ideally we'll just try it out ourselves. So Gemini 3.1 Flash Live, it's the model that is now in Gemini Live in your phone. So if you're using Gemini app, uh, on your phone, um, you're talking to that model.

  415. 55:23

    Uh, as well as, I think, Search Live has it now in there as well. So if you're talking to Google Search- Uh, I think that's the same model. And then you can build applications using this model on the Live API.

  416. 55:38

    So the Live API is a stateful kind of WebSocket API. Um, you are able to send real-time text, audio, video feeds, uh, to the model. So audio, you're sending in a kind of, you know, buffer chunks.

  417. 55:55

    So if off the, the real-time audio, you're streaming that in. Uh, video, you can stream in, uh, at a maximum frame rate of one frame per second. So this can be, you know, a camera feed, this can be, uh, a canvas, so, uh, it could be like your, your, your screen share, right?

  418. 56:13

    So you could share the screen with the model. So for example, uh, Shopify is using this for, uh, Shopify Sidekick, um, where it's actually kind of like a tech support walking you through.

  419. 56:24

    You know, if you're like, "Oh, how do I set up a custom domain for like my Shopify store?" It would basically like talk you through how to do that, and it can see kind of where you are on the screen by sort of ingesting, uh, the frames of the screen.

  420. 56:37

    And then in return, uh, the WebSocket gives you kind of real-time events back, and so these are basically streaming back audio buffers. Uh, and then also you can get the audio transcription, so that's kind of the text, um, of it.

  421. 56:51

    And then we have, um, tool calling built-in, so Google Search grounding is built in by default. So if you need kind of real-time weather information, you can access that as well.

  422. 57:01

    Um, yeah. Some key features. So what's really cool about this model, again, it's, it's kind of a native audio model. So what that means is we're not going through text, so it's kind of not a cascading pipeline where, um, you're transcribing the text, running the text through an LLM, and then generating speech.

  423. 57:19

    Uh, but rather the model itself, um, is, you know, going sound token to sound token, and the intelligence is kind of baked into this audio model. Um, so it's based on, you know, Gemini 3.1, so decently intelligent.

  424. 57:34

    Uh, you have different kind of thinking levels that you can enable. Um, and so the great thing with that is kind of the multilingual support. So it's, I think, 97 languages that are kind of supported in preview, uh, at the moment, which, uh...

  425. 57:48

    And the great thing is because it is kind of a, you know, a native audio model, it, it can actually, it has sort of the audio understanding of, of Gemini built into it.

  426. 57:56

    So, um, it can understand a mix of different languages. It can, you know, sort of a Denglish for example, which is like a mix of [REDACTED:origin], Deutsch and, and English, right?

  427. 58:06

    So it, it would be able to sort of n-n-naturally switch be-between kind of different languages as well, which is really great. Um, yeah, barge-in. You know, obviously there's kind of automatic, um, uh, voice activity detection sort of built into the model, so you can interrupt it.

  428. 58:22

    You saw it earlier with DJ, you know, I was kind of trying to have a conversation, but we're trying to get it to, to, you know, uh, use the tool.

  429. 58:28

    Tool use is the other big thing. Um, so major improvements in kind of tool use and instruction following here with that model. Um, and so you can build some, some really cool things with that.

  430. 58:42

    Um, so obviously we currently give, only give you, uh, a WebSocket API. So that is kind of a downside. If you used something like gPC Realtime before, you know, you'd get a direct WebRTC, um, kind of infrastructure which, which can be helpful.

  431. 58:59

    So we have partnered with, you know, a lot of, uh, sort of integration partners like Livekit, Pipecast. Uh, Software Mentioners, uh, in Poland, they, they built a great service called Fishjam.

  432. 59:10

    Uh, Vision Agents, Vox Implant. So these partners have integrated kind of the Live API directly, and then give you sort of easy WebRTC integrations if you, if you, um, you know, want that or need that kind of for your system.

  433. 59:24

    Um, yeah. Let's, let's try it out. So you can try it out yourself, uh, and it's gonna be interesting if we all try it out in this room at the same time.

  434. 59:31

    So, uh, we'll see how that works. But, uh, again, ai.studio or ai.dev and then /live. Uh, you can try out the model, and so you can, you know, ingest your webcam in here as well.

  435. 59:43

    Uh, so we g- we can give it our webcam feeds, um, allow this time. Uh, and then, you know, we can send text as well. So we could send like, uh, "How is my outfit?"

  436. 59:56

    Uh, so in this case, you know, I'm not ingesting any-

  437. 59:59

    You're wearing a green jacket over a blue T-shirt paired with a black cap. The combination looks casual and comfortable. Is there a specific occasion you have in mind?

  438. 1:00:09

    Yeah. Okay. So, um, you know, that obviously is a bit further away from our upbeat sort of British, Australish, uh, Australian, uh, uh, live DJ. So what we can do is we can, um, you kind of adjust our, our voice, so through the system instructions.

  439. 1:00:28

    Um, now in terms of the base voices, we don't have that many. There's kind of 30 different base voices.

  440. 1:00:34

    Got a project in mind?

  441. 1:00:35

    And they are, you know, fairly generic sort of... But because, you know, Gemini has very deep audio understanding, what we can do is actually we can modify the voice through prompts, through system instructions.

  442. 1:00:47

    So what we can do is actually we can give it, um, kind of a system instruction here, for example, and we can just say, "Okay, speak in a friendly Irish accent."

  443. 1:00:56

    Right? Um, and so now we have that base voice, Puck, and, uh, speak in a friendly Irish accent. [device beeping]

  444. 1:01:05

    Hey, can you hear me?

  445. 1:01:08

    Well, hello there. Loud and clear, so I can. What can I do for you on this fine day?

  446. 1:01:13

    What do you think of my outfit?

  447. 1:01:16

    Well, now, you're looking very smart, so you are. That green jacket suits you well, I must say. A grand casual look. Were you thinking of heading out somewhere?

  448. 1:01:28

    Uh, no, I'm just here at the AI engineer... Okay, anyway. So I, I realized I didn't put on... So what I wanted to show you as well is, um, this grounding with Google search So what we can do is, um, you know, obviously the, the model itself is trained to a certain knowledge cutoff date.

  449. 1:01:46

    And so, um, you know, if we need kinda real-time information like this gorgeous weather that, you know, no one would actually believe us, the model wouldn't believe us- [laughs] ...

  450. 1:01:54

    you know, if it, if it didn't have, um, Google Search grounding. But now, you know, we can, we can have Google Search grounding. We can have different thinking levels as well.

  451. 1:02:04

    So, um, no thinking is actually very minimal thinking. Um- [coughs] ... but it means, you know, the model will respond faster. It's kinda lower latency because the more thinking we have, uh, obviously the more thinking steps are kind of involved and, um, it'll, you know, increase the latency.

  452. 1:02:24

    So you can see in the benchmarks as well, there's kind of thinking high, uh, thinking low, uh, sort of different thinking levels. Um, but so now if we turn on kind of the, the Google Search here.

  453. 1:02:37

    Oh, hello there. What range of shenanigans can I help you with today? Anything at all, just give a shout.

  454. 1:02:44

    Uh, yeah. What's the weather right now in London, actually?

  455. 1:02:51

    Well, now, it looks like it's about nine degrees and mostly cloudy in London right now. [laughs] Might be a bit chilly, so you'd be wanting a jacket for sure. Anything else you need to know about the goings on over there?

  456. 1:03:03

    What, what are you talking about? Can you check again? What date is today?

  457. 1:03:08

    Ah, apologies there. I seem to have gotten ahead of myself. Today is Wednesday, April 8th, 2026. Is there anything specific you needed to know about today?

  458. 1:03:18

    Yeah. Now look up today's actual weather. [laughs]

  459. 1:03:23

    Right you are. For Wednesday, April 8th, 2026, in London- [laughs] ... you're looking at temperatures between 5 and 13 degrees with a decent chance of rain, so don't forget that umbrella.

  460. 1:03:34

    Anything else on your mind?

  461. 1:03:37

    All right. Uh, here we are. It looks like I didn't pray to the, the demo gods. Okay. [laughs] There seems to be something going on. I wonder if we messed up the UI there.

  462. 1:03:46

    Uh, it should, it should [laughs], it should work a lot better than that. Um- [coughs] ... I think the Google Search grounding for some reason isn't, isn't working. So but what we can do now is we can obviously try it out, uh, ourselves in an application.

  463. 1:04:00

    So the easiest way we could do that is, you know, like the Life Jukebox D- Life Jukebox DJ, um, we could use Google AI Studio to kind of vibe coach our integration.

  464. 1:04:13

    Um, so we have this pill here, which is called, uh, add, you know, voice conversation real-time voice with, you know, Gemini Life API. Uh, and then we could say, um, build kind of a multilingual interview assistant that allows me to, you know, train for interviews in different languages, uh, like [REDACTED:origin] and English and Spanish and, you know,

  465. 1:04:36

    what have you. Uh, and so we can now kind of fire this off. So this uses Gemini 3.0 Flash Preview. Um, it is limited to, uh, kind of JavaScript full stack, um, environments at the moment.

  466. 1:04:54

    So I think you can choose between kind of Next.js, Angular. Um, there's like XR building blocks as well, if you're building for, for kind of glasses, um, sort of web VR experiences.

  467. 1:05:06

    Um, so, you know, feel free to kind of fire one of these off, uh, right now, or you can also clone, uh, the Life Jukebox DJ that I shared with you earlier, and you can try that out.

  468. 1:05:17

    Um, it'll, it'll take a little time, uh, and you'll hear like a little chime once it is, uh, ready. So in the meantime, what we can do is, uh...

  469. 1:05:26

    So if you go to the Gemini Life API docs. Gemini Life API. Uh, there we are.

  470. 1:05:37

    So, you know, we've done this. We tried out kind of the Life API in Google AI Studio.

  471. 1:05:43

    Um, we also can use the coding agent skill. So, uh, Phil showed you this earlier. Um, we have dedicated coding agent skills also for the, the Gemini Life API, so you can install that.

  472. 1:05:56

    It'll help, you know, your coding agent, uh, integrate the Life API, um, more easily, more quickly. Uh, but then also, you know, we have good old example apps on, on GitHub, which can be very, very helpful.

  473. 1:06:10

    So what you can do is, um, you can clone these example apps. Uh, so in GitHub, you know, you can, uh, like you do in GitHub, right? [laughs] Uh, you can clone this.

  474. 1:06:25

    So we'll open kind of new terminal, and yes, feel free to follow along. Let me do that a bit bigger. Uh, we'll make a new directory. We'll call it AINGEU.

  475. 1:06:40

    Uh, I like that they call it Europe, right? But it's, uh... I mean, I guess, yeah, the UK is still- [laughs] ... part of Europe, just not the EU. Fair enough. [laughs]

  476. 1:06:51

    Um, and so we'll, we'll go in there, uh, and then we'll just do a Git clone, um, of our app. Uh, and so now we have our app in here.

  477. 1:07:02

    Um, so there's actually a couple different examples that we can use in here. Uh, so if you're using Antigravity, there's a handy agy command to, uh, open your examples in Antigravity.

  478. 1:07:16

    Uh, and so we can, you know, look at our different examples here and how we kinda need to set that up. So we have, you know, two different scenarios.

  479. 1:07:28

    So the Gemini Life GenAI Python example uses the Gemini Life API on the server. So it creates a WebSocket connection from your server to, uh, you know, Gemini Life API, and then- On your front end, you basically set up a proxy, um, to proxy the WebSocket

  480. 1:07:53

    connection to your client side, right? Because your, your browser window is kind of your client, and so that is what is capturing, um, your, your, uh, audio feed, your video feed.

  481. 1:08:05

    And so in this example, we're just using FastAPI, uh, and we're basically, uh, just setting up kind of a WebSocket here, uh, that our client can connect to. Uh, and then we're basically just receiving sort of our, um, you know, audio cues, our video cues from the client side, uh, or, you know, our text input queue as

  482. 1:08:29

    well. Uh, and so we're receiving that. We're then setting up, um, a Live session. So, um, we're using our Gemini client, so we kind of abstracted sort of all the Live API stuff into this Gemini-Live file here.

  483. 1:08:45

    Uh, and you can see, like, starting the session, we're basically setting up kind of our LiveConnect config. Maybe I close this for now, uh, so you can see that.

  484. 1:08:55

    Um, we're setting up some system instructions. So that was, you know, earlier, kind of set a helpful assistant. We also said, kind of, "Speak in a friendly Irish accent," for example.

  485. 1:09:05

    Um, you know, this is where we put our, uh, sort of system instructions, our guardrails. We can, you know, make that pretty long in terms of covering sort of what we, what we want.

  486. 1:09:16

    Uh, and then we can define kind of our tools here. Um, and then we're basically setting up our, um, you know, session, and, uh, our session sort of is, you know, the WebSocket session.

  487. 1:09:29

    Uh, and then we're just receiving our audio and video cues from the client side, uh, and proxying that through. So that is kind of one approach. That's sort of the server-to-server approach.

  488. 1:09:40

    Um, and, you know, we're just using kind of UV here. Um, so if we're setting this up for the first time, uh, we can go into... So this is our Gemini-Live, um, GenAI Python, uh, example here.

  489. 1:09:54

    So we can set up, uh, our virtual environment. We can then, um, activate our source here.

  490. 1:10:03

    Uh, we can install our dependencies. Uh, again, I might have... This is a, a fun part of, uh, uh, Google Laptop

  491. 1:10:19

    security. Come on. All right. Just look away. Don't look, don't look at that. [laughs] Um, and then, yeah, install our requirements. And so we'll need, uh, an API key.

  492. 1:10:36

    Uh, so the API key, you can see kind of here, uh, how the conf-fun-configuration is. So we basically just need our Gemini API key, uh, and we need to set up, uh, an environment variable for that.

  493. 1:10:48

    So you can see we have an example file here. Not a lot in there because it's basically just that. So we can copy our .env.example into our .env,

  494. 1:11:01

    uh, file here. And then do you remember how to get your API key? Where do you get your API key?

  495. 1:11:10

    AI.dev. Yes, AI.dev. There we are. Fantastic. Love it. Um, okay. AI.dev. Uh, so that is where you get your API key. Uh, I think I have a couple API keys, so it takes a little while to, um, load them here.

  496. 1:11:28

    Uh, which one is... Maybe we'll, we'll use this one. So once you've created your API key, uh, you can copy the A- API key from here. Well, actually, maybe I should create a new one, because later I'll need to delete it.

  497. 1:11:42

    Uh, so we'll say ai-eng-europe. Uh, we'll, we have a couple projects here. We'll just use...

  498. 1:11:51

    We'll just... Uh, which one? Which one are we using? Too many projects. Okay, we'll just use this one.

  499. 1:12:00

    Uh, and so now... [laughs] I really don't like us actually returning the clear API key here. I think we learned something today. Um, okay, we save that. Uh, and so now we have sort of our API key set up, and now what we can do is, uh, we can run our demo, and that was just the main.py.

  500. 1:12:22

    And so now our demo will be up and running on [REDACTED:url] here.

  501. 1:12:30

    And so we can see, um, it's just kind of a basics of demo. Uh, and when we connect...

  502. 1:12:37

    Top of the morning to you. I'm Gemini Live, a little demo of what this API can do. Why not try out some fun features, like hearing me speak in different accents?

  503. 1:12:47

    Hey, can you see me seeing?

  504. 1:12:52

    Aye, I can see you all right. You're sitting there with your handsome face looking straight at me.

  505. 1:12:57

    Yeah.

  506. 1:12:57

    Is-

  507. 1:12:57

    Can we speak in [REDACTED:origin], please?

  508. 1:12:58

    Of course. I'm happy to speak [REDACTED:origin] with you. How can I help you today?

  509. 1:13:02

    Sorry, I can't speak [REDACTED:origin]. Can you speak Chinese?

  510. 1:13:05

    Of course. No problem. What would you like to chat about?

  511. 1:13:09

    Have you eaten? [laughs]

  512. 1:13:09

    I am an AI, so I don't need to eat. You? Have you eaten? Do you have any good experiences?

  513. 1:13:09

    Ah, that was, that was a tough, uh, tough diss.

  514. 1:13:30

    Uh, I was asking- It understood you, though. Sorry? It understood you. Ah, yeah, yeah, yeah, but it didn't transcribe it. Oh. [laughs]

  515. 1:13:40

    All right. Uh, we're finding out a lot of things here, uh, to improve, which is nice. Um, but yeah. So what we can see is, um... So this is...

  516. 1:13:50

    You could actually notice that the latency ... is a bit worse because we're actually having that jump from our client to our server. Um, I would love to blame the Wi-Fi, but, uh, so with the server-to-server setup, you just have that additional latency of sort of going through, um, your client.

  517. 1:14:09

    So what we can do as well is we can go directly from our client to the server. Uh, and so this is the other example that we have, um, which is this one here using ephemeral tokens.

  518. 1:14:25

    So, uh, for ephemeral tokens, we can just kinda look into, uh, the setup here. Uh, very similar. We'll just go, uh, back. Uh, actually, let me get a new terminal up here.

  519. 1:14:41

    So we'll say, uh, ephemeral tokens. So our ephemeral tokens are basically short-lived tokens, uh, that we generate with our API key on the server side, and then we send that ephemeral token to our client, so, you know, our phone, our browser, uh, to then initiate the WebSocket connection directly from the client to the Live API.

  520. 1:15:07

    Uh, again, similar setup here. We want, uh, a virtual environment. Uh, we then, uh, activate our virtual environment, and we install our dependencies.

  521. 1:15:23

    Uh, we also need our, uh, API key, uh, again. So I think what we can ch- do is just use the same one. So we'll copy... Maybe just copy this one here and then paste, uh, paste that in.

  522. 1:15:39

    Is it big enough? I don't know. Can people see that? Maybe we'll zoom in a little bit more.

  523. 1:15:46

    So now we have our key here, um, as well. Uh, and so

  524. 1:15:52

    now that we have our dependencies installed, uh, we can run our server. Uh, and so our server here,

  525. 1:16:01

    um... We can look at the server real quick.

  526. 1:16:08

    So this server is basically just, uh, a very, you know, slim sort of back end that just has our Gemini API key.

  527. 1:16:18

    Uh, and then it generates an ephemeral token for us. So sort of, um, ephemeral token's currently on the V1 alpha API, so you'll need to use actually a different API, uh, at the moment for this.

  528. 1:16:31

    And then you would pass in kind of this expiration time, uh, because ideally, you know, the token should be short-lived. Uh, so should the token ever leak, you know, uh, it shouldn't be too costly because it'll expire, um, pretty soon.

  529. 1:16:47

    So there we are. That's our token, and then we return our token back to our client. So on our front end, we then have our Gemini Live kinda integration here.

  530. 1:16:58

    And so this is an example of just a pure WebSocket integration without sort of any SDK. Um, so, you know, if you... You can use kind of any sort of WebSocket framework, um, here, and you can see sort of, you know, how all the, the different sort of raw WebSocket events, um, are handled.

  531. 1:17:19

    Uh, and so you can see here we have, um, you know, our... This is kinda our WebSocket, uh, API. So here we need to use kind of the V1 alpha.

  532. 1:17:29

    Um, and then this is the bi-di, so bidirectional, um, you know, we're streaming in both directions. Uh, and here we, we pass our access token, which is our short-lived token.

  533. 1:17:42

    Uh, great. So what we can do now is now our server's up and running. Uh, so we can see, um, this beautiful interface here, which was, uh, handcrafted. Um, you know, no agent involved in the creation of this one, um, back in the day.

  534. 1:17:59

    And so we can just see kind of all the different, um, knobs here that we have. So we'll try this kinda Google grounding as well, and hopefully it was, uh, just a thing we need to fix in the UI.

  535. 1:18:11

    Um, here's our Flash Live, uh, preview model, so that's 3.1 Flash Live.

  536. 1:18:17

    Uh, and then we can, uh, connect here.

  537. 1:18:22

    And so we see kind of the, the server events. Uh, actually, let me maybe, uh, do that again. So we say enable Google Search grounding, connect. So what we can see is kinda our WebSocket, uh, setup.

  538. 1:18:38

    So if we look into the network, we now see we have kind of this, um, WebSocket connection here. So we see we have our access token. So our token request to the back end...

  539. 1:18:50

    Actually, let me... Let me see that. Uh, okay. So you can see we're just getting our token. So, um, the response was just our auth token and when it expires.

  540. 1:19:03

    Uh, what we sent in terms of the payload. Where's our payload? Mm. Okay, actually, there was no payload. Uh, yeah. We're, we're just getting a token back. So once we have our token, we then set up our, um, WebSocket, and so we can see our messages here.

  541. 1:19:21

    So basically, the first message in our WebSocket, um, we're sending our setup, our configuration. So you see we're, you know, sending our model as, like, Gemini 3.1 Flash preview.

  542. 1:19:34

    We have some real-time, uh, input there. We have the tools. So hopefully now the Google Search, uh, should work. And so now what we can do is we can, um, start our audio.

  543. 1:19:49

    Hey, can you hear me?

  544. 1:19:51

    Yep. I can hear you loud and clear. How can I help you today?

  545. 1:19:55

    Um, I'm in London right now, and I'm wondering, do I need an umbrella? What's the weather today?

  546. 1:20:04

    I'm not getting the current weather for London right now. Sorry about that. Your best bet is probably to quickly check a weather app on your phone or search online.

  547. 1:20:12

    That'll give you the latest on rain.

  548. 1:20:14

    Oh, you're killing me. Just use Google Search.

  549. 1:20:20

    My apologies. I seem to be having some trouble pulling that up right now. I know that's frustrating. You can get the forecast easily by just asking your phone or searching online.

  550. 1:20:29

    I'll try to get this sorted on my end.

  551. 1:20:32

    Gemini, why are you doing this to me? That's just, uh...

  552. 1:20:37

    I understand your frustration. Sometimes technology gives us hiccups, and I'm sorry I couldn't get that weather for you right now. I'm always learning, so hopefully next time will be smoother.

  553. 1:20:47

    Was there anything else I could help you with?

  554. 1:20:49

    All right. That was painful. Uh, I do apologize for that. Okay, so something is definitely going on. That is, um, not good. Not, not nice. Um, let's see if we have more, uh, luck with the custom tools.

  555. 1:21:07

    So we're also passing in... Do we pass in some custom tools, or is there actually some issue with our, uh... Da, da, da, da, da.

  556. 1:21:24

    Hey, can you show me a Hello World alert?

  557. 1:21:34

    There you go. A Hello World alert has been displayed. Anything else I can help you with?

  558. 1:21:40

    Yeah. Can you change the background, uh, of the page to green, please?

  559. 1:21:49

    Ooh. [clapping]

  560. 1:21:49

    You got it. The background color should now be green. Let me know if you need anything else.

  561. 1:21:54

    Yeah. Can you tell me what's the weather right now in London? [laughing]

  562. 1:21:58

    I understand you're asking about the weather in London, but unfortunately-

  563. 1:22:02

    I did not enable the Google grounding, but for some reason it's... Yeah. So with the Google grounding it should work, so I, I wonder if I messed up something.

  564. 1:22:13

    No worries at all. The weather information isn't something I can grab right now, but if you have any other questions or need help with different functions, feel free to ask.

  565. 1:22:22

    Let's try this again. Hey, can you hear me?

  566. 1:22:32

    Ah, yeah, we need some audio. Hey, can you change the, the background back to white, please?

  567. 1:22:40

    I wish I could, but I don't actually have control over your display settings. You might need to look in your device's settings menu to change that. Anything else I can help with instead?

  568. 1:22:51

    Oh, no. Sorry, of the page. The, the background of the page back to white.

  569. 1:22:57

    Oh, I see. Sadly, I still can't change that for you. Those appearance settings are part of the website or app you're using.

  570. 1:23:06

    Oh yeah, you're right. Um, no, but I th... Okay.

  571. 1:23:12

    We'll disable custom tools. That shouldn't be the case anymore. All right. Um, lots of work to do, [laughs]

  572. 1:23:20

    but, uh... So yeah, this is sort of how you can, um, get up and running.

  573. 1:23:27

    Uh, did anyone manage to get it running on their machine?

  574. 1:23:33

    Yep.

  575. 1:23:33

    Yeah? Is, uh... Is Google Search grounding working?

  576. 1:23:37

    No.

  577. 1:23:38

    No?

  578. 1:23:38

    Sadly, no.

  579. 1:23:40

    Okay. That's, uh... That's a shame. All right. Um,

  580. 1:23:50

    what else did we have? So we need to fix, uh... Lots of takes away from this. Um, yeah, if you didn't get the link, so this is the link to the Live API examples.

  581. 1:24:02

    So, um, they also link from the docs. So if you go to kind of the Gemini Live, uh, API docs, you can find them there.

  582. 1:24:12

    Um, is that the end of my slides? I do think we have... Yeah, the agent skills as well. So I mean, if you just go to the Gemini Live, uh, docs.

  583. 1:24:30

    Uh, Gemini Live API. Uh, and in the... On the main docs, uh, we've actually linked it from, uh, the top there. Uh, try the Live API, Google AI Studio, uh, the GitHub examples, or use the coding skills.

  584. 1:24:47

    Um, so that is... Yeah. That is how you can get started.

  585. 1:24:54

    Cool. That was, uh, not how I was hoping this would go. Um, yeah. We'll, we'll figure out what, what went wrong there. But, um, yeah, appreciate you all joining.

  586. 1:25:05

    And yeah, we'll, we'll take questions.

  587. 1:25:07

    Yeah. So, uh, I was just wondering, uh, how does context look like in this, uh, Gemini?

  588. 1:25:14

    Yeah. So you can look at kind of the session management. So, um, there's actually sort of...

  589. 1:25:22

    So the Live API has kind of this context window compression. Um, so without compression, kind of audio-only sessions are limited to 15 minutes. Um, audio-video sessions are limited to two minutes.

  590. 1:25:36

    Um, so what that means is that sort of

  591. 1:25:41

    you will... The session will kind of terminate. It will give you sort of this go-away ping. Um, and what you can do is sort of enabling context window compression.

  592. 1:25:53

    So context window compression is kind of like a, you know, sliding window where you basically say, you know, like, "I want to keep that much context sort of in my window."

  593. 1:26:02

    And sort of as the conversation progresses, it will then actually, like- Forget kind of the previous context before that window. Um, so yeah. So context window compression is something that you can, uh, enable kind of, you know, to have that sliding window to, uh, make the sessions longer.

  594. 1:26:22

    But then, you know, there's only so much context depending on, um, the frame rate that you're feeding in in terms of images, depending, um... Yeah, mostly, mostly it's images.

  595. 1:26:33

    Audio is sort of, uh, audio-only sessions, there's more kind of context you can keep in the window. But then as you're adding kind of video frames to it, uh, it does,

  596. 1:26:46

    yeah, compress it. Uh, yeah. In the... Oh, yeah, yeah.

  597. 1:26:57

    Um, is there any real-life use cases that this is being used for that's very creative from a, like, a business context?

  598. 1:27:06

    Very-

  599. 1:27:06

    Obviously, what you've done today is a lot of fun, but, like, applying it to, to business. Is there any businesses that are actually doing this really well right now?

  600. 1:27:14

    Yeah. Um, I mean... So Shopify has it in production with Shopify Sidekick.

  601. 1:27:20

    Mm.

  602. 1:27:20

    Um, there is, uh, a bunch of... So, like, uh, actually one that I really enjoy if you Gemini Life API blog.

  603. 1:27:32

    Uh, we had kind of a, a case study, which is, um, this startup. Yeah, so I mean, Stitch is using it as well. You can, like, vibe code w- w- vibe design with your voice in Stitch.

  604. 1:27:47

    But then, you know, Stitch is built by Google, so, um, you know, you probably gotta discount that. Um, Waymo is also integrating it, so, um, you know, we... First we got rid of the drivers in the cars, and then, you know, you do wanna talk to someone in the car, you can then, uh, talk to Gemini, uh,

  605. 1:28:06

    in the future. So they, they are working on that. Um, I like this one. So this is, uh, Hey Ado. Um, it's a, it's a great startup from Argentina.

  606. 1:28:17

    Uh, so they are building these, um, voice, uh, sort of companions for the elderly, uh, in combination with, uh, kind of an app, uh, for sort of the caretakers or, you know, the, the, the, the children of the elderly, where, you know, they can get notifications sh- yeah, should the elderly mumble about something that...

  607. 1:28:41

    No. But so it's, it's a, it's a really nice interaction where, like, the multilinguality, um, really shines, you know. Because, like, in Argentina, uh, you know, a lot of it would be Spanish.

  608. 1:28:53

    Um, but there's a nice example you can kind of look at, um, which is really sweet.

  609. 1:29:00

    I came here... I came to visit my grandma, Maria. I want you to say hi to her but remember she-

  610. 1:29:03

    Yeah. You might... You, you can, you can look at it in your, in your own time. Uh, and then I think there was another one. So, um, yes, it is somewhat

  611. 1:29:14

    more of a future music use case, just kind of looking at, you know, some of the rough edges and limitations. And I think for now, if you're,

  612. 1:29:24

    you know, like in, in a real business use case, you might be, you know, better off with kind of the cascading pipeline because it gives you sort of the observability at each step of the pipeline.

  613. 1:29:39

    Um, which kind of with the, the real-time, you know, native audio, um, you don't really have that fine-grain control and observability in terms of, you know, plugging into, say, like rewriting the response before it's being said.

  614. 1:29:56

    Um, you know, obviously there are certain benefits with that in terms of the natural flow of the conversation, but for certain business use cases it might, might just not be there yet.

  615. 1:30:06

    So it is somewhat, you know, a bet in the future of, like, what, uh, kind of real-time conversational interactions will look like in the future. Um, but yeah, depending on your business use case today, uh, it might not be the best fit just yet.

  616. 1:30:25

    Are the transcripts all stored and available?

  617. 1:30:29

    Uh, no. So you... In the session, you can't, uh, retrieve them, so you would have to store them on your, on your end. Um,

  618. 1:30:37

    a- again, so that is sort of where the, um, integration partners come in. So if, uh... Yeah. So LiveKit, Pipecast, they all have, like, really good offerings to, you know, store the entire audio as well as the entire transcript and sort of give you additional observability tooling on top of that as well.

  619. 1:31:01

    Um, so it's not something that is kind of all available sort of,

  620. 1:31:06

    you know, from just the Google site. Uh, so that's something we're currently, we're relying on kind of the partner integrations to, to sort of give you that additional functionality.

  621. 1:31:17

    Yeah. The issue I always see- Oh, sorry. Uh, behind you. Okay. Yeah.

  622. 1:31:22

    Do you wanna go first?

  623. 1:31:23

    Uh, no. [REDACTED:gender] first.

  624. 1:31:25

    Okay. Um, so you said that this is not, you know, like super business ready for more s- more complex use cases, but what are your thoughts, for example, in, um, replacing interview, interviews for recruitment using this?

  625. 1:31:40

    Yeah. I mean, I'm not sure I would, you know, replace your entire interview pipeline with that, but I think it is a great scenario, you know, for certain steps in the interview process or, you know, being able to screen more candidates, for example.

  626. 1:31:58

    Um, yeah. I do-

  627. 1:32:01

    Do you think it's, is that ready?

  628. 1:32:04

    Do I think that's ready? Um-

  629. 1:32:07

    Because there's a lot of, like- Um, context that you also wanna give it, right? Especially even if it's like a first screen or the quick screens that you're mentioning, you still want to put some criteria that would guide guardrails on how to guide the, the conversation.

  630. 1:32:22

    And then the second part to this is also,

  631. 1:32:26

    can it work, for example, with if my company's in a, a Google environment, Google Meets, right? Right now we have auto transcript, so can you put these tools together?

  632. 1:32:39

    Um, yeah. So it is, uh, a preview, so it is something you can use in production. Um, depending on your use case, I think you need to evaluate, like, you know, do you need like SOC 2 to- z- sort of there's potentially a bunch of things that you need to build around it for it to actually fit

  633. 1:33:00

    your business use case. Um, so yeah, it is ready for experimentation. [laughs]

  634. 1:33:12

    Does that help?

  635. 1:33:13

    I mean, I think that's what we're doing a lot in the company, is experimenting.

  636. 1:33:18

    Yeah. Yeah. Uh, oh, yes.

  637. 1:33:22

    Um, an issue I always have is when I'm demonstrating voice agents, a lot of people are talking, [laughs] and it's pr- uh, it can't differentiate the speakers between each other.

  638. 1:33:33

    So is there a solution for this already?

  639. 1:33:37

    Uh-

  640. 1:33:37

    Like, train it on a voice sample or something

  641. 1:33:39

    ... in terms of like identifying the different speakers?

  642. 1:33:42

    Yeah, and so it only listens to you if it's your agent, for instance.

  643. 1:33:46

    Oh, interesting. Um, it only listens to you, so it needs to-

  644. 1:33:54

    Imagine you have a coding agent and somebody makes a prank and says, "Delete the f- file" or-

  645. 1:33:59

    Yeah

  646. 1:33:59

    ... stuff like that. You don't want that. [laughs]

  647. 1:34:01

    Yeah, no, that's, that's interesting. No, I don't think there's any specific ability to sort of like say, "Just listen to me." Um, so there is sort of,

  648. 1:34:15

    kind of this proactive audio where you can tell it to kinda only respond in certain,

  649. 1:34:23

    you know, to certain contexts, so like ignore things that aren't relevant to the conversation. Um, so to some extent, that works. Um, but I don't think it's super reliable at the moment where you'd say like, "Only listen to me, ignore anyone else."

  650. 1:34:39

    I've done that with Parakeet from NVIDIA, where you can train a little 10 seconds, and it can differentiate the speakers that way.

  651. 1:34:46

    Nice.

  652. 1:34:46

    But it would be nice to have something like you talk to it, and then it recognizes your voice-

  653. 1:34:50

    Yeah

  654. 1:34:50

    ... and ignores the other voices for the rest of the session.

  655. 1:34:53

    Ignores everyone else. Mm, that's cool. That's a great, great idea. Yeah. Thanks.

  656. 1:34:58

    Uh, yeah, I think in, in the... Yeah.

  657. 1:35:02

    What does thinking look like? Uh, is it, does it think in text, or is also thinking in speech?

  658. 1:35:09

    Yeah, so you get the thinking, uh, only in, in text. So there's, uh, text events, uh, on the WebSocket channel that, um... So you can, you can opt into getting the thinking, um, as text.

  659. 1:35:25

    Yeah. It wouldn't speak out the thinking. Yeah.

  660. 1:35:28

    Uh, thanks for the demo and, uh, for being brave to go up against the demo gods. Um, I had a sort of question around the multimodality side of things.

  661. 1:35:37

    Mm-hmm.

  662. 1:35:37

    Uh, one of the areas that I really want, uh, good text to... No, speech-to-text models is having them be grounded in what I'm looking at. Like, when I'm coding, I wanna just, you know, word vomit into Cursor, or sorry, Antigravity, uh, and have it understand the context and, you know, when I say something that is a specific

  663. 1:35:57

    class name, it should just actually use that. Um, how does that work inside of this framework? Like, would, would this be a way to actually do, do that grounding, or would you recommend some other API for that?

  664. 1:36:09

    Um, and this is for... So, so general Gemini models are really good at audio understanding. So, um, if your use case doesn't require like fully real-time transcription, I would actually recommend using like Gemini 3 Flash, um, to basically

  665. 1:36:33

    transcribe but also get, you know, like ingest context and sort of basically get contextually aware transcription. Um-

  666. 1:36:44

    Does that go before the text, uh, before the sample, or... You can do it at the same time. So when you are in your editor, right, you have maybe like two files open with main.py and util.py, and you can provide...

  667. 1:36:58

    So since we are multi-modal, you can provide the file as text input and your audio as audio input in the same request. And then it has like awareness of both, and it can use both.

  668. 1:37:10

    And it can, I don't know, like generate new code or do something else. I think Stitch does this in a way. I'm not sure like if you can show it or not.

  669. 1:37:17

    But in Stitch, you have like, uh, Figma-like mock-ups, and you can speak to it. So you can like... It has the visual awareness plus the audio input on, I don't know, which button you should create.

  670. 1:37:29

    And if you only have one button, it knows which button it has to change.

  671. 1:37:34

    Yeah. There is, uh... So I mean, depending on your use case, if you need it to like be fully real-time conversational, um, then yeah. So you can kind of use text to sort of ingest additional input or, you know, imagery if it makes sense.

  672. 1:37:51

    But then again, that, you know, reduces the context, um, window size. Um, or, you know, if you don't need kind of the fully real time, um, then like using Gemini, you know, just Flash to, you know, transcribe is actually pretty good

  673. 1:38:18

    ... every second I send a new image. But if you need, like, visual awareness and your visual doesn't change, you can send the visual as, like, the first input, then speak for five seconds and not send an additional image, then you are not using so much context.

  674. 1:38:34

    I think, like, one image is around 1,200 fil- uh, 1,200 tokens.

  675. 1:38:38

    Yeah.

  676. 1:38:39

    So not too much. Um, so if you are, like, in your editor, you want real time, you basically can use the API. First input is send real-time image, and then send real-time audio, and then you stop basically, and the model has the image input and the audio input and can respond to it.

  677. 1:38:54

    So you... There's no need to stream the image consistently if you don't, if it doesn't change or if you don't need, like, it to react to it.

  678. 1:39:03

    Yeah. Cool. Yeah. Uh, I think we have a bit more time. Uh, there in the back. Yeah. How do we get you the microphone?

  679. 1:39:17

    Sorry.

  680. 1:39:17

    Or do you just wanna sh- shout it out? Oh, yeah. [laughs] We'll do a...

  681. 1:39:23

    Thank you for your interesting presentation. Uh, I have a question about the, uh, personalization or adaptation. Can it, uh, recognize the speaker's level or the knowledge during the interaction, and then based on the speaker's knowledge, produce the result or not?

  682. 1:39:40

    Sorry, can you repeat? Can it, uh, g-

  683. 1:39:42

    Can it, uh, recognize the speaker's knowledge or the background during the interaction to produce the response based on the speaker's knowledge or something like that to personalize itself to the-

  684. 1:39:54

    So you, you want to ingest kind of initial context? Is that what you're saying?

  685. 1:39:59

    Not context. For example, suppose you are talking to that about the civil engineering, and can it recognize I'm a civil engineer, and based on my knowledge, produce the result, use the advanced keyword in civil engineerings or not?

  686. 1:40:13

    Based on my knowledge. So you would, you would have to, if it's, like, special- specialized knowledge, you would have to give it a way to access that knowledge.

  687. 1:40:23

    Oh, that it doesn't have any memory to recognize the human's background or to find the main context of the information, uh, of, of the interaction, and based on that, produce the next result.

  688. 1:40:38

    So you, you have to somehow feed in that knowledge. So you could either do that sort of before you, you know, like, as you set up the session. You can ingest kind of the, the knowledge as initial context, for example, and then it has that in context to talk about.

  689. 1:40:55

    Or you would give it kinda function calls to access knowledge sort of during the session as you, as you converse.

  690. 1:41:03

    Yeah, I see. Thank you.

  691. 1:41:04

    Does that...

  692. 1:41:05

    Yeah, no. I was to find that, uh, look at the GPT, for example. During the some turns, it can find the speaker's or the user's knowledge or the main concept of the information, and based on that, the GPT can produce some results.

  693. 1:41:21

    That means step by step, gradually it personalize to the main context of the conversation. So my question is, can it, step by step, personalize itself to the main context of the in, uh, of the interaction, and then produce the result to point to the...

  694. 1:41:39

    Yeah.

  695. 1:41:40

    So, like, as long as the, the context stays within the context window during that conversation-

  696. 1:41:46

    Yes

  697. 1:41:46

    ... yeah. It, it would... It can-

  698. 1:41:48

    Okay

  699. 1:41:48

    ... just, like, it can identify different speakers and sort of remember what was said in the conversation. And, like, if they introduce themselves with their name as well, it can remember kind of that person's name.

  700. 1:42:03

    Yeah.

  701. 1:42:03

    And so, like, yeah, that's kind of the, the, the audio understanding sort of within Gemini.

  702. 1:42:09

    Thank you.

  703. 1:42:09

    Okay, cool. Uh, yeah. Do you just wanna pass it forward there?

  704. 1:42:15

    Yeah, forward.

  705. 1:42:16

    Yeah. [laughs] [coughs]

  706. 1:42:18

    Yeah. Uh, thanks, uh, for the nice presentation. Uh, could you share maybe some, um, of your experience on how to evaluate these, um, live, uh, voice apps? Because I can imagine that this becomes a lot more complicated than typical apps.

  707. 1:42:34

    Um, yeah. I've... So i- it definitely depends on your use case in terms of, like,

  708. 1:42:43

    uh, you know, like, what are your requirements in terms of, you know, do you have HIPAA? Do you have SOC 2? Like, what is, uh, the amount of function calls, the amount of guardrails?

  709. 1:42:56

    So there's definitely a lot. You know, these demos are nice and, and fun, but, like, to bring that in a bus- business context, there are definitely a lot more, um, steps involved.

  710. 1:43:09

    Uh, and so that is kinda where the partner integrations come in. So, uh, you know, LiveKit has built kind of their entire business around sort of giving you all the batteries around sort of voice agents.

  711. 1:43:21

    And so I would recommend if, you know, sort of looking at the partner integrations for the sort of real business use cases potentially.

  712. 1:43:31

    Thanks.

  713. 1:43:35

    Cool. Yeah.

  714. 1:43:41

    I hope you don't mind if I just ask a simple question about the Interactions API, going back to a previous talk.

  715. 1:43:46

    Yes, please.

  716. 1:43:47

    When's that gonna be available on Vertex?

  717. 1:43:49

    Um, hopefully soon. I mean, if you speak to some Google Cloud person, some Vertex person, the more you tell them, "I need it on Google Cloud," the, the easier it gets. [laughs] [laughs]

  718. 1:44:02

    Will do.

  719. 1:44:02

    Yeah, it's certainly not in our control. Um-

  720. 1:44:04

    Okay. Yeah, that's fair. That's fair. That's cool. Thank you.

  721. 1:44:05

    But, but, I mean, it, the API will be the same, so you can start today on Gemini API, uh, start testing. If you need higher rate limits, anything else, you can, like, always reach out.

  722. 1:44:17

    If you need Vertex enterprise specific features, then you might need to wait a little bit.

  723. 1:44:24

    Uh, can I ask, like, in terms of, like, PII in any, like, conversation history, do you know how that's, like, stored in terms of, like, like, data sovereignty? Can you specify your own data sources, or is that all handled, like, back-end, um, with- with- like, within the API?

  724. 1:44:37

    So you can always disable storing anything. So we have a store equals false flag, so we would not store anything. But not storing means no server-side state. So if you would like to use this, that's a bit difficult.

  725. 1:44:49

    For other Vertex features in terms of data sovereignty, where you call the model, I would expect them to be, like, similar to generateContent. So if they have it today, they will have it there in the future as well.

  726. 1:45:00

    Okay. Cool. That's great. Thanks. Appreciate it.

  727. 1:45:07

    Cool. Uh, one last one. No? Okay. Ah. [coughs]

  728. 1:45:13

    Sorry.

  729. 1:45:13

    No, please.

  730. 1:45:16

    Thank you. So I have a question about, uh, hallucinations. So we-

  731. 1:45:21

    Sorry, the what?

  732. 1:45:22

    Hallucinations.

  733. 1:45:23

    Yes.

  734. 1:45:23

    So you've have shown some examples, uh, with the weather that, uh, [laughs] didn't work so well. Uh, but, uh, how do your clients actually deal with that stuff on production?

  735. 1:45:38

    Because I can imagine that for, like, some examples that we've seen here, this is fine, but, uh, in real life this is a different story. So can you give some best practices or how to deal with that?

  736. 1:45:49

    Yeah, definitely. Um, so I mean on, uh, for the demos, the, there's definitely a lack of best practices in terms of, like, system instructions and, you know, there's a lot that you can do sort of with- [coughs] ...

  737. 1:46:04

    uh, the, you know, better system instructions to, uh, have the agent actually follow, um, the system instructions and not, you know, go off and, like, hallucinate the weather, for example.

  738. 1:46:16

    Um, so yeah, I think we, we have, like... There's some best practices docs, um, so I'd recommend kinda going through, through those. We have an example as well, sort of, you know, how to sort of structure, um, your, your system prompt and sort of put, you know, your guardrails in there, guidelines and, um, kind of the tool

  739. 1:46:37

    definitions as well. And so once you've built that up, uh, the agent gets a lot better, you know, at following the system instructions and kind of staying within those, those parameters.

  740. 1:46:53

    Cool. Cool. Um, yes, thanks so much everyone. I, I do apologize for the hiccups, but we, we learned something, and we'll, we'll improve upon it. Uh, but yeah, would love for y'all to test it out and, um, you know, let me know over the next couple days, I'll be around, what you find, and let me know your

  741. 1:47:12

    feedback. Thank you. Cheers. [applause] [upbeat electronic music]