AI Engineering with the Google Gemini 2.5 Model Family

How do you get from a personal Google account to a working model request, without first setting up a cloud development environment? Philipp Schmid’s workshop starts with that practical constraint. Gemini 2.5 Pro and Gemini 2.5 Flash accept text, images, audio, video and documents and produce text; Flash is the workshop choice because free-tier API access is available at the time. Image generation and the newly introduced text-to-speech models are separate parts of the family. Model cards describe the supported modalities, context capacity and output limits.

The working materials are in Schmid’s Gemini 2.5 workshop repository. A setup notebook precedes four parts: text generation, multimodality, structured outputs and tools, then MCP. The notebooks directory mixes working examples, partially completed cells and exercises; solutions contains completed counterparts. Participants can work at their own pace, using the workshop Slack channel for questions during or after the session.

Each notebook has an Open in Colab button; cloning the repository into a local Jupyter environment is the alternative. AI Studio provides the first place to experiment. Schmid asks about the AI Engineering Summit, enables Google Search in the settings and runs the Flash preview model. The resulting answer includes sources. The Get Code action turns that configured experiment into SDK code, so the prompt and settings can move into a Python application.

Authentication follows a short sequence:

1. In AI Studio, choose Get API key, then create a key after selecting or creating a Google Cloud project.
2. In Colab’s Secrets panel, store the value under GEMINI_API_KEY.
3. For a local notebook, expose the same name as an environment variable.
4. Run the setup notebook’s first generation cell.

The setup notebook includes screenshots and a cell that detects Colab. Its smoke test uses Gemini 2.0 Flash, even though the subsequent workshop focuses on 2.5. That initial test establishes that credentials and the client work before adding more capabilities.

The Google Gen AI Python SDK organizes requests around a client and its models interface. generate_content accepts a model ID and contents; replacing the model ID changes the model without changing the basic request structure. The first text task asks for three names for a coffee shop that emphasizes sustainability.

import os
from google import genai

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
model_id = os.environ["GEMINI_MODEL_ID"]

prompt = "Generate three names for a coffee shop that emphasizes sustainability."
response = client.models.generate_content(
    model=model_id,
    contents=prompt,
)
print(response.text)

Set GEMINI_MODEL_ID to the model you intend to use. The recording uses preview-era models and pricing; its particular IDs and rates should not be treated as a current deployment configuration.

count_tokens accepts the model and input before generation. It measures the prompt, not an answer that has yet to be produced, and it does not return prices. Schmid’s token-count example returns 11 input tokens; he separately estimates approximately $0.00002 using pricing he looked up for Flash. After a generation, response.text gives convenient access to the answer, while response.usage_metadata exposes the accounting needed to understand the request.

The important distinction is between prompt tokens, thought tokens and candidate tokens. Thought tokens represent computation before the final answer; candidate tokens are the answer itself. A cost calculation that only counts the visible response misses the reasoning usage. In the thinking mode discussed here, thought and candidate tokens share the output rate:

Here each p is a price per token. Schmid estimates the demonstrated haiku request at less than 0.2 cents. For comparison, he quotes workshop-era Gemini 2.0 Flash rates of $0.10 per million input tokens and $0.40 per million output tokens. Those figures describe that model’s pricing, not the thinking-mode split explored next.

The haiku example uses 639 thought tokens. Why that many for a short poem? The model chooses its reasoning usage dynamically from the prompt; the raw internal reasoning is not directly exposed, although thought summaries are available. A thinking budget gives the caller a way to constrain that usage. It measures tokens, not a number of parallel reasoning traces.

Flash is described as a hybrid model: a zero budget disables thinking, while a positive budget allows reasoning before the answer. Schmid connects the extra cost to the attention computation over a growing sequence. His preview-era pricing explanation distinguishes two modes:

The key accounting point is that enabling thinking changes the applicable output mode in this explanation; it is not merely a separate charge for a hidden preamble.

In configuration, thinking_budget=0 disables thinking. Schmid rounds the upper limit to about 24,000 tokens; the Flash preview launch documentation specifies 24,576. He illustrates the control by proposing a budget of 512 for the haiku request that had consumed more than 600 thought tokens. The usable engineering control is the token budget; the discussion does not establish how internal reasoning paths are chosen.

Waiting for a complete answer is often the wrong interaction for a chat interface. generate_content_stream takes the same basic inputs as generate_content, but returns an iterator. Each chunk can be printed immediately or forwarded through an HTTP service:

for chunk in client.models.generate_content_stream(
    model=model_id,
    contents="Explain why token accounting matters in an LLM application.",
):
    if chunk.text:
        print(chunk.text, end="", flush=True)

Streaming changes how the application receives the response; it does not require a different prompt format.

The Chats API addresses a different concern: retaining the sequence of user and model turns. In the trip-planning example, the first response proposes European cities. A follow-up asks for food recommendations, and the model can interpret it using the earlier exchange.

chat = client.chats.create(model=model_id)
print(chat.send_message("Help me plan a trip to European cities.").text)
print(chat.send_message("What food should I try in those cities?").text)

history = chat.get_history()

Chat history lives in the client SDK. The backend receives the conversation again with each request, much as it would if the application manually constructed the turn array. get_history() exposes that state for persistence. If the client process disappears without saving it, the session is lost.

Generation configuration carries system instructions and sampling controls. Schmid suggests lower temperature for a retrieval-based task that should stay close to supplied context, and higher temperature for creative writing. max_output_tokens constrains output length; top_p and top_k provide additional sampling controls. The same configuration object also contains the thinking settings:

from google.genai import types

config = types.GenerateContentConfig(
    system_instruction="Answer in English and use the supplied context.",
    temperature=0.2,
    max_output_tokens=1024,
    thinking_config=types.ThinkingConfig(
        thinking_budget=512,
        include_thoughts=True,
    ),
)

Requesting thoughts exposes summaries, rather than the full internal reasoning.

The next input is the complete text of The Adventures of Tom Sawyer. Schmid downloads the book, uploads it with the Files API, and supplies the returned file alongside a summarization instruction in contents. The request reports approximately 100,000 tokens. The follow-up exercise combines the book with a chat session so the reader can ask questions across multiple turns.

The file reference changes transport, not what the model must read. The client sends a URI; the backend resolves that reference and puts the file contents into the model’s context. Reusing the URI avoids uploading the same bytes for every question. Schmid describes access as controlled through the caller’s API credentials rather than through a publicly accessible link.

A file still has to fit the selected model’s context window. For a document exceeding the million-token context discussed in the workshop, Schmid expects an error rather than automatic chunking. Count the file-aware input first, then decide whether to split it, summarize it or extract only the information needed for the task:

book = client.files.upload(file="tom-sawyer.txt")
contents = [book, "Summarize the book's main events."]

count = client.models.count_tokens(model=model_id, contents=contents)
print(count.total_tokens)

response = client.models.generate_content(
    model=model_id,
    contents=contents,
)
print(response.text)

A reusable file pointer does not increase context capacity. Reducing an oversized input remains the application’s responsibility.

Uploads are described as free and temporary. The recording’s retention estimates are tentative; the current Files API documentation specifies automatic deletion after 48 hours, with manual deletion also available. It does not establish the configurable upload lifetime suggested earlier in the session. For storage under an organization’s own control, Schmid points to Vertex AI with an owned Cloud Storage bucket and different client initialization.

Does a PDF input preserve more than its text? The workshop tests that question with a supermarket receipt. Schmid describes a pipeline that supplies OCR text together with a page image, avoiding a manual conversion-and-OCR step in the notebook. He reports better results from combining text and imagery, while acknowledging that he does not know the exact internal preprocessing. The document-understanding documentation supports treating PDFs as visual inputs, but its newer model-specific details should not be read backward into this demonstration.

The first attempt lacks the local receipt asset. Colab opens the notebook without necessarily bringing along all repository files, so the image, audio and PDF exercises may require downloading and uploading assets separately. Once the receipt is available, Schmid shows purchases from REWE, including butter, bread and sweet potatoes. The REWE receipt demonstration returns the correct total of 20.20, in German, despite an English prompt.

An audience question returns to thinking budgets: does turning thinking off affect the answer, or merely hide a process? Schmid does not claim to know the internal mechanism. He compares token budgets with coarse reasoning-effort settings, using 1,000, 12,000 and 24,000 tokens as illustrative low, medium and high values—not named API presets. He reports that math tasks can lose quality without thinking, while ordinary OCR and image tasks may work well without it. The proposed experiment is to evaluate budgets of zero, 1,000, 2,000 and 4,000 tokens, then select a setting using the required accuracy and acceptable cost.

Input support also needs a concrete boundary. The workshop covers JSON, PDFs and supported image, audio and video formats. For source files such as .jsx and .vue, Schmid suggests using .txt when the extension is rejected. An Excel workbook needs conversion into a supported representation, such as extracted text or Markdown, rather than assuming every document format is accepted directly.

The receipt can also be tested in AI Studio and exported through Get Code. Schmid describes the UI and SDK as reaching the same underlying model and APIs with equivalent parameters. In this exported example, however, the document appears inline as base64 rather than as a Files API URI. That distinction matters when deciding whether an application should embed input bytes or reuse an uploaded object.

For token accounting, pass the same model and complete contents to count_tokens that will be used for generation. In the receipt example, Schmid describes roughly 500 document tokens and around 20 prompt tokens; the audience refers to a displayed count of 560. The preflight count matches the completed request’s prompt usage, and the response reports 42 thought tokens and 78 candidate tokens. His subsequent explanation of OCR billing and roughly 500 tokens per image is tentative; the useful practice is to measure the actual input, not derive a universal image-token rule from this receipt.

Page imagery lets the model use tables, diagrams and spatial relationships. To test a simple hierarchy, Schmid finds a mind map about making mind maps and asks for its central concept. AI Studio answers HOW TO MIND MAP. This demonstrates a question about visual organization, rather than merely requesting a transcription of every label.

A more difficult case is a dense table whose structure becomes ambiguous at page-image resolution. Schmid suggests separating the problem: use existing document tools to extract a table or visual region, then ask the multimodal model questions directly about that region. The next proposed invoice task is arithmetic over entries, which tests understanding beyond text extraction. He also points to video summarization, timestamp lookup and sectioning as applications of the same direct-question approach. For YouTube videos under an hour, Schmid describes getting a summary in roughly 80 seconds.

Content filtering is a separate control. AI Studio exposes safety settings under Advanced Settings, and the SDK also supports them. Schmid has the configurable filters off during the demonstration; he describes classifications before and after generation for categories such as explicit or hateful content. Those settings do not answer whether a badly laid-out document will be interpreted correctly.

For overlapping headers, footers and images in automatically generated PDFs, Schmid has no calibration result to offer. Human readability is a rough diagnostic: if people struggle to untangle the page, the model may struggle too. The stronger test is a representative evaluation set. He suggests moving beyond a handful of UI examples to a substantial batch—potentially a thousand PDFs—and investigating failures with the support team. The quickly established Flash API and Pro UI access make that experimentation accessible, but successful setup is only the beginning of document evaluation.

For defense, finance and organization-wide governance, the discussion moves beyond notebook settings. Schmid recommends Vertex AI for enterprise controls and support, mentions regional endpoints, and recalls an announcement about delivering Gemini into a customer environment. He does not supply deployment details for that appliance or a complete mapping from application guardrails to global governance. The audience’s cost, performance and policy-integration questions therefore remain requirements to resolve with the cloud platform team.

Returning to the invoice, Schmid asks for the sum of unit prices rather than the printed total. He reports that the calculation works on this well-formatted example. The practical starting point is to ask the question directly against the image, then add preprocessing where evaluation shows it is needed. The next step is to make the answer usable by software.

Structured output defines the shape of the result. The workshop uses Pydantic to describe a recipe with a name and a list of ingredient strings, then nests those recipes inside a list container. The response configuration supplies a JSON MIME type and a response schema:

from pydantic import BaseModel
from google.genai import types

class Recipe(BaseModel):
    name: str
    ingredients: list[str]

class RecipeList(BaseModel):
    recipes: list[Recipe]

response = client.models.generate_content(
    model=model_id,
    contents="Generate two popular cookie recipes.",
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=RecipeList,
    ),
)

recipes = RecipeList.model_validate_json(response.text)
for recipe in recipes.recipes:
    print(recipe.name, recipe.ingredients)

The JSON can be parsed into typed objects instead of requiring downstream code to interpret prose.

The same mechanism works with multimodal input: provide a document and the schema for the information to extract. A useful mistake occurs when Schmid leaves the recipe schema attached to the supermarket invoice request. In that run, the model returns an empty recipe list. After changing the schema to invoice data, it extracts the date, purchased items and prices. The schema must match the task; a structurally valid result can still be useless if the application requested the wrong structure.

Function calling uses structured output for a different purpose. Instead of returning an invoice record, the model returns a function name and arguments. A declaration tells it the function’s name, description and parameter schema. With a weather tool available, asking about Tokyo produces a request for get_weather with location="Tokyo". Changing the prompt to a greeting produces a conversational response without a function call.

The model selects the call; the application executes it. The manual sequence is:

1. Send the user’s question and tool declarations.
2. Read the returned function name and arguments.
3. Dispatch to the corresponding application function.
4. Add the function result to the conversation.
5. Ask the model to continue using that result.

In this demonstration, the weather function returns dummy data. The final answer describes Tokyo as sunny, at 22°C and feeling like 24°C. The same dispatch boundary could instead call a database, a real API or an email service; those effects occur in the application’s environment.

MCP preserves this loop while standardizing tool discovery and invocation. An MCP client calls list_tools, obtains declarations and supplies them to the model. When the model selects a tool, the client invokes it through the server instead of dispatching to a local Python function. A provider-maintained weather or storage integration can therefore be shared by many applications instead of each team writing the same wrapper.

Tool availability does not eliminate tool-selection difficulty. Schmid’s workshop-era recommendation is to expose roughly five to ten tools, using embedding similarity between the user request and tool descriptions to choose a smaller candidate set when necessary. Asked about sustained autonomous operation, he recalls a Gemini 2.5 Pro Pokémon Blue completion and an approximately 200-hour run, without validating the audience’s comparison to an Anthropic run. The general engineering constraint is context growth: long-running agents need a strategy for summarizing or compressing the conversation, and elapsed runtime depends on the task and orchestration.

Native tools move execution to the service backend. The developer enables a capability rather than writing its declaration and executing it locally. Schmid notes that these internal steps are not exposed as the same explicit conversation sequence used in the manual tool example. Structured output, despite appearing near tool settings in AI Studio, is not itself a native tool.

The distinction becomes visible in the population example. Schmid initially asks for a function to sort five cities by population. Gemini writes code but does not run it. An audience member points out that the prompt asked to create a function. After changing the request to use Python, the model produces executable code and runs it. Schmid describes a Matplotlib chart and points to the notebook’s chart-retrieval example; the selected frame shows a population table and a chart note, not the chart itself.

The URL context tool handles another routine application step: fetching a known page and supplying its content to the model. In the workshop’s Python-benefits example, the caller provides the URL rather than manually copying the page into the prompt. Schmid states that URL context accepts up to 20 URLs in one request. The retrieved material becomes context for the answer; code execution can also return generated images through the API.

Google Search begins with discovery instead of a caller-selected page. A question about recent renewable-energy developments is transformed into one or more search queries. Search results return to the model, which synthesizes the answer. Grounding metadata associates parts of that answer with the websites used, even though the complete internal tool-call sequence is not exposed in the same way as custom function calls.

Backend execution still has an accounting boundary. Retrieved context contributes tokens to the request. Schmid tentatively quotes a Google Search allowance of 1,500 free searches, followed by $35 per 1,000 searches; he describes Python execution as free and URL context as a preview whose future pricing he cannot specify. These are the workshop’s billing descriptions, not a current price contract. The composition exercise combines Search with URL context or code execution—for example, discovering a current language version before writing and running a script.

A request for a Deep Research API leads to a composable alternative rather than a release commitment. Schmid describes an open-source LangGraph example using Gemini 2.5 Flash and Gemini 2.0. It turns a question into search queries, runs multiple searches, reflects on whether the question has been answered and loops back when more research is needed. Reflection supplies the decision between another search and answer generation.

The loop makes cost accumulation easier to see. In Schmid’s example, retrieving a 10,000-token blog post through URL context adds 10,000 billable prompt tokens. Custom function calling also generates tokens, but the function itself runs on the application side; any external API pricing belongs to that service. A research agent therefore needs to account for both model context and the tools used to obtain it.

An audience question about overlapping weather tools leads into execution dependencies. Starting a disco ball and starting music are independent actions. Looking up outdoor weather and then setting a thermostat is sequential: the second call needs the first call’s output. System instructions can establish the intended order, and the next model turn may request another function rather than produce a final answer.

For ordinary parallel orchestration, Schmid describes collecting results before continuing; a slow function may need an error or not-ready response. The Live API’s asynchronous calling addresses a different problem: a spoken support conversation should not fall silent while a backend lookup runs. The application starts the work and injects the result when available. Python and JavaScript examples are referenced in the documentation, with scheduled or interrupted states discussed as signals to the developer. Asynchronous conversation is not synonymous with parallel execution.

Observability remains an application concern. At workshop time, AI Studio lacks the requested API-call log view; Schmid suggests LangSmith, Arize AI Phoenix or Vertex AI. A separate question asks whether Live API bounding boxes and coordinates have improved. He notes the move from Gemini 2.0 Flash availability to 2.5 Flash, but does not claim a measured spatial-quality improvement. That comparison needs testing.

The source-selection distinction is straightforward: URL context retrieves a known link, such as a personal blog; Search derives queries from a question when the source is not already known. Neither explanation promises access to paywalled material. For content the application is authorized to retrieve through a subscription, a custom Live API function can fetch it and return the result as context.

The SDK’s MCP integration packages the manual loop into a smaller interface. Start the weather server, initialize an MCP session and pass that session in the generation configuration’s tools list. The SDK discovers tool declarations, sends them to Gemini, invokes selected tools through MCP and returns their results to the conversation until the model can answer.

The local weather server requires a Node-based launch command. The Colab attempt fails in this environment, so Schmid moves the same notebook into Cursor locally. There, the server uses Open-Meteo to answer a San Francisco weather question. The response includes a forecast value of 17°C. The visible result demonstrates the completed MCP loop after the environment switch, rather than merely a configured session.

The connection pattern is compact:

import asyncio
import os
from google import genai
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
    server = StdioServerParameters(
        command="npx",
        args=["-y", "@philschmid/weather-mcp"],
    )
    async with stdio_client(server) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            response = await client.aio.models.generate_content(
                model=os.environ["GEMINI_MODEL_ID"],
                contents="What is the weather in San Francisco?",
                config={"tools": [session]},
            )
            print(response.text)

asyncio.run(main())

For a remote service, the exercise replaces the stdio connection with a Streamable HTTP client. DeepWiki, from Cognition, provides repository documentation and question answering through a public remote MCP service. The transport changes; the initialized session remains the SDK integration point.

ADK, the Agent Development Kit, sits above this client-level integration. It adds orchestration, MCP integration, multiple-agent management and deployment support. Schmid presents it as a choice about abstraction: frameworks can accelerate the first implementation, but understanding failures later may require digging into the layers they hide. Direct SDK use keeps those layers explicit; ADK supplies more of them ready-made.

The later video discussion explains the timestamp mechanism. Schmid describes processing video at one sampled frame per second, pairing each frame with its timestamp. At that sampling rate, a 60-second clip contributes 60 images. Sampling avoids sending every frame of a source video running at 24 frames per second or more, and he connects that reduction to fitting roughly an hour of video into a million-token context. Questions can target an action, a spoken statement or its time in the recording. Ordinary MP4 uploads work as multimodal input without depending on a YouTube transcript; an audience member reports good results with Dutch-language material.

Memory introduces a different distinction:

• Short-term memory: the conversation included in the current request state.
• Long-term memory: information from earlier interactions or about the user, stored outside the model and supplied when relevant.

Schmid describes Mem0 as extracting a useful representation from conversations, but does not establish its effectiveness with Gemini. The quality of extraction and selection determines what useful information reaches the long context. Asked about larger windows, he says two-million-token support is being worked on without a general-availability date. The subsequent ten-million-token research discussion raises cost as a constraint; his $50-per-million-token example is hypothetical, not a product rate.

MCP likewise changes how context is assembled, not who chooses a tool. A manually written JSON schema and a schema discovered from an MCP server look the same to the model. One internal team can maintain a wrapper that other teams reuse. With four available tools, the caller supplies four names, descriptions and parameter schemas; Gemini uses the prompt and declarations to choose no call, one call or multiple calls. Easily attaching servers with 50 or 100 tools makes this selection problem harder, even though integration becomes easier.

Agent2Agent, or A2A, addresses interoperability between agents rather than this model-to-tool boundary. Schmid describes agents built with frameworks such as LangChain, LangGraph and LlamaIndex calling one another through a shared protocol. He points to ADK examples rather than claiming that A2A is a direct Gemini SDK feature.

Programmatic URL retrieval is not the same as operating a website. For JavaScript-heavy pages whose useful content is difficult to extract, Schmid describes browser and computer-use work then being tested with selected companies. The model would return structured actions for a browser running locally or in a hosted environment. Work with the Cloud Run team aims to simplify running a controllable Chrome instance. These are preview directions in the conversation, not a demonstrated browser-control implementation.

MCP authentication adds another application responsibility. Schmid describes OAuth login through a client, using an Atlassian login pop-up as an example, and raises the more consequential question of scope: should an agent access every repository, or only the repository needed for the task? One protocol detail needs precision: his spoken example uses 403 as the login trigger, but the March 2025 MCP authorization specification uses 401 for missing or invalid credentials and 403 for insufficient permissions or scopes. Auth0 is mentioned as a participant working on these agent-authentication problems.

The attribution discussion separates fields that can look interchangeable in an application. Search grounding support links a span of the generated answer to source websites, allowing a UI to highlight text or add reference numbers. The audience then clarifies that it is asking about citation_metadata, not grounding. The response reference defines these separately and associates citation metadata with possible recitation of copyrighted training material. It should not be treated as a replacement for search provenance.

Direct quotations from a user-supplied document are a third requirement. An audience member describes Anthropic’s document-citation experience and asks whether structured output or URL context could provide something similar. Schmid says Gemini does not offer the same experience at that point and suggests trying prompting strategies. The follow-up about NotebookLM receives no implementation answer. A requested citation-shaped output therefore should not be mistaken for a built-in guarantee of exact source quotation.

The closing MCP availability question concerns desired first-party services such as Gmail and Drive. Schmid says those remote integrations are not yet available in the form being requested; this does not negate the earlier public DeepWiki example. His security argument is that providers understand their own OAuth and permission systems and are well placed to maintain narrowly scoped integrations. Asked how websites should prepare for Google Search AI Mode, he offers only tentative advice—high-quality content and ordinary web standards—without a special schema requirement.

Repeated document questions introduce a simple prompt-layout choice. Put the stable PDF first and the changing question afterward:

pdf = client.files.upload(file="invoice.pdf")

for question in (
    "What is the total amount?",
    "Which items were purchased?",
):
    response = client.models.generate_content(
        model=model_id,
        contents=[pdf, question],
    )
    print(response.text)

Schmid says this arrangement should allow automatic caching to reuse the stable prefix. Changing the beginning of the prompt prevents that long shared prefix from matching. The session does not show a measured cache hit; the demonstrated principle is how to arrange reusable input.

The final technical questions concern grounding failures. An audience member reports start and end offsets outside the answer’s range. Schmid has not encountered that result and asks for a reproducible example; he also notes that a null start index can represent the beginning. The exchange does not resolve offset units or diagnose the reported bug, so an application should not silently reinterpret unexpected indices.

Grounding supports and grounding chunks also stop short of providing the source passage itself. The returned span identifies part of the generated answer and connects it to a website; it does not include the corresponding excerpt from that page. That boundary matters for a product promising inspectable evidence: a linked answer span is not the same artifact as a quoted source passage. Schmid asks for these needs and failures to be sent to the Search tool team, then closes by encouraging participants to continue the notebooks and report concrete problems through social channels or GitHub issues.