AI Engineer Summit 2025
Function Calling is All You Need
Read the talk
Function Calling Is All You Need
Build from a weather-function round trip to persistent memory, delegated model calls, background tasks, and agents that can change their own tools.
From a talk by Ilan Bigio
Before you start: Basic Python, JSON, and chat API familiarity will help; the article introduces the function-calling loop before extending it to asynchronous tasks.
From completing a question to taking an action
Ask a base language model how to get to the park, and it might continue the story by explaining that Sally asked that question yesterday. The text is plausible, but the question remains unanswered. With GPT, GPT-2, and GPT-3, building a chatbot meant arranging examples of questions and answers so that answering became the likely continuation.
InstructGPT made instruction following an explicitly trained behavior. User and assistant roles then gave conversations a more useful structure. Adding functions extended that progression again: an assistant could request an interaction with external state. The old completion-only Playground makes the difference tangible—one text window had become a conversation with instructions and available actions.
Coherent continuation was itself a milestone. Bigio recalls the GPT-2 unicorns-in-the-Andes example: the model produced multiple paragraphs whose later details referred back to earlier context. But maintaining a story and obtaining information from outside that story require different capabilities.
WebGPT supplied an early action interface. In 2021, a GPT-3 model was trained to use a fixed set of browser commands. The application parsed its requested actions, performed them, and returned the observations to context. Human demonstrations on Reddit Explain Like I’m Five questions provided examples of searching and answering; preference training helped select desirable responses. The model could now learn a cycle of action and observation, although its tools were specialized.
Meta’s Toolformer explored learning API use from a small number of demonstrations, with tools such as question answering, calculation, and translation. Bigio describes the central intuition through a calculator call: insert the call where its answer makes the remaining sentence easier to predict. Comparing continuation probabilities provides a signal for whether the inserted tool use helped, reducing the need to label every useful call by hand.
OpenAI’s June 2023 function-calling launch turned that capability into a general interface developers could supply at request time. Bigio corrects himself from pre-training to post-training: the model had been taught to select functions and generate arguments in the expected format. Additional systems and training can still matter, but this interface is enough to begin constructing agent loops, retrieval, memory, workflows, and delegation from ordinary code.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model requests; the application executes
Functions serve two broad purposes: fetching information and changing something. Reading APIs, retrieving documents, and loading memories belong to the first group. Writing through an API, updating frontend or backend state, and advancing a workflow belong to the second. A workflow action can also change the assistant itself by replacing its prompt, loading another tool set, or handing the conversation to a different agent.
A function call expresses intent; it does not execute your code. The exchange has five steps:
- Supply function definitions and the user’s input.
- Receive the model’s requested function and arguments.
- Parse the arguments and execute the corresponding application code.
- Return the result to the model.
- Let the model use that result in its next response.
The application owns the boundary between a proposed operation and an actual state change.
The function-calling guide treats tool design as interface design. Explain each parameter, describe when the function should be used, and provide instructions or examples where necessary. Apply the principle of least surprise: if a person cannot infer how to use an interface, the model may struggle too.
Types should rule out contradictory requests. A toggle represented by two Boolean parameters can allow both on and off to be selected together. An enum such as state: "on" | "off" expresses the actual choice directly. Object structure and enums reduce the invalid states that the model can represent before execution even begins.
In OpenAI’s terminology, tools are the broader category. A raw function is an interface whose execution you manage; hosted tools such as Code Interpreter and file search also include an execution service. Bigio presents this as an adopted API convention rather than a universal definition. Audience questions immediately push beyond the single-function case: how should a model choose among a large library, could permissions help narrow it, and what happens when one tool supplies another tool’s inputs? Those questions motivate the later routing and delegation examples; the workshop does not build a separate dependency scheduler.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A weather call becomes an agent loop
The first weather example stops at a requested call. Nothing handles it yet. Bigio then fills in the application side using the documentation’s weather implementation: parse the generated arguments, invoke get_weather, append its result, and request another completion. The final answer reports the temperature in Paris, although Zoom obscures the numerical value during the demonstration. This completes one request–execution–response cycle.
An agent extends that cycle until the model has no more tool calls to make. Bigio names the operation run_full_turn: one user turn can contain several model requests and several rounds of tool execution. Each assistant message enters the history before its tool results; each result is associated with the call that requested it. When the assistant produces a message without tool calls, control returns to the user.
The essential Python control flow can be written independently of the functions it will execute:
python
import json
from openai import OpenAI
client = OpenAI()
def run_full_turn(messages, tools, functions, model):
while True:
response = client.chat.completions.create(
model=model,
messages=messages,
tools=tools,
)
message = response.choices[0].message
messages.append(message.model_dump(exclude_none=True))
if not message.tool_calls:
return message.content
for call in message.tool_calls:
arguments = json.loads(call.function.arguments)
result = functions[call.function.name](**arguments)
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": json.dumps(result),
})
Here tools contains the schemas visible to the model, while functions maps allowed names to Python callables. The loop is the same whether a function reads weather, searches documents, or updates application state.
Writing schemas by hand is not necessary for every example. Bigio adds a functions-to-schema utility that converts a Python function object into the API’s schema representation, then supplies a mock get_weather that returns 20 degrees Celsius. After correcting a helper invocation that was mistakenly treated as part of the agent class, the call produces a completion through agents.run_full_turn.
The workshop then switches to Swarm for its convenient demonstration loop. Its basic execution pattern remains the one just built: call the model, handle functions, append observations, and continue. Keeping the underlying loop visible makes the framework a convenience rather than a prerequisite for understanding the examples.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Memory starts with a list and a file
The smallest useful memory bank is a list with two functions: append a memory and retrieve memories. The interesting part is the interface description. Bigio tells the model to store factual information about the user’s life or preferences and to keep the stored text concise. A function docstring becomes part of the tool description, so it influences both when the model writes and what it chooses to preserve. Expiration is floated as an additional field, but no expiry behavior is demonstrated.
Persistence requires moving the list outside the conversation. The example reads a local JSON file at startup and writes it after changes. To fit the existing demo loop, Bigio instructs the assistant to call get_memory on its first turn. That instruction is a workaround for initialization, not a property of memory itself: an application could load the relevant memories before making the first model request.
The storage layer is small enough to inspect directly:
python
import json
from pathlib import Path
memory_path = Path("memory.json")
memory_bank = (
json.loads(memory_path.read_text())
if memory_path.exists()
else []
)
def add_to_memory(memory_text: str):
"""Store a concise fact about the user's life or preferences."""
memory_bank.append(memory_text)
memory_path.write_text(json.dumps(memory_bank, indent=2))
return "Memory stored."
def get_memory():
"""Retrieve the user's stored memories."""
return memory_bank
Registering these functions gives the model an interface to the file-backed list; the file operations remain ordinary application code.
The first test exposes a wiring error: the functions were not registered under Swarm’s expected functions field, so apparent memory behavior was not backed by the intended calls. Once corrected, get_memory returns an empty bank. Bigio supplies the fact that he is six feet tall, checks that it was written to the file, ends the session, and starts another conversation asking how tall he is. The new conversation retrieves the stored fact.
Loading every memory is sufficient for this example. A larger bank could use semantic similarity or search to retrieve only relevant entries. That is also the connection to retrieval-augmented generation: expose retrieval as a function, execute the search in application code, and put its results into the next model context. Selective memory retrieval is proposed here, not implemented in the live example.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Delegation is another function call
Before adding delegation, Bigio opens Swarm’s implementation. The non-streaming client.run path repeats the same completion-and-tool-result loop, with additional handling for context and handoffs. Delegation can therefore be introduced at the function boundary without replacing the whole agent.
| Pattern | What changes | Where the result goes |
|---|---|---|
| Handoff | Active prompt and tools | The new agent continues the conversation |
| Nested call | A function invokes another model | Back to the calling agent |
| Managed task | Work runs under a task handle | Retrieved or delivered later |
A handoff transfers control. A nested call preserves the foreground agent and gives it another source of answers. Managed tasks add an explicit lifecycle for work that should not hold up the conversation.
The first implementation is a nested call to o1. A function accepts a task_description, calls client.chat.completions.create, and returns the result. Its description tells the foreground model to use it for difficult work instead of trying everything itself. The request supplies the task as a user message without a system message:
python
from openai import OpenAI
client = OpenAI()
def delegate_to_smarter_model(task_description: str):
"""Use for difficult tasks that need additional reasoning."""
response = client.chat.completions.create(
model="o1",
messages=[{"role": "user", "content": task_description}],
)
return response.choices[0].message.content
The stronger model is reached through the same function interface as weather or memory. Even the parameter name helps communicate what information the caller should supply.
The test begins with a poem about dogs and Earth whose alternating words follow the alphabet. Bigio then adds a one-sentence response limit and changes the request into a haiku with both starting-letter and ending-letter constraints, explicitly describing it as hard to encourage delegation. The terminal appears idle because the function call is printed only after the nested request returns.
Eventually a result arrives. Bigio judges it only tentatively correct and suggests GPT-4o might have struggled with the task; the workshop does not establish a controlled model comparison. The clear result is an interaction problem: while the nested call runs, the user waits. Delegation has moved the computation elsewhere without making the foreground conversation available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the background-work interaction first
Before choosing concurrency primitives, decide what should happen when delegated work finishes. Should the result enter the conversation automatically? Should it trigger a new response? Can the user inspect running tasks, launch more work, or batch requests? Audience suggestions range from asyncio and JavaScript’s event loop to batching and smaller models. These can affect implementation or latency, but they do not by themselves define the interaction.
Bigio sketches a smaller contract: return a pending acknowledgment immediately, then expose check_tasks for later inspection. At this point the code is only an interface sketch. Returning a string that says work is pending does not make a blocking model request run in the background. The executor still needs to start the work and retain a handle to it.
The terminal also needs a different arrangement. A loop blocked on user input is awkward for displaying unsolicited results, especially if output appears in the middle of the user’s typing. Bigio separates input from processing with a socket server and a terminal client. The server listens on a host and port, accepts user input, and starts a message processor; the client connects and handles input. He corrects his initial description from WebSockets to sockets.
The agent uses AsyncOpenAI and awaits completions. Its tool handler creates tasks for multiple calls and awaits their results together. This allows network waits to overlap because those operations yield control to the event loop. CPU-heavy code that does not yield still occupies the execution thread; adding async does not make it parallel CPU work.
Concurrent work needs serialized conversation updates. If two results each start a generation against the same old history, they can produce conflicting continuations. A message queue provides a single place to accept user input and, when supported, completion events. The processor removes one message, runs a full turn, updates the history, and then proceeds to the next. A simple name-recall exchange checks that ordinary conversation still works before background delegation is added.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Parallel calls can still hold up the conversation
The async example first reproduces a normal weather round trip with a mock response of 67 and sunny. Bigio then adds a location parameter and randomizes temperatures between 50 and 80. The same function is copied into the synchronous version so that both implementations can answer requests for several cities. These are mock weather values; the comparison concerns scheduling.
A model can emit several tool calls in one response while the application still executes them one after another. With immediate-return functions, that distinction is easy to miss. Bigio adds time.sleep(1) to make it visible. He observes that five mock weather calls, each using time.sleep(1) in a sequential executor, take over five seconds.
Replacing the blocking sleep with await asyncio.sleep(1) allows the waits to overlap in the async executor. The live example returns the group together, without reporting an exact asynchronous runtime. Real network requests could occupy the same place as the artificial waits.
| Execution strategy | During tool work | Foreground conversation |
|---|---|---|
| Sequential calls | One call runs at a time | Waits |
| Concurrent calls, awaited together | I/O waits overlap | Still waits for the group |
| Background tasks with handles | Work continues independently | Can accept another turn |
Several o1 calls could run concurrently too. But if the foreground turn awaits all of them, the user still cannot continue that conversation until they finish. Shortening the aggregate wait and releasing the foreground turn are separate changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Return a task ID before the result
create_task gives background work an identity. It generates a short random ID, schedules the operation, stores the handle, and returns the ID to the model. A separate check_task function looks up that handle and reports whether the operation is still running or has a result. The initial tool call can now finish while the underlying work continues.
The following version keeps the workshop’s SF weather example and makes the pending state explicit:
python
import asyncio
import random
import secrets
_tasks: dict[str, asyncio.Task] = {}
async def get_weather(location: str):
await asyncio.sleep(10)
return {
"location": location,
"weather": f"{random.randint(50, 80)} and sunny",
}
async def create_task(description: str):
"""Start a weather task; use a location as its description."""
task_id = secrets.token_hex(4)
_tasks[task_id] = asyncio.create_task(get_weather(description))
return {"task_id": task_id, "status": "pending"}
async def check_task(task_id: str):
"""Check a previously created task without waiting for it."""
task = _tasks.get(task_id)
if task is None:
return {"task_id": task_id, "status": "unknown"}
if not task.done():
return {"task_id": task_id, "status": "pending"}
try:
result = task.result()
except Exception as error:
return {
"task_id": task_id,
"status": "failed",
"error": str(error),
}
return {"task_id": task_id, "status": "completed", "result": result}
The pending response acknowledges scheduled work, not a weather answer. The task handles must remain in a running event loop for the work to continue.
Tool registration is missed once more and then corrected. To make the behavior unambiguous, Bigio increases the artificial weather delay from five to ten seconds. He creates an SF task, exchanges greetings, checks its status, asks for a joke, and checks again. The completed mock result is 78 and sunny. The useful demonstration is that those intervening turns happen while the task is outstanding.
The next change replaces get_weather with a model call. Bigio requests a haiku as a task, then asks for a second one with a theme chosen by the model. Both can be inspected through the same task interface while the foreground chat remains available. He questions whether the unexpectedly quick result actually used o1, so the example establishes multiple background model tasks without establishing which model produced those particular poems.
Polling is only one delivery mechanism. The separate input and output architecture was intended to support pushing a completed result into the conversation without a new user message. Bigio identifies that as the next extension, but the demonstrated version still requires checking tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expose progress without hiding the loop
An audience question suggests representing agents as Python generators, composing nested agents with yield from, and yielding tool calls as they occur. Bigio agrees: instead of exposing only a final answer, an agent can emit steps, calls, and results throughout its execution. That would make the long nested call visible rather than leaving the terminal apparently idle.
The additional work is event handling. With several agents emitting events, a consumer must listen to them and associate each event with its originating agent. A flat collection of event-producing agents is possible; the workshop instead keeps one foreground manager responsible for starting and inspecting subordinate work. Generator-based progress reporting is recommended as an extension, but not implemented live.
Framework choice follows the same preference for visible mechanics. Bigio likes Swarm for prototyping because he knows it well, but often copies a small custom loop between projects to retain granular control and avoid unnecessary dependencies. Handoffs, for example, require special handling of particular tool results. He also likes the interface of Pydantic AI, while explicitly noting that he has not used it extensively.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Selecting from a larger tool library
A large function library presents a selection problem before it presents an execution problem. Bigio offers three approaches, each changing what the model needs to choose from:
- Group by responsibility. Give related functions to separate agents and invoke the relevant group for the task.
- Fine-tune flat selection. Keep functions available together when routing overhead is undesirable. Bigio reports fine-tuning GPT-3.5 for approximately 120 functions in an OpenAI project.
- Load functions dynamically. Use the input or conversation to retrieve likely relevant definitions, through embeddings or an initial function call that loads another set.
The last approach starts to resemble a handoff: a call changes the functions available for the next model request. The reported GPT-3.5 project illustrates a possible approach, but no accuracy metric or evaluation set is supplied.
The ensuing question asks whether reasoning models call tools inside their hidden reasoning. For the o1 API at the time of the workshop, Bigio says the exposed function calls occur after reasoning rather than within it. That is a historical description of the demonstrated API, not a general restriction on what post-training could enable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Route to the right functions with a handoff
The routing example divides functions into email and calendar groups. The email agent gets operations such as sending and checking email; the calendar agent gets event creation. A triage agent chooses the appropriate destination. Before adding this structure to a real application, Bigio recommends trying the complete function set on one agent and using evaluations to determine whether the extra routing is needed. His hypothetical larger library motivates the split; it is not evidence that a particular count requires multiple agents.
Swarm represents a transfer with a function that returns an Agent object. Register those transfer functions on triage, and the framework can replace the active agent when one is called:
python
from swarm import Agent
email_agent = Agent(
name="Email agent",
instructions="Handle email requests with the available email tools.",
)
calendar_agent = Agent(
name="Calendar agent",
instructions="Handle scheduling requests with the available calendar tools.",
)
def transfer_to_email():
"""Transfer email requests to the email agent."""
return email_agent
def transfer_to_calendar():
"""Transfer scheduling requests to the calendar agent."""
return calendar_agent
triage_agent = Agent(
name="Triage agent",
instructions="Transfer the user to the appropriate specialist.",
functions=[transfer_to_email, transfer_to_calendar],
)
The transfer functions are the routing interface; each specialist’s application functions can be attached to its own functions list.
A request to send an email first transfers the conversation to the email agent. Bigio then makes the request complete: send to Bob, use a taxes subject, and put a short reminder to do taxes in the body. With instructions to act when the necessary information is already present, the agent transfers and immediately invokes the demonstration’s send-email function. It feels like one operation to the user, although routing adds a model hop. The demonstration shows function dispatch, not independently confirmed delivery to a mailbox.
A request for a Streamlit- or Gradio-style one-command application wrapper receives no concrete implementation plan. On function counts, Bigio is tentative about hard API limits, then gives practical guidance: Bigio suggests roughly 10–20 available functions as a rule of thumb for reliability without extensive prompting. He explains that the earlier 120-function fine-tune served an unusually latency-sensitive application that needed flat selection. The decision is therefore about evaluated selection quality and acceptable routing latency, not a universal cutoff.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An agent that adds its own functions
An audience question about code-generating agents leads to an unplanned experiment: let the agent write a Python function, register it, and use it. Bigio starts a new Swarm agent and considers a self-handoff—returning the same agent after modifying its functions—to refresh the available interface. The proposed add_tool accepts a string containing a Python implementation.
The key operation is converting source text into a callable object. Bigio initially reaches for eval, then corrects it to exec, which can execute a function definition. The desired output is the newly defined function object so that it can be appended to the agent’s tool list. Merely running a script in a subprocess would not provide that in-process object. Cursor assistance and an audience implementation help work through the parser and registration code.
The first behavioral test is deliberately observable: create a tool that prints hello. When the generated tool merely returns text, Bigio asks for printing instead, then calls the revised tool and sees the output. He next asks the agent to make a calculator and tries a multiplication expression. The useful result is the cycle of generating, registering, and invoking a new callable; the spoken arithmetic is not clear enough to recover an exact calculation.
Executing model-generated Python inside the application process is an unsafe capability boundary. It grants generated code the process’s access rather than confining it to the apparent tool description. Bigio repeatedly labels the example dangerous and treats it as an experiment. Dynamic tool registration is a powerful mechanism, but this implementation does not provide an isolation or permission system for the code it accepts.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A function can also mean staying silent
Voice interfaces introduce another kind of action: deciding not to respond yet. A silence-based voice activity detector can signal that the user might be finished even when they have only paused mid-thought. Bigio proposes treating that signal as an eager candidate boundary, then letting the model choose a stay_silent function if the utterance still sounds unfinished.
For example, a user might pause after saying they have been thinking about something, before identifying what it is. The detector triggers, but the model selects silence so the user can continue. Bigio reports that this works well with an appropriate tool description; he does not demonstrate it live. The mechanism adds semantic judgment after the acoustic trigger rather than requiring every detected pause to produce speech.
A separate DevDay experiment uses XML-tagged scripts to direct how the Realtime model speaks. Bigio describes asking the model to read a script according to the tags, then tries to play an example. The room cannot hear the playback, so no shared-audio result is established. He also distinguishes this from function calling: it is an observed instruction-following behavior, not an explicitly trained XML speech-control interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When a remembered fact changes
Returning to memory, an audience question asks how to enforce consistency and represent more structured information. The append-only list can retain facts, but it cannot by itself distinguish a contradiction from a legitimate update. Bigio proposes retrieving semantically similar memories before storing a new one, then asking a model whether the new information updates or contradicts an existing entry.
Consider the project example: an earlier memory says the project is not ready; a later message says it is done. Both statements can accurately describe different moments. Bigio proposes timestamping the entries and adding an explicit link from the older memory to its successor. The older record stays available, while the relationship identifies which state is current.
With that link, retrieval can return either the latest state or the whole update chain. A question based on an old assumption—such as how long the project remains delayed—can be answered using the completion update, while a historical question can still use the earlier record. This is an off-the-cuff design proposal, not a feature added to the live memory implementation.
Preserve the old fact and link its update
Constructed example: Record IDs, JSON field names, and dates are teaching values. The project-state change and proposed successor-link mechanism come from the workshop.
The project is done now.
Operation: Proposed storage operation: retrieve the related project memory, classify the message as an update, preserve the earlier record, and link it to the new record.
Earlier project memory
{"id":"m1","text":"The project is not ready yet.","recorded_at":"2025-02-01","superseded_by":null}{"id":"m1","text":"The project is not ready yet.","recorded_at":"2025-02-01","superseded_by":"m2"}Completion update
Not present
{"id":"m2","text":"The project is done now.","recorded_at":"2025-02-10","superseded_by":null}Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carrying the same contract into a phone conversation
The final demonstration opens the OpenAI Realtime API with Twilio Quickstart. Its setup interface tracks the account, phone number, and local web server with a live checklist. Bigio explains that repeated Twilio and ngrok setup had been tedious; the interface is intended to make progress visible and reduce trips through the Twilio console. His local application is already running, with sensitive settings masked.
He creates a whimsical universe-answers function schema in the Realtime Playground, pastes it into the tool configuration, saves the changes, and calls the phone number. The first attempts do not produce a working conversation. The interface nevertheless illustrates two distinct ways to supply tools:
- Schema-only tools: the interface displays a requested call, and a person can enter a mock response.
- Backend handlers: application code executes the function and returns its actual output.
Bigio points to a backend weather handler as the implemented example and suggests that delegation to o1 could be added through the same handler mechanism.
The consequential difference from the earlier chat loop is the pending-call interaction. Bigio says the Realtime API permits a function call to remain outstanding while the user continues talking to the model. He attributes that behavior to specific training: a voice conversation cannot simply stop whenever an external operation is slow. It is the background-task requirement from the Python example expressed in a realtime conversation interface.
A final retry also fails, so the workshop ends without a successful end-to-end phone call. Bigio then shares the experimental self-writing tool-agent code in Slack, repeats the unsafe-code warning, and offers to share the repository and slides. The shared code returns to the most expansive use of the execution contract explored here: application code can interpret a requested operation as a change to the functions the model will see on its next turn.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The educational orchestration framework used in the workshop. Its README now directs production users to the Agents SDK.
A phone-assistant example connecting Twilio and OpenAI through a WebSocket backend, with setup instructions and mock tool responses. The README cautions against deploying without security review.
The original announcement of function calling for GPT-4 and GPT-3.5 Turbo, including the historical API parameters.
Research on teaching language models to select and use external APIs with only a few demonstrations per tool.
OpenAI's 2021 account of training GPT-3 to browse, collect sources, and answer questions using human feedback.
Official documentation for the Python agent framework discussed during the framework Q&A.
Updates since the talk
Current examples for defining functions, executing requested calls, returning results, and designing manageable tool interfaces.
Configuration for silence-based and semantic turn detection, including controls for how eagerly the system treats an utterance as complete.
Read the complete timestamped transcript
- 0:00
Cool. Okay, let's get on with it. So hi, everyone. My name is Ilan. Uh, I'm on the developer experience team at OpenAI. Um, unfortunately, I can't be there in person, as much as I would love to.
- 0:12
I'm in a wedding in Costa Rica, um, which is happening later today, so I just wanted to take this opportunity to just talk through, um, one of my favorite concepts in maybe all of, like, AI and language models.
- 0:25
So, uh, title of this talk is Function Calling is All You Need. It's a talk workshop. There's gonna be, um, a lot of coding. Please save your questions... No, I'm kidding.
- 0:34
Like, just interrupt at any point. Uh, we have a Slack, send them here. Um, yeah, just send them at any point. Uh, and if you wanna, like, unmute yourself and, or raise your hand, I'll call you off, like, as we go.
- 0:46
The idea is to keep this super, super dynamic, um, since we have a bit of time. Um, I'll be fielding a lot of, like, questions and requests and then trying to be coding as much as possible.
- 0:57
So yeah, this talk is gonna be some lecturing, uh, a lot of coding from scratch, and then some debugging, hopefully not a lot. Uh, so this is a little bit of what the, uh, workshop is gonna look like.
- 1:09
We're gonna go over a little brief history of the toolformers, of, of, I'm sorry, of function calling, um, then do a little crash course on function calling, um, talk about just agents, how they're just loops, how RAG workflows and more are just function calls, delegation, and asynchrony, couple random things I've found, and then we are gonna do
- 1:28
a Q&A. Um, if you just wanna see, like, the meat of it, this is pretty much it. We're gonna do, like, this is everything I wanna talk about, uh, and everything that we're gonna implement, so this is, uh, just a little bit upfront.
- 1:45
Great. So a little history. Um, if we look at the abstractions and sort of patterns that we managed to do with language models, it started as text completion, right?
- 1:55
Like, the original GPT, uh, and the GPT-2 and the GPT-3 were all just base models, where you gave them some, some input text, um, and then they would just continue the sentence.
- 2:05
Um, this was, at the time, really, really, like, interesting. This was the first time we were able to do, like, very, like, uh, English-sounding, like, real-sounding language. Um, but getting it to follow instructions was pretty hard.
- 2:16
So if any of you were testing this back then, you might remember how setting up a chatbot was, um, non-trivial, right? You had to, like, get it to answer questions, but if you just say, like, you know, "What is the best way to, uh, get to the, like, park?"
- 2:31
or something, it would continue, like, like, "What is the best way to get to the park? Um, that is what Sally said yesterday," right? And you wouldn't actually get a, a response.
- 2:41
Um, so you had to, like, structure it in a way where you would say, like, "This is a question. This is the answer." Question, answer, few shot, and then give it.
- 2:48
Um, then they introduced, uh, I think this was actually us, we introduced, uh, function ca- um, instruction following with, uh, InstructGPT. Now you could give it some input, uh, and it would actually do what you're asking as opposed to just completing.
- 3:01
Um, finally, we started to introduce this notion of, like, users and assistants and roles, um, and this was all, all done through, through post-training, where you actually gain these personas.
- 3:11
Um, and then finally, uh, we, we eventually landed on this, like, you can give it additional tools, um, in order to do, like, external, um, any, like, interact with external states.
- 3:22
So this is what the previous playground used to look like. But as you can see, there's no chat. This is just, like, a window, and it'll complete. Now, looking over at, like, the original papers, which was pretty interesting, um, one of the first times that we actually started to do this, like, function calling was through this WebGPT,
- 3:41
um, which was this, uh, version of GPT-3 that we trained to be able to-
- 3:45
Okay, um, Ilan, I have to cut in here a little bit. We got the Zoom working. Yay. [laughs]
- 3:51
Sweet.
- 3:52
Um, all right. We're gonna cut over in the audio, so can everyone hear Ilan when he speaks? Uh, Ilan, say something.
- 3:58
Hello. Hi, everyone. Hello, hello. How's it going?
- 4:05
Okay, so, so we're gonna cut over a little bit in terms of, like, um, people having their own personal audio situation going on. So if you mute your machine, you should be able to hear it on the room, uh, audio, whatever you wanna do.
- 4:19
But you can also obviously connect. But, uh, yeah, now we have him on the big screen.
- 4:24
Sweet.
- 4:24
So okay, yeah. Uh, let's, let's keep going for a bit. I'll, I'll cut in again if there's audio issues. [laughs]
- 4:29
Sounds great.
- 4:29
Or maybe you wanna say some test words. [laughs]
- 4:32
Uh, you're all great.
- 4:33
Okay.
- 4:33
Thank you for coming.
- 4:35
Can we bump up the audio? Where is the guy? [laughs]
- 4:43
I think it's not a conference if you don't have AV issues.
- 4:46
Okay. Uh, it's, it's still too soft. We can't, we can't hear you. Um, I'm gonna try and bump up the audio. I'm so sorry. [laughs]
- 4:53
I'll put up a more interesting slide in the meantime. You guys can look at this while we figure this out.
- 4:58
Where's, where's the fucking video guys? Hey, um, Ilan.
- 5:22
So anybody got, uh, good jokes? Maybe.
- 5:30
I always ask OpenAI for my jokes.
- 5:33
It's always the same one. [laughs] I think it's, uh, what was it? Why did the...
- 5:41
No, I had it, I had it here. Wait, wait, wait, wait, wait, wait. There we go.
- 5:45
I don't trust-
- 5:45
Why can't you trust an atom?
- 5:47
Yes.
- 5:47
Yeah, because they make up everything. [laughs] These are all real, by the way. And for, for those who, like, have a keen eye, the first one is, like, actually from the GPT-2 paper.
- 5:57
Um, we, like, gave the model- This, like, description of, like, unicorns in the Andes Mountains, and that was, like, the big, like, first time that it was, like, doing multi-paragraph completion, like continuations that referenced earlier parts of the conversation.
- 6:16
It's cool stuff. It's cool history. Can you guys hear me? [laughs]
- 6:22
I feel like some people are having a great time.
- 6:24
Yes, we can.
- 6:25
Some people don't know what's going on. Can, like ... I don't know. Am I good to k- keep going, or should I keep waiting?
- 6:31
Uh, I think, I think you're good.
- 6:33
Okay. Great. Great, great. I do think this is the vibe of the whole talk, by the way. Th- there's n- it doesn't get more structured from here. Um, okay, so as I was saying, uh, we did, like, this WebGPT paper.
- 6:46
Um, essentially we trained a GPT-3 version of the model, uh, or, like a GPT-3 model to be able to use the, like, this very specific set of functions, um, to do web search, and this was, like, back in 2021, so really we had, like, WebGPT a long time ago.
- 7:01
Um, but this is, like, one of the first times or maybe the first time that, like, we were having, like ... It's not just generating text, but it's generating actions, and then we're parsing those actions and then introducing it back into context so it can use the responses itself.
- 7:14
Um, and how we trained it, there, um, there, there's, like, these, these, like, you know, clever ways of, like, we, we essentially gave people, uh, an interface and let them do the searching.
- 7:25
Um, and then I think we took it, like, Reddit, um, yeah, explain like I'm [REDACTED:age], um, and then just, like, had people complete tasks, and they could use these commands.
- 7:35
And so we taught the model, and this was GPT-3, right, um, how to essentially imitate users' behavior and then produce responses that were preferred. Um, and, and this was pretty cool.
- 7:45
Like, this is how you, you, we, like, start to saw, um, to see, like, this, uh, this use of, like, structured, like, actions essentially. So but this was very specific, right?
- 7:55
We were training, like, very specific, um, tools. So then you might be familiar with this paper. This was from Meta at the time, um, where they essentially had a way to teach the models how to use any tools.
- 8:09
Um, and they taught, uh, like, they, they used a few tools. I think it was, like, QA, calculator, um, what was this? Like, translation. Um, couple other, couple other tools.
- 8:20
But, um, it was actually a pretty clever way where they, like, looked at the log probs at each spot to, like, see where it was, like, best suited to, like, retroactively put in a function call given some, like, completion.
- 8:34
Um, so here we can see a few examples. Like, um, essentially it's like if you have a calculator call, if you insert a calculator call that, like, you know, it'll, it'll insert the, um, the actual call to the calculator, which then, you know, you're pretty familiar at this point, will get the answer.
- 8:51
If you insert it in the right spot in the sentence, it actually reduces the perplexity of the sentence. Um, and so they didn't actually have a lot of human labeled examples, or I think it was just a few, but it was, uh, really cool because it was this way of, like, um, it could learn to use, uh,
- 9:08
any of these tools through this, like, crazy, like, log probs technique. Um, and it was, uh, I was pretty excited when I saw this paper. Now, this is, like, how it learns to use any of these tools.
- 9:21
Um, but then finally, uh, in June of 2023, O- uh, OpenAI launched just general function calling, where we essentially, like, pre-trained it to be able to use these tools, or act- actually post-trained it to be able to use, uh, tools.
- 9:32
So now you don't actually have to, like, give it ... Like, you can give it examples, but, uh, we just showed it, like, this, like, syntax in with functions that we still use today, and it's just able to call functions.
- 9:43
So this is a brief history of, of function calling. Um, and my, I guess, my argument is this is really most of what you need for all the exciting stuff that's happening today.
- 9:55
Um, there's, there's obviously, like, additional, like, systems you can use and, like, um, more post-training you can do, but fundamentally, like, functions are so, so, so powerful, and we're gonna look at a few cases today.
- 10:07
Um, I'm gonna try to keep an eye on questions, but, um,
- 10:13
yeah. Okay.
- 10:15
I think you're good so far.
- 10:17
Cool, cool. So let's do a super quick crash course on function calling. Uh, two main purposes, and I'm ripping a lot of this from the docs. So, um, they're the fetching data, right?
- 10:28
Reading APIs, retrieval, memory, or taking action. Any APIs you can use to write. Managing application state, which is actually pretty overloaded. That can be, like, UI, front end, back end, whatever you want.
- 10:39
And then workflow actions, which is, um, any, like, multi-step processes or even, like, meta actions, like switching its own prompt, or, like, loading in different tools, or, like, handing off a conversation, right?
- 10:53
Um, so this is a- also a diagram straight from the docs, but, um, I'll quickly brush past this. I'm gonna assume, like, most people have at least seen this, but here it is.
- 11:02
It's, uh, you, you essentially tell the model which functions you, you want it to be able to use. Um, and you also provide, like, whatever the user input is.
- 11:11
Um, the fu- the model tells you what it wants to do with that function, but it doesn't actually do it. This is one of the, like, a big sticking point of function calling.
- 11:18
It doesn't actually use the function itself. It tells you, like, it tells you the intent of what it wants to do with the function. You are then responsible for parsing that, executing the code, doing whatever you want with it, and then providing the result back to the model.
- 11:32
And then the model can use that respo- like, result in a, in the generation. Uh, take a look at a quick question here.
- 11:40
Oh, no. It's just, it's just winks. Okay.
- 11:44
Cool. Um, these are just a few, uh, best practices. This is all taken from the docs as well. Um, you wanna write clear functions. An important one is you gotta apply software engineering best practices when you write these functions.
- 11:56
So, um, you know what? Maybe I'll pull up the docs for this.
- 12:00
Um- So this is a lot of big text, but essentially, um,
- 12:11
th- this is a lot of value here. I tried to pack as much, like, useful information here as, as I could, so I'm gonna quickly go over it, right?
- 12:17
Um, you gotta explain the purpose of each parameter, use a system prompt, and include examples. That's, you know, pretty, pretty, like, n- non-controversial. Um, software engineering best practices is a little bit, uh, more interesting, right?
- 12:31
You gotta make functions obvious and intuitive, and they gotta follow the function of least principle. Um, like if you give this to a person and they don't know how to use it, then the model might not either, right?
- 12:42
You know, models are getting smarter than us, but still you gotta make it, uh, gotta make it easy. Um, also you gotta use enums and obj- object structure to make sure that, like, you are not letting the model make invalid calls, right?
- 12:55
Um, like here you, you have this like toggle, like, uh, um, function that takes in like two, two Boolean params and like, obviously, like this is pretty wrong, but, um, there's actually many, many more like subtle cases where, um, like the-- you can, like you're letting it represent invalid states.
- 13:17
Um, [laughs] okay. Still no questions. It sucks. Okay. Um, great. And then this is really, uh-
- 13:27
There actually... Sorry, there are a couple of questions in the Slack.
- 13:30
Okay.
- 13:30
If I saw it rather than
- 13:32
I see. I see. Um, here, maybe let me see if I can pull up the Slack.
- 13:41
Okie dokie. Da, da, da, da, da, da, da.
- 13:45
Between functions and tools in my opinion. Hey, Sam. Yes. Um, I, I think we've all been kind of like gravitating toward functions and tools as like the two main ways.
- 13:57
Originally it was just functions. Tools was like later we renamed it. Um, I think now, uh, and the way that I tried to specify it in the docs is functions are like the, like raw function calling, right?
- 14:10
Like you provide an interface and you are responsible for executing the code. Um, tools, and this is sort of how we treat it in our API, is a super set of functions.
- 14:19
Tools include functions, but it also includes things like, um, code interpreter or file search or any of these like, um, like hosted solutions.
- 14:32
I'd say this is not be like end all be all definition, but this is a definition that we've adopted, right? There's like tools which are like hosted tools, uh, including functions and then functions is a subset.
- 14:42
Cool. Maybe we can just jump into it. Um, if people have questions on this, happy to, happy to field them as well, but I kind of wanna get coding.
- 14:50
Okay. Uh, when you start to approach dozens or hundreds of functions, what technically should we apply in order to effectively tool call?
- 14:58
Permissions is one technique. Okay. Interesting. And then question two, when you require one tool to provide inputs to another, I have seen tools become layered. How should a reasoner hard code?
- 15:08
Great. Okay. These are actually great questions and, um, I think to, to answer them, I might, I might like do a little hack and like, uh, use some existing, uh, code.
- 15:20
I'm gonna use Swarm for a little bit of this, um, because it does some nice function calling. Actually, no. You know what? I'm gonna get to these func- uh, in a second.
- 15:28
Um, let's just go straight into the, uh, and as always, we should start from the docs.
- 15:38
Uh, cool. So-
- 15:40
Uh, so this is where I always step in. Can you zoom in on your screen?
- 15:45
Yeah.
- 15:45
Every single screen page.
- 15:47
Yeah.
- 15:47
Bigger.
- 15:49
Cool.
- 15:51
Yeah. There. Yep.
- 15:54
Sorry.
- 15:54
This is the thing I always do.
- 15:56
Yeah, no, this, this is a good call out. Uh, and then I think the
- 16:02
terminal here should be good. Okay. Sweet. Uh, we have, we have a function. Great. Um, and then ...
- 16:17
And just run it. And we have, uh, the function call right. Now we're not handling it yet. So this is where like I might skip ahead a little bit, um, and start doing some of the like agentic stuff.
- 16:34
But first, uh, first off, uh, we, we gotta have a loop, right? Like, you know, you, you, I can sort of hear myself. Um, I think Sohum. There we go.
- 16:45
Perfect. Okay. So, um, I think the idea here is like,
- 16:51
let's make a very, very, very simple like, uh, input loop. Like let's do, uh, you know, while-
- 16:58
Can you also switch to light mode?
- 17:01
Light mode?
- 17:02
Yeah.
- 17:04
Jesus. Okay.
- 17:05
Yeah.
- 17:06
Sure.
- 17:07
I'm, I'm just very experienced with this. There we go.
- 17:12
What's the least painful? This is good, right? This works?
- 17:18
Yeah.
- 17:20
Ugh, my poor eyes. Um, actually maybe I, I, I can't quite see the room, but show of hands, like who has used or implemented function calling in the past?
- 17:31
And someone's gonna have to gauge this for me.
- 17:37
Okay. It's everyone. Great. I'm gonna skip ahead a little. Um-
- 17:41
There's like 10, 20 people, 20% that hasn't done it.
- 17:46
Okay. 10, 20%. Um, the important part that you have to know is you can define the function, um, schema, right? Then the tool will specify what you want, and then in this case, you know, if I have like a
- 18:01
get weather tool-- Actually, I'm gonna grab this from the docs as well. When in doubt, just go to the docs, you know? It's always good. So-
- 18:12
Step one. No, this is all node. Yeah, and this will be useful for later too. Um,
- 18:20
step one, call the function. Step two, execute your code,
- 18:27
right? So here what we're doing is we're taking, we're parsing out what the, what the function told us. We're parsing out the args, and then we're calling this get weather function, which we don't have yet.
- 18:36
Um, but conveniently it's up here, right? So have it up here,
- 18:42
get weather requests. And then the last step is we provide the result back to the model. Um,
- 18:53
uh, provide the result back to the model and then ask for our completion, right? So just in order, specify the tools, call it the first time, get the tool calls, parse them out, call the function, append, and do that, right?
- 19:05
So if we do that, and we just add maybe a, like, add... Nah, I'm gonna do this by hand.
- 19:14
Why not? Yeah, so I'll print the completion here, sort it by hand, and then print the last completion.
- 19:24
Then what we can see is we'll get the first one that includes a tool call. It'll call the actual temperature, uh, call the weather API, and then we'll get a response.
- 19:38
And then it says the current temperature in Paris is something. I can't see it because of Zoom. Um, cool. So this is like the very, very basic setup for a function calling.
- 19:49
Let's take a step forward. So this is an agent, a very, very, very basic implementation of an agent, um, that I'm gonna go through really, really quickly because we're gonna start using it.
- 19:58
This is very familiar to what you'll see in Swarm or any of the o- other, like, like, basic frameworks. Um, but the idea here is in-- uh, as you can see, like, in the original one, essentially, like, when it had a tool call, I wanted to provide it back.
- 20:15
So what I do here is while, like, just keep looping, and this is, like, the very famous, like, agents are a loop. This is, this is that loop. Um,
- 20:25
specify the tools, call the model, get the message, print it out, handle the tool calls, append it. And once we have no more tool b- calls, break. This is the whole loop.
- 20:39
I called it run full turn in my head. One turn is just like, you let the model do everything. Um, and then we have this, like, execute tool call, yada, yada, whatever.
- 20:49
Um, so now we can use this. So
- 20:59
agents.py. Did I export it? There we go.
- 21:25
I love it. Okay, so now we're just gonna specify one.
- 21:31
Um, you know what? Yeah. We w- we, um-- And we'll, we'll do it, we'll do it with a, with a simple loop.
- 21:42
So we have this, that, and we have this. Okay. So now we can just do agents.run_full_turn. The one other thing that I'm adding here that I didn't show is, um, this simple utils,
- 21:56
uh, that defines this very, very, very useful function. So functions to schema essentially takes in a raw Python object function and then provides it into the, uh, like, correct schema, um, so that you can just define functions directly.
- 22:12
And this is the same thing that we have in Swarm, and there's in a few other frameworks now as well. Um, so as an example, we can do like, you know, get weather, and I'm just going to like, you know, return twenty.
- 22:26
Like, degrees Celsius. Great. I'll do that here.
- 22:40
And then I'll have my messages, imprint messages.
- 22:48
Is this emergency reference? I think it is.
- 22:58
What did I call it? Oh, oh, oh, oh, oh.
- 23:07
I see. This is not part of the agent class.
- 23:12
Yeah. Boom. So we called the weather, printed it out, uh, gave us a nice little completion.
- 23:31
Um, this is essentially what we wanna see. If I say, um... And then I can just keep, keep going, right? So we, we, we have this basic, um, basic loop.
- 23:42
Um, now let's get back to the presentation real quick. That was like a very, very immediate crash course. Now let's, let's get interesting. Um, and, and just for convenience, I'm gonna like pivot to the Swarm implementation.
- 23:56
Um, the main reason being it's pretty much the same, um, except we have like some convenient, uh, like looping tools. So, uh,
- 24:06
Swarm, Swarm imports agents, and then There we go.
- 24:23
So now we can do a simple agent and run a demo loop, and let's see this work.
- 24:44
Cool. Um, we have this very basic setup. Let's do everything now. So, um,
- 24:51
another show of hands, how-- who here has implemented, uh, RAG?
- 24:57
Okay. What about memory? Okay, fewer. And then what about like, you know, multi, multi-step things and workflows?
- 25:07
Cool. Um, how about this? There's a lot of stuff we could talk about, but I wanna keep this valuable to you all. Out of this list, can you like just type in the Slack what you wanna, what you wanna see?
- 25:20
And I can just change the order in which we'll cover this, um, because they're pretty, they're pretty interchangeable. We can, we can build up. Um, but essentially there's, there's more interesting things
- 25:30
later on, but I wanna make sure we, we can build up to them. So just, uh, if you can pull up the Slack and just dump in, you know, what you're interested in seeing.
- 25:48
Okay. Lots of memory I see. Um, function generation from the docs, delegation async. Okay.
- 26:00
Good old delegation. [laughs] And random cool stuff. I will get to the random cool stuff. Um,
- 26:10
okey-dokey. Back to light mode. Um, yeah, let's start with like a very, very basic form of memory, right? Um,
- 26:20
how would we-- how do we do this? Let's see.
- 26:24
Honestly, like we can have, um, just a list, right? Memory can just be this list, and I'm gonna implement this similar to how it's done in ChatGPT.
- 26:35
Um, but ju-just to show like I think the whole point of this talk is like-- or th-this workshop is like doing things from first, first principles and just really removing the complexity.
- 26:45
I think there's like a lot of like, not fake complexity, but like added complexity on things that like doesn't really need to be there a lot of the time.
- 26:51
Like concepts are a lot more simple, and it's all about function calling. So, uh, we can do, you know, add like add to memory, [laughs] append it, and then get memory.
- 27:05
Uh, th- like this is super, super simple. Um,
- 27:11
is this, is this good enough? Let's see. Maybe. Maybe. And so what we can do is
- 27:19
give it the tools. Let's see. And then when, when would we wanna use them? Let's say you like-- I can just say, so this comment is gonna be used as the, uh, string, uh, as the, as the description in the function.
- 27:32
So I can say like, you know, like when the user tells you something like factual
- 27:42
about themselves, their life or... Man, I can't see anything. Okay. Or their
- 27:56
references, call this function. Um, memory. Um, and we'll add a couple more cool things like expiration, which is one that I've kind of wanted to add for a while.
- 28:15
So, you know, for now, let's say false.
- 28:21
Um, memory.append, uh, bank. And then we have, I guess what we'd call it, uh, you know, memory text.
- 28:41
And we can say, you know, keep the memory text short size. Great. Cursor knows what I want.
- 28:48
And then we can just return maybe like...
- 28:57
Like this is a super, super, super naive implementation. Um, but now, uh, when we start off, I guess we could even start off with a...
- 29:13
I mean, this is gonna be kind of hacky, but I can just like, um, in your first turn, always call get memory. This is not, uh, great. It's just because of the demo loop, but maybe I'll break, I'll break the demo loop out so we can actually do this by hand.
- 29:30
But so how, how could we prove this? Um,
- 29:39
maybe let's say like, you know, write this
- 29:43
memory bank or like keep this memory bank in a local file, JSON file. Read it in at
- 29:56
beginning and write it out at every case.
- 30:08
Cool, cool, cool, cool. Okay, I trust this. Shall we test it out? So, [echoing]
- 30:15
wait, someone mute themselves. Uh, da, da, da, da, da. Okay, we got the memory. Memory. Can any-- Can anyone see any bugs? [laughs]
- 30:26
'Cause we're about to test this out. So we got the loop. It's gonna call it- Um, and maybe like let's--
- 30:34
memory. Okay. Sure. Yeah, let's try it. Let's try it out, see what happens.
- 30:44
So hi. [laughs] Did I not give it the functions?
- 31:00
Is it hallucinating this? Oh, yes, because I think we called it functions.
- 31:10
Hi. There we go. Okay, so called get memory. There's nothing there. I can just say like, um, I am [REDACTED:physical_attribute] despite what people think.
- 31:26
Uh, cool. Uh, so now let's just check, right? It should have written this out. There we go. So now we have it in the memory bank. So now I can actually, uh, end this, right?
- 31:36
And then be like, you know, uh, how tall am I?
- 31:43
Ta-da, we've implemented memory, right? There's a-- Yes, I'll take a clap. I saw L-L-Luis, you're like my proxy for the-- Luis, uh, Luis Costa. You're my proxy for the audience.
- 31:54
You're like the only person I can really see. So please don't, uh, turn off your camera.
- 31:59
Um, amazing. Now we can do more interesting things, right? We can, uh, if we want, do a little bit of like smart querying, uh, where instead of just like loading in all of the memory, um, we can like do a little bit of like retrieval, uh, to load in the right ones.
- 32:17
Um, and use like semantic similarity or use some kind of search. Um,
- 32:23
I, I could try to implement that. Um, that might take a little bit longer, but not that long. But I do wanna pause here. So like given this, and like this is gonna be the style of things that we do, like what, um, what do we wanna see next?
- 32:36
I could just keep going with this example. I can pivot. Uh, it could be fun just to keep building on this, see how far we can get.
- 32:43
Let's see. Uh, delegation async, have it chat and work in the background. We'll get there. We'll get there. I got that working this morning on Python because I f-- didn't wanna switch to, to Node yet.
- 33:00
Uh, delegation async. Okay. Let's get into delegation then. So there's a few different ways we can do this. I'm actually gonna leave the memory, and we're just gonna keep building th-on this, on this agent.
- 33:14
Uh, and I am using the Swarm agent just because I didn't wanna debug the one that I implemented. But like if we actually look at what this is, um, like
- 33:23
it is very, very, very simple. And then the like run demo loop itself is like, uh, just printing out messages.
- 33:34
Um, and what it's doing is like appending client.run, whatever. And like if we look at this client.run, um, if it doesn't stream, it essentially does exactly what we did before.
- 33:44
Like keeps looping, get the completion, throw the messages in,
- 33:50
uh, append them, handle responses, et cetera. There's a couple more things around context and handoffs that we like don't, don't really have to look at today. Um, but we can.
- 34:01
So cool, we have memory. Let's do delegation. Um,
- 34:08
so there's a couple ways we can do this, right? Um, there's--
- 34:12
If you think of like functions and agents and, and everything, you can, um-- Maybe I'll skip to this slide.
- 34:20
Skip, skip, skip, skip, skip. These are like a few of the forms of like agents and delegation that people might be familiar with. We have handoffs, which is like the Swarm style.
- 34:28
You take a conversation and fully swap it to a different agent. Um, and what that means is just like replacing the system prompt, replacing the tools. Um, you can have nested calls, which are the easiest to implement and like often somewhat overlooked.
- 34:42
Um, and then you can have manager tasks, that's more async. Uh, we will get to that today. So let's do a very basic one, right? Let's say I wanna do like, um,
- 34:58
maybe let's give it a chance to like call a bigger model to do a harder task, right? So we can say like, you know, uh, it can delegate to smarter model.
- 35:12
Uh, and I'm gonna give like the task description.
- 35:21
Um, [laughs] so I'm laughing because this is saying like, it's just gonna make it smarter by telling it to be smarter. That's not-- Despite how well that would work, usually we're not gonna do that.
- 35:34
So we're-- we can just make an API request directly. Actually, let's just do that. Let's just do that. So, um, let's do from openai client. And here we could do client.ChatCompletion.create.
- 35:50
Let's call o1. Um, I won't provide a system message. I think that's okay. Content is the description. Look at that. And we did it. So now, um...
- 36:10
Man, I love Cursor. Okay. Did everyone catch all of that, by the way? All we did was like implement this, this function that calls OpenAI API, and then here.
- 36:20
So now I can say like, uh, so now I can add a bit of a description here. I'm like, like, uh, if, you know, if the user asks you to like, um, I don't know, do something
- 36:38
Like, that seems difficult or says it's hard, use this instead of
- 36:47
trying, trying it yourself. Uh, and it infers, you know, how to use this based on, like, the fact that I called it test description. It's, you know. So let's give this a shot.
- 36:58
So hi. Um, let's see. Uh, you know, give me a poem about,
- 37:08
I don't know, dogs and the Earth where each
- 37:15
other word starts with the next letter the alphabet. It might just try it, because I know GPT-4 can do a variation of this, but, um... Oh, it's giving it a shot.
- 37:27
Okay. Let's see if it gets it right.
- 37:34
Okay, I don't have, uh, I don't have the patience. Let's say
- 37:46
answer briefly, one sentence max. Okay, great. And then
- 37:57
to, you know, uh, write a haiku. There we go. It's shorts. Where each other word starts with the next letter of the alphabet, and ends with the previous letter of the alphabet.
- 38:18
I don't know. You guys want to try this while this does it?
- 38:27
Is it around? Did I lose, did I lose something? Let's see.
- 38:33
Hi. Okay. You there? Cool. Um, did I not copy it? Oh, no.
- 38:48
Okay. Uh, give me a haiku. Now make sure each word starts with
- 39:03
letter of the alphabet starting with A, and each word ends with the previous letter. But this is hard. [laughs]
- 39:22
Okay, so it should be making the function call.
- 39:28
Oh, I see. I see. I see. Okay, so I'm only printing the function call when it returns, so that's probably why we're waiting so long here. Um, but this is actually a really good, uh, example of like, okay, we are doing, um, this like task delegation technically, and like it is happening in the background and I'm gonna
- 39:44
give this a, a sec to figure this out. Um, but like this is obviously a bad experience, right? Like you don't wanna be waiting here. You essentially wanna keep doing other stuff.
- 39:55
So... Wow, it's still not, still not back. We'll, we'll, we'll let, we'll let it figure it out. Um,
- 40:03
so maybe let's skip straight to async. Um, we're actually got like both more and less time than I thought, so [laughs]
- 40:12
I'm gonna let this keep churning away. Um, now let's think about this for a sec, right? If we wanna do something async, it means that like what do we wanna happen?
- 40:21
There we go. So call the model, yada, yada, yada. Buzz breezy whispered. Is this correct? I don't know.
- 40:32
Wow. Yeah, this is sort of correct. What did it say? Note. [laughs]
- 40:42
Great. Great. So it, it did something, right? Um, and it did something that 4o probably could not have, which is great. That's delegation, but we were just sitting there waiting for it.
- 40:52
Um, so let's try doing this async. Now
- 40:58
I, I, I, I actually want everyone to like kind of just stop and think like how would you implement this, um, in terms of like what behavior do you want?
- 41:06
And maybe I, I wanna see people like drop this in the Slack. Just like take a couple minutes, um, and like drop in the Slack like a proposal for how you would wanna do this.
- 41:19
And then I might just pick one or we'll like talk about them.
- 41:32
Let's call o3. Yeah, not, not yet. Not yet. Yes, import asyncio definitely feels like, uh, important. But I guess the questions are like, you know, when we delegate something,
- 41:45
uh, obviously we wanna have this happening in the background. Um, do we want-- like when it finishes, do we want to be... Like do we want it to be injected into the conversation?
- 41:55
Do we want it to give us a response? Um, how many-- like do we wanna be able to like
- 42:01
interact with tasks that are running? Um, like do we wanna be able to batch stuff?
- 42:08
I am not setting up a Kafka cluster. [laughs]
- 42:16
I'm gonna give people maybe a couple more, couple more minutes to just like dump some ideas here. Um, batch calls. Uh, yeah, I guess ba- but how, how would, can batch calls work?
- 42:28
Maybe Stephanie, if you wanna add some like detail there. Just keep working until it generates a stop word.
- 42:34
Right. Okay. Yeah, this is, this is a good idea. JS and setTimeout. Yeah, switching to JS is always a good option because you already have the event loop implemented.
- 42:46
Smaller model, smaller model. Okay, you guys are all suggesting some good ideas. Um,
- 42:52
but we can actually go simpler, right? Like, like, essentially what we would want, and I can implement like the basic interface of this. Where is my code? Um, is like instead of actually doing this, right, I can like
- 43:10
return delegated, right? Response pending. Um, and then later I can say like, you know, check tasks or something.
- 43:31
So this, this pattern is actually one I, I quite like a lot, where you, um, you call a model. It's a, a, a function. It's non-blocking, and then later you can, you can check up on them.
- 43:42
Now, the thing is, Python i- is like single-threaded, and, uh, async is real. Uh, it's just a little bit tricky. Um, but let's, uh, let's give it a shot.
- 43:54
So I do have some async emergency reference. Now, here's the, here's the, um, thing to notice. When something-- like, for this to work correctly,
- 44:06
we have this loop that asks for our input and blocks on our input, um, and like is displaying that on the screen, and we don't wanna have that be displayed while it's also, like, injecting messages.
- 44:18
Um, so we essentially wanna separate out, like, where you give user input and where actually interesting stuff happens. So that's what I've done in this, in the basics of this async folder, so let's take a look at that.
- 44:35
Cool. Um, it looks complicated. It's not that-- really that bad. So, um, I'm using WebSockets. Essentially what I'm doing-- or Socket, sorry. Essentially what I'm doing is, um,
- 44:49
the-- I have a server, and all it does is, like,
- 44:53
has a handle user input function, uh, and a start message processor. Um, th-these aren't super important. We can take a look at them in a second. But essentially it just, like, waits, uh, on a specific, like, uh, port and, and host.
- 45:07
Um, and then I have a client that connects to that, and it just does while true loops and let me, lets me enter in. Right? So if we look at what that looks like,
- 45:17
I have it here. Um, so let's go to async. Let me zoom in here a little bit.
- 45:32
Also, um, while I do this, is there-- are there any questions that I've, that I've missed? Um, and if there are, just feel free to shout them out.
- 45:47
No? Great. Cool. I've covered everything. Everyone is, uh, perfectly up to date with everything said. Okay, um,
- 45:59
server dot py. Okay. So I have this, and now I can say like, "Hi."
- 46:14
Let me show you the, the rest of this. I guess that's probably important. Uh, da, da, da, da, da. So we have the server. We have the client. We also have the-- a very, very, very basic agent imple-implementation.
- 46:26
It looks just like the old one, with the main exception that I'm using Async OpenAI instead. Um,
- 46:33
handling tool calls is happening in parallel. So for each tool call, grab it, create the task, um,
- 46:43
await for them, and then just, like, await for them all at the same time. Um, and this just lets it, like, create a bunch of tasks that are all gonna run in parallel, and once they're done, like, if they, if they yield back at any point, we can keep doing other stuff.
- 46:56
Um, by the way, if these are just, like, heavy functions that are doing heavy processing, it's not gonna matter. The fact that they're in parallel, like, in sync, it means that it's still gonna happen one after the other.
- 47:04
But if they're like network calls, so to other models, then it's perfect. Um, the run full turn is the same one. You, like, call a model, check to see if there's any function calls.
- 47:14
If not, break, return the response. Um, the only difference here is we're awaiting, uh, the chat completion, right? This is the only difference. And then if we take a look at our agent handler, we've declared, we've declared our agent like normal, right?
- 47:28
Right now it has no instructions. It looks very familiar. And this loop is also pretty familiar. It's like,
- 47:34
get the messages. Um, here, here's the only difference, right? Um, and you'll see why this matters in a sec. I have a message queue. Uh, and this will be useful because we don't want to process multiple messages in the same conversation at the same time.
- 47:51
Like, we want work to happen at the same time, but we don't want, um, like, multiple generations to happen with the same history because then you'll get conflicting. Like, like, if two messages or if two functions return and they both need to be, like, handled, uh, you essentially, while you want them to happen in parallel, the results
- 48:09
should, should only come in one after the other. So we have a queue for that, uh, and it treats user messages as well like that. So this is the handle user input that is being called from the server.
- 48:19
Essentially just throws in a user input message into the message queue. Uh, and all we're doing here is, like, pulling off a queue, run it, like, you know, uh, put, put the messages back in the, in the message array, uh, and then, like, sleep.
- 48:37
Right? This is, like, just a very simple, uh, loop, but it does it with async. So this is what we saw, and, and it feels pretty normal, right? I can say like, you know, "My name is Ilan," and you can say, you know, "What is my name?"
- 48:53
See none. Great. Um, now this still doesn't exactly answer how we're gonna do things asynchronously, but it does give us the space to play with it because now things are happening async, and we can do a few more delegations that, that are actually async.
- 49:09
So let's start with a, you know, simple blocking get weather function. I think I required all functions to be async here.
- 49:25
Um... Yeah, sixty-seven and sunny. Uh, and then
- 49:31
let's give it that, and let's test it out.
- 49:34
The only annoying part is we have to, like, restart two things now. So hi, uh, what's the weather?
- 49:46
Great. So we can see it called a function, got the response. So far everything's normal. Now let's do some delegation stuff. So let's say,
- 49:59
let's say we want to call this function,
- 50:03
uh, three times for different places. So we have location. Um,
- 50:12
yeah. And let's say, you know, like pick a random
- 50:16
number from fifty to eighty to return. Um,
- 50:26
so if we do it in the previous case, do I still have the round? Yeah. Maybe, maybe not.
- 50:42
Okay. Uh, so if we look at... Okay, this is the non-async one. Do we have weather? We don't have weather. Let's give it weather.
- 50:52
Let's give it the same weather function. Cool.
- 50:59
Um, we only need random. We don't need this, and it's not async.
- 51:12
Okay, so now we have this weather function. Let's test th- this out in the non-async case.
- 51:18
Um, I'm gonna get rid of the rest for now.
- 51:22
Um, so if I say, you know, uh, weather in SF.
- 51:40
Cool. Calls it, fantastic. And now it's like weather in SF, New York, uh, you know, and the [REDACTED:age] other cities.
- 51:53
It's, um, it's still gonna do the parallel function calling, and here it's fine because they all return immediately, right? But now let's add like an artificial weight. So let's say time.sleep one, right?
- 52:18
Um, cool. So now what's gonna happen is we're gonna ask for that again, but it's gonna take so long.
- 52:26
Weather in [REDACTED:age] random cities of your choosing.
- 52:34
So it's gonna take a while. And the reason is each of these is having to run, um, in like o-one after the other. There we go. It took like over [REDACTED:age] seconds.
- 52:47
Now let's do the same thing here. So the equivalent, uh, is, um, there's a, a, asyncio sleep.
- 53:00
Someone let me know if I'm doing this wrong, but, um, this should emulate like a very, very similar behavior.
- 53:07
So if we instead run it here. Clear that, and that's-- I can say, you know, give me the weather
- 53:22
of [REDACTED:age] random cities of your... Um, we should see it return. Okay, so it called all of them,
- 53:32
and it got back because they all happened in parallel. Um, this is the magic of asyncio, right? Um, anything that can be parallelized or, I guess, scheduled in a way where, where it's non-blocking, uh, like sleeps, these sleeps are non-blocking.
- 53:49
Um, essentially, we can, we can do. And so you can imagine switching this sleep for like an actual API call. Um, so,
- 53:56
uh, we can actually do that right now with o1, right? Like we could call o1 multiple times, and they would all run in parallel. Now, the main problem is that we're still gonna be waiting back, like waiting to, to get all of them together, right?
- 54:12
Like, like the fact that it-- we can run them in parallel means that like we can have [REDACTED:age], ten-second tasks running in parallel, so it'll take ten seconds, but it's still gonna take ten seconds where I can't talk to the model.
- 54:24
So instead, let's have this notion of tasks.
- 54:28
There's many ways to do this. Um, I'm going to do a pattern that I quite like. So let's do this. Um,
- 54:41
let's define a, you know, create task. So I have a create task function that makes a task.
- 54:57
Um, no, and I wanna create like a task ID.
- 55:05
And I wanna keep it short. Okay. So now what I've done is I have this function that makes a random task ID, sets it, creates it, and then calls like, let's say, get weather, something.
- 55:26
Um, and then it's, you know, it's suggesting this check tasks, which is the next thing that I wanna do. So the next one is check tasks. And so
- 55:36
maybe I can just say, like, check task,
- 55:42
you know, do for... Amazing. Um, okay. So let's take a moment to, to look at what just happened. We now create a task with a random ID. We give back the ID to the model, and then later we can call check task, get that task, see what the status is, and see if it's done.
- 56:06
Um, so let's like add, I don't know, [REDACTED:age]-second delay here.
- 56:11
And right now let's just get weather. So description would be maybe create task for, you know...
- 56:22
Let's see. Again, this is all like sort of live coding, so we'll see if it works. But I can say like, you know, create a task,
- 56:34
um, and set the description to the, to be
- 56:42
San Francisco or SF. Let's see what happens.
- 56:51
Oh, I didn't give it th- these functions.
- 56:55
There we go. Um, let me just check. Cool, cool, cool, cool. Okay, let's do it again.
- 57:10
Say, you know, create a task where the location-- where the description is just this.
- 57:21
So now it creates the task, and I can say, "Hi."
- 57:26
And look at that, it... That's not working correctly.
- 57:31
Maybe it was. Maybe it just took a while to respond. Um, here, let's add a longer delay to know for sure.
- 57:38
Ten seconds, let's run it back. So create a task
- 57:51
with description SF. Uh, cool. So I can say, "Hello there."
- 58:00
Cool. So it can still respond to me, right? Um, check the task.
- 58:06
So I can still interact with it, and it, it can check to see if it's not done. Um, I guess we need to keep checking to see if it'll work.
- 58:14
So, you know, how about or, you know, tell me a joke. [laughs]
- 58:22
Who would've thought? Who would've thought? Um, cool. So now, like, check them again.
- 58:31
So it called the task. It is done, and we have seventy-eight and sunny. I will take a clap for that. I wanna see everyone clap. Please clap. Thank you.
- 58:42
Thank you. Thank you. Thank you. So what you just saw [laughs]
- 58:47
was, uh, live asynchronous, uh, programming. Uh, it's very impressive. The models can also, you know, do pretty well. Anyway, here's why this is interesting. We now have a system where we can give it tasks, and it'll queue them up, uh, and then we can check on their progress.
- 59:05
So, so already we have the basis for like a really, really interesting thing, right? Like this, this thing here right, right now is just get weather. But if we say just like, you know, uh, run...
- 59:20
What is it called? Like, um, you know, call model,
- 59:25
and then... No, but we want AI ports, Async OpenAI, get the clients. Um, I guess we could-- Actually, we can just leverage the agents.
- 59:41
They're already async. Let's not deal with that again. Um, or whatever. Let's do, let's do it this way.
- 59:48
I did it all. I did it all. Fantastic. Okay, so now we can switch get weather for call model,
- 59:55
and let's try this again. So let's give it the same task as before,
- 1:00:01
which is, you know, "Hi." Just make sure it's running. Um, you know, write me a,
- 1:00:09
uh, haiku about, [laughs] I don't know, uh, a coup where the theme is so incredible it makes me cry. Make this a task.
- 1:00:27
Okay, so it's created the task. And now I can actually say, you know,
- 1:00:34
now make a second one, but pick a theme yourself.
- 1:00:41
So now I have this interface where I can essentially keep chatting with this, and it can essentially spin off additional tasks. Um, and you check,
- 1:00:51
check all tasks. Um, let's see what the progress is like.
- 1:01:00
Look at that. So we have, we have both. Oh, was this o1? That's pretty fast. Um-
- 1:01:10
But yeah, so now we have this system that you can actually just call, call multiple ones. So I'm gonna pause here, just open it up for questions for a little bit, conversation, like other directions we can take this, other ways we could have done this.
- 1:01:20
Um, but what I wanna look at next is like how can we do this in a way where we don't actually have to check, um, because right now, like we don't need these two terminals.
- 1:01:29
This is just over-complicated. Like, I think the, the, the point of having the second terminal is so that we can push things to the conversation without a user having sent a message first.
- 1:01:39
So I'm gonna pause here, I'm gonna field some questions, and then we can get back to it.
- 1:01:50
Okay. Um, would you not use generators for nested tool calls?
- 1:01:59
Um, Tom, do you wanna... I don't know if there's a mic in the room or like, uh, just get, get it, get it to them.
- 1:02:07
Yeah, there is. Give me one second. Also, they can unmute themselves.
- 1:02:16
Uh, who's... Yes.
- 1:02:39
Uh, sorry. Sohum is the one that was, uh, answering it. Is Sohum in the room?
- 1:02:48
Yeah. Can, can you hear him?
- 1:02:51
No, I can't hear him.
- 1:03:04
Hello. Can you hear me in the Zoom?
- 1:03:07
Yeah, I can hear you now. I can't hear you anymore. [laughs]
- 1:03:21
I don't know if you're saying a question right now.
- 1:03:32
No, I was just wondering if it's possible to treat agents objects as a generator
- 1:03:44
in Python so you can nest, like, you can yield from individual agents and then yield those tool calls.
- 1:03:56
Does that make sense?
- 1:03:58
Yeah. Yeah, that makes sense. This is also very impressive that you have to do this. I think someone needs to mute themselves. I don't know if it's Alon or, uh...
- 1:04:07
Like, someone in the room un-mute it and, uh, put, put this man through, through hell. Um-
- 1:04:12
No, it's me. It's me.
- 1:04:14
Oh. Is your laptop un-muted? I can't hear anyone anymore.
- 1:04:26
Maybe just speak into your laptop. Okay, but I'll, I'll answer your original question of like, you know, if you put, um... Can you like, essentially make agents into generators so that you can like yield the results as you go?
- 1:04:41
The answer is yes, and this is actually like, um,
- 1:04:44
the right way to do this. The way I'm doing it now is I'm only exposing the final response, mostly because, um, implementing e- like, the, the generator gets a little bit tricky and I don't wanna have to like debug that.
- 1:04:55
Not too much, but you can essentially just surface each of the steps, each of the function calls, each of the everything. Um, that would, um, that would let you
- 1:05:07
essentially keep track of more agents at the same time, and you could maybe have like a bit of a flat structure where you just have multiple agents going, yielding events, and then you can essentially see like which one's coming from where.
- 1:05:19
Um, but I guess the thing is like there you have to deal with a bit of the complexity of, um, handling all the like, like essentially listening to like all the different events and like, uh, associating them back to like a specific agent and like maybe there is an agent in the front, maybe there isn't.
- 1:05:35
Um, here the idea is just I have one agent that I'm interacting with. I'm always just gonna interact with like that one agent, and it's responsible for spinning off other, other ones a- a- and like dealing with, dealing with that as well.
- 1:05:46
But you can absolutely do it the way that you're saying. And like if you wanted to build this out like more fully, you probably would in order to be able to surface progress.
- 1:05:55
Um, did that answer your question? I'm now scared to get people's talking in the room, but I'll just read them from Slack. Um,
- 1:06:05
is there any, is there any good design patterns to create projects with agents?
- 1:06:12
There's like a million. Um, you know, like I like prototyping with Swarm. Um, I know PydanticAI is also like a really nice one. I think Sam's there in the room or around.
- 1:06:24
Um, there's... Yeah, I don't wanna like... I don't really use, uh, any myself, mostly because like you saw, it's pretty simple to implement your own loop, so that's what I end up doing.
- 1:06:38
I think usually for every, every project I either write my own like... It's like, I don't know how many, how many lines is this? Like,
- 1:06:48
like 70 lines. And so like depending what you want, you might want more, you might want less, and I'm sure there's good solutions. Um, but like I just ha- like I just copy this around.
- 1:06:57
Um, I don't really like working with too many dependencies, especially for something that's so lightweight. Like I, I want granular control. Um, and like as a for example for Swarm, I ended up having to hop in here and like handling specific kinds of tool calls in certain ways so that I can do handoffs, um, which we can
- 1:07:13
actually look at and we can implement this without too much trouble. Um, but yeah, my answer is like- There's many you can choose from. I don't have many I would recommend personally.
- 1:07:24
Um, but I, I am a, I am a fan of, of PydanticAI. It's pretty cool. I haven't used it too much, but, like, the interface looks nice. Um, it reminds me a lot of Swarm.
- 1:07:34
The one I use the most for prototyping is Swarm. I think it's just because it, it's the one that, like, I'm most familiar with.
- 1:07:41
Um, when you have to start to approach dozens or hundreds of functions, what techniques should we apply in order to effectively tool call? There's, there's a few answers there, right?
- 1:07:52
Um, you can have multiple agents and essentially, like, split up the responsibilities or, like, the, the groupings of the functions, um, and so into, into, yeah, like clusters where you have, like, a set of related functions that are needed for specific tasks, and then you can invoke the correct agent.
- 1:08:16
Uh, and this is where, like, multi-agent patterns start to make sense is, like, specifically when you have tons and tons of functions, like, how do you, uh, go to the right ones?
- 1:08:26
If you, for some reason, need them all at the same time, uh, you could try fine-tuning. Um, in, in projects for OpenAI, I've ended up fine-tuning up to, like, hundreds of func-- like, it was like 120 functions.
- 1:08:38
This was with, uh, GPT 3.5. So the fact that that works gives me, like, pretty high confidence. Like, you can fine-tune, uh, smaller models with a lot of functions and get them to work pretty well.
- 1:08:48
Um, the last one is, like, some kind of dynamic function loading where based on the input or based on the conversation, you load into memory or, like, you load into context, um, the most likely relevant functions.
- 1:09:05
Um, and there's a few different ways to do this. You can do this with embeddings. You can do this with, like, um,
- 1:09:13
having, like, a two-step function call. At that point, you're essentially having agents. Like, if, if you call a function to then load more functions, that's what a handoff is essentially.
- 1:09:21
So a lot of these start to look very similar. Um, it's just like how are you loading multiple different ones? Um, okay. For reasoning models, are there tools being called within the thought text?
- 1:09:33
Um, so because we don't expose the thoughts, like the, the, the chain of thought, um, that's a little bit hard to answer. For, for, for o1 right now in the API, the answer is no.
- 1:09:47
Um, I'm trying to see how much I can say here. Um, it is something that is technically possible, right? Like, you can do anything you want with, with post-training.
- 1:10:01
Um, we do not currently allow you to call functions within the chain of thought. Um, so yeah. Right now, the, the function calls happen at the very end. Uh, do you have any good code examples for router patterns in these lots of functions cases?
- 1:10:19
Yeah. So, so my, like, my answer there is the-- some-- one of these. Like, if you really just wanna route, um, then, like, the, the idea of having, like, multiple agents and handing off to one of them is actually really nice.
- 1:10:31
Um, like, you can just define multiple agents, each of them with, like, multiple functions, um, and then have the first one have like a... We can actually do this quickly.
- 1:10:41
Like, we-- I'll, I'll do this in Swarm, but like I said, you can implement this yourself. Um, I don't actually know who else supports handoffs the same way Swarm does.
- 1:10:48
But, um, essentially here, let's start a new file. Like, uh,
- 1:10:57
routing. So no. Let's see. Swarm repl demo loop. I can say like, you know, triage
- 1:11:14
equals one, and then I can have my other two. It's like, you know, maybe I have like some collection of like, um, you can call them an agent, but I can also just call them, yeah, like, uh, what would it be?
- 1:11:25
Like, you know, uh, email functions agent. And you can have like, you know, like, send email.
- 1:11:40
Check email. I don't know. What else does Cursor want me to write?
- 1:11:45
Not get weather. Um, cool. We have th-these two, and then we can do like maybe we have the emails, and then we have maybe a calendar. I don't know.
- 1:11:54
So make, like, create event. Can do like calendar. Yeah. I'm just gonna say like, you know, finish what I'm doing.
- 1:12:10
Update the prompts and tools, because I'm lazy. So let's see what, uh, let's see what Cursor thinks. Great.
- 1:12:20
So now we have an email functions agent and a calendar agent. Call them whatever you want. Um,
- 1:12:28
now, like, let's pretend that instead of just having three functions, we have 30, and each one of these has 10 or 15. Um, then, like, if you maybe give all 30 to one agent, first of all, try it.
- 1:12:42
Like, if that works, amazing, right? You don't have to deal with the extra complexity. Um, so you don't really wanna do handoffs and stuff until you really, really need to through, through like evals.
- 1:12:52
Um, but, um, yeah. So, so and then the special, like, handler functions here is like, you know, uh, transfer to
- 1:13:05
email agent, and I'll return email agent, and then cool. So now I have my two transfer functions.
- 1:13:22
Oh. Oh, it did it. Okay, cool. So I think this should just work, right? So n- I've defined, like, the actual functions and agents that in your case would be many, many more functions.
- 1:13:34
Um, I've defined the transfer functions, and I've given them to the triage agent. So now if I run this.
- 1:13:45
Say, "Hi." Um, you know, "I want to send an email."
- 1:13:56
Cool. So now I'm talking to the email agent. Now, if you wanna do, like, more, you know, ch- like, transfers.
- 1:14:05
Uh, oop, this is not delegation. Okay, so trail assistant. Um, maybe we can tell the assistant, like,
- 1:14:15
you know, if you already know what the user is asking,
- 1:14:22
just call that function, right? And this is if you wanna have, like, a, a case where, you know, it still routes you, but it's like a faster two-hop. So maybe I can say, like, what are the functions here?
- 1:14:36
Um, what are the parameters? Send email to subject body.
- 1:14:43
So I can say, like, you know, "Send an email to [REDACTED:email_address] about taxes."
- 1:14:55
What was the other one? Body. Um, saying, "Yo, do your taxes."
- 1:15:04
So it should transfer me to the email one, and then there we go. It immediately sends the email. So this felt like an immediate function call, but there was a transfer in the way.
- 1:15:14
So this is kind of an example of, like, triaging does work. It's really convenient to model it with agents and handoffs. Um, I'd say the primary use case for agents and handoffs is just a glorified triage through, uh, multiple functions.
- 1:15:28
Um, yeah. Let's see. Let's go back to questions.
- 1:15:37
Uh-huh. Consider having something like Streamlit Gradio until you-- Oh. Are you considering having something like Streamlit Gradio util that allows porting all interaction functionality with one command to app?
- 1:15:52
Uh, I don't think so. I don't know. Uh, what did people answer? There we go. [laughs] Sam's in with, uh, PydanticAI, so check that out. Um, how many tool calls can you get in one iteration?
- 1:16:06
You-- Like, parallel function calls, I don't think we have hard limits on either the number of functions or the number of parallel function calls. How big a tool library will the models perform well with?
- 1:16:18
Super, super general rule of thumb is, like, ten to twenty you shouldn't really pass over. Um, but like I said, I've gotten-- Like, this was a very specific case where, like, it was extremely latency sensitive, and so, like, we had to have flat, like, flat function calling, and we did a hundred and twenty functions with GPT-3.5.
- 1:16:39
So you can go pretty far, um, with fine-tuning, but, like, I'd say reliably without, without very extensive prompting, yeah, probably, like, ten to twenty. Like, pa-past that point, you really ask yourself, like, what are you trying to do?
- 1:16:54
Right? Like, why are you putting so many functions? Is it super latency sensitive? Can you split it up? Like, yeah. Um, was there a follow-up here?
- 1:17:05
Okay. Um, rather than tool calls, I feel like we're moving to generated code agents. Will I soon be able to supply my tools functions? Ah, okay, that's a good idea.
- 1:17:14
We should try that. Um, so essentially have something write its own function and then use it. Uh, yeah. I feel like it-- we can probably... How would we do that?
- 1:17:24
Yeah, we can try that. We can, we can find a way to do that. Uh, I'll, I'll do that. Actually, let's do it right now. Why not? Let's do it right now.
- 1:17:30
Uh, I've never done this before, but I feel like it shouldn't be too hard. So, um,
- 1:17:37
okay. Uh, you know, what do we call this? Like, bootstraps.
- 1:17:43
Okay, so I'll keep using Swarm because I, I will use, um, handoffs.
- 1:17:50
From Swarm import agent. Cool. So now-- [laughs] It's always bother. So let's see. We want an agent that writes its own functions. So agent
- 1:18:02
was this one. Um, we want it to write its own functions.
- 1:18:12
So maybe we can have it ha-handoff to itself. [laughs]
- 1:18:16
So we can do, you know, like, refresh, you know, autoBE refresh functions.
- 1:18:25
So we can actually return the same agent. Um,
- 1:18:31
and then... Okay. Is it not declared? Is it unhappy?
- 1:18:37
Uh, yeah, it's not defined yet. Okay, so now we can define it down here.
- 1:18:46
Cool. Functions. Okay. Um, bootstrap. Bootstraps, bootstraps. Okay, so we have this, and we want it to write its own functions. So how are we gonna do this?
- 1:19:02
We want it to produce Python. Does anyone have any ideas? Uh, if you wanna shout them out, like, make-- just pass around the mic while I code. Just feel free to...
- 1:19:12
I can't, I can't read while you're, uh, what, what you write, but we can, we can try this. So, um, let's say we want it to,
- 1:19:19
you know, um- You know, add tool. There we go. And then let's call this, you know, Python implementation.
- 1:19:39
Okay, I'm gonna do something very unsafe. So, you know, what would it be? It's like function obj equals
- 1:19:51
eval of Python. Don't do this. Don't do this, kids. Um,
- 1:20:01
and then will this work? Is this how Python works? Sam's there. Uh, okay, so if I have this
- 1:20:10
and I eval, like, you know, def a...
- 1:20:27
Does that do it? Um, okay. You know what?
- 1:20:41
Um, so I guess it's like, uh, I want to write a,
- 1:20:47
write a function that takes a string presenting an implementation of a Python function and returns the actual Python function as interpreted.
- 1:21:06
I don't know. Let's see what it does.
- 1:21:10
Um, so essentially we wanna do that. Once we have the function, then we can just append it to the agent, and then we might need to reload it. Uh, so we can then just return the agent, and this might be it. [laughs]
- 1:21:26
This might be it. So add tool. Exec, not eval.
- 1:21:35
Why didn't anyone say that? You guys can shout out.
- 1:21:39
Okay. Okay, so it's exec. And then what is this, a?
- 1:21:57
I don't know. Someone save me. Like, ah.
- 1:22:08
Uh, hey, there's a couple implementations in the Slack.
- 1:22:11
Sorry?
- 1:22:12
Check the Slack.
- 1:22:18
Did someone do this? Hey. Okay. [laughs]
- 1:22:23
It's almost like someone's
- 1:22:25
No, no, no, but this is not-- Okay, no, no, but this is not what I want, right? Like, I don't want it to run it in a subprocess. I want it to, I want it to evalu- like evaluate the function-
- 1:22:35
Uh-huh
- 1:22:36
... and then turn it into a ... Is that-- Oh, go up. Oh, Sam, of course. Amazing.
- 1:22:43
Uh, [laughs] Jesus Christ. [laughs] Okay. Um, make it so the add tool function, uh,
- 1:23:04
evaluates the implementation and adds it to the tools, similar to now.
- 1:23:15
Here's a great reference for how to. That's-- And then we wanna grab-- Was this a good idea? This was a terrible idea. I hope people are enjoying this.
- 1:23:30
Um ... Okay, let's see. Um, and let me just read this, 'cause it might not be that complicated to add in. So
- 1:23:49
I can't-- Okay, it got in the way. No, not Zoom. Where's Slack? [sighs]
- 1:24:17
Was this it? Is this it? [laughs] Okay, I'm gonna, I'm gonna put this on hold for now. It sort of ignored your implementation, Sam. I'm sorry.
- 1:24:33
Um, let's see. So if you have parse function, can I just
- 1:24:40
do this? Does this work? Uh, okay, let's see. I've-- There's a first time for everything. Uh, wrong one. Sub
- 1:24:53
bootstraps. Hi. Um, add a tool that prints hello when called.
- 1:25:11
Call it. Oh, but no, no, no, no, no. Make it print it, not just return it.
- 1:25:27
My heart's pounding. Okay. Um, so say hello tool, call it.
- 1:25:35
Look at that! Look at that. Okay, so now we have-- Yes, I will take a clap for that. We did this together, guys. Um, so this is actually a lot less code than I was expecting, but now we have a system, look at that, it's tiny, um, that can write its own tools.
- 1:25:51
Um, and so, like, maybe we can do something like, you know, um, what is, you know, like, make yourself a little calculator.
- 1:26:05
I can't believe this just worked. Uh, you know, what is
- 1:26:10
two times three times... Can someone check this? Uh, wait, wait, wait. [laughs]
- 1:26:23
Three, four, six, nine, eights. Three, four, six, nine, eights. This is, this is crazy. This is, this is so fun. Uh, yeah, I hadn't done this before, and this is a lot less code than I thought.
- 1:26:34
Look at this. Look, this is all you need. I mean, it's... This is super dangerous code. Like, this, [laughs]
- 1:26:41
this is not good. Don't do this. But it's fun. Uh, I would put this squarely in the fun things. We can, we can now transition to looking at other fun things.
- 1:26:51
Um, they're definitely not as fun as this one, I think. Um, they're just a couple of, like, random things that I've found related to real-time, since I did sort of put it in this, uh, title.
- 1:27:02
So whatever. Okay. These are the two main tricks that I, that I kind of thought were pretty cool, and you might have already seen them. Um, one is if you've ever dealt with, like, the Realtime API,
- 1:27:15
um, like jumping in, uh, before you're done with an idea or, like, done talking or something, um, like, what I was thinking is, like, if, if really what you want is you want it to, like, use the model's own intelligence to decide if you're done talking or not, um, and you can treat our VAD, like our voice
- 1:27:35
detection, as like a trigger-happy version of that. It's like a, it'll always tell you when maybe the user is ready to stop talking. But if you want the model to check, you can have a stay silent function or something else that essentially handles the other side of that, where it's like, okay, we're definitely gonna let you know
- 1:27:53
whenever the user might be done talking, but then you can actually verify with a function call. And so this implementation is, like, super, super simple. You literally just give-- you give the, the function itself
- 1:28:05
to Realtime API and tell it to call it when the user is not quite done talking. And I'm not gonna try to demo this live, like realtime demos are tough, audio is tough, all that fun s- fun stuff's tough.
- 1:28:18
But, um, it works, like, surprisingly well. Uh, if you just describe, uh, uh, I can... You can probably find this tweet. It has the full prompt. But essentially it's like, you can, you can like pause, you can say like, "You know, I've been thinking about," and then like stop, and like it'll get triggered, call stay silent, and
- 1:28:36
then you can keep talking. Um, so pretty, pretty cool, uh, useful thing. Um, and then the other one is also related to Realtime API. And if, again, if you, if you, uh, are following me, you might have already seen this.
- 1:28:49
But, um, someone at Dev Day just, like, came up to me and asked me like, "Hey, um, is there a way to like make the Realtime API talk in like a specific way?"
- 1:29:00
Uh, and I was like, [laughs] "I don't know. Let's try it." And so like we had a demo booth, and I sort of pushed someone out of the demo booth and grabbed the laptop, and I was like, "Let's just like try, try this stuff."
- 1:29:10
And apparently, if you ask the model to just-- you give it like a, a script, and you're just like, "Hey, read this out according to these XML tags," um,
- 1:29:19
you can have it follow them. Here, let me, let me find, um,
- 1:29:24
the original, the original one. Yada, yada, yada.
- 1:29:30
Sorry, this is-- I'm not trying to show you the whole timeline. Where is it? Okay, this is the stay silent one. Um...
- 1:29:47
Where is it? Where is it? There we go.
- 1:29:52
Um, uh, it's impossible you'll hear this, right?
- 1:29:58
You guys don't hear this. Yes? No?
- 1:30:03
Uh, we don't hear it.
- 1:30:04
Okay, that's fine.
- 1:30:05
But you can share your computer audio.
- 1:30:08
Can I?
- 1:30:10
It's doing things.
- 1:30:10
I'm, I'm, I'm actually-- I'm not gonna try. It's fine. Um,
- 1:30:15
wait, can I? Seems... No. No, it's fine. Whatever. Anyway, um, yeah, so if you, if you, if you like give it like
- 1:30:25
a script like this, um, we didn't actually train it for this, right? Like this is just a really nice consequence of like behavior. It's technically not function calling, I'm realizing now, but it's like, it's very function calling-esque.
- 1:30:37
It's real time. The title of this talk does include real time, so this is, uh, this is it. Um, I will pause here. I have so much more I can go through, but I wanna give you guys a chance to like ask questions, poke around with some of these ideas.
- 1:30:49
Uh, Swarm is public, you can just try it out. Um, we can also just try creating other functions. Uh, this, this is one of the-- this is easily one of the coolest, like, little programs I've ever made.
- 1:30:59
Um, so yeah. Uh, let, let me see if there's any other questions in the Slack.
- 1:31:14
Uh, da, da, da, da, da. It's a room full of people autocompleting. [laughs]
- 1:31:20
Make... Yeah, yeah, yeah. Okay, let's not do that.
- 1:31:25
Uh, okay, revisiting memory. Let's see. Do you have any suggestions for trying to enforce consistency of stored memories?
- 1:31:33
Identifying inconsistencies and figure out how to resolve them. Okay. And then the second part is, what data structures do you suggest for more structured memory, helping enable more meaningful comparison of objects?
- 1:31:43
I think-- I mean, this question opens, like, you start to go down a path where you can get as complex as you would like, right? Like, like you can start simple, and you can end up with, like, an entire operating system to manage memory, right?
- 1:31:57
There's like a whole range. Um, one way to do... O-off the top of my head, like, one way I can think to do this is, um, when you are about to store memory, do a retrieval, like do search, to find similar memories or like memories that like are semantically similar.
- 1:32:18
Um, and then do an explicit check, uh, with a model to see if like, you know, th- one is like updating or contradicting another, right? An example of this is like, you know, what is the latest state of a project, right?
- 1:32:31
You know, if someone's like, "Is the project..." Like, at some point you say like, "I'm working on the project. Like, it's not ready yet." And it saves that, and then later you're like, you know, "I, um, ha- like the project's done now."
- 1:32:44
Right? Like, those are two contradicting memories. Um,
- 1:32:49
what you can do is essentially, uh, have a timestamp for them, but also when you are about to store the second one or any memory, check for similar ones and create like a direct, like, essentially,
- 1:33:06
um, like node, uh, pointing from the original memory to the new one. And so I guess, I guess what I'm thinking is like that way if anybody asks
- 1:33:23
y- and you surface both, like you can keep both in memory. And when you do like retrieval and semantic similarity, you can surface both. But essentially, you can present like the whole chain of updates.
- 1:33:34
And so you can just present the last one if you want, or you can present the whole chain and the model have so- has some idea of like, you know, maybe if you ask it like, um, you know, how long is this project delayed?
- 1:33:45
Because that was the last you heard about it. But somebody at some point said, "Oh, it's actually done." Um,
- 1:33:51
like if it raises both and has one with a later date, you know, maybe. But if you like, if you've already made this chain explicitly, um, like you can actually represent that, and you can choose to not show the previous ones.
- 1:34:03
So that's one idea. Like I said, there's so many. You can like-- there's so many ways to do this. Uh, and, and like, as soon as I stop talking, you guys walk out, someone's gonna like be like, "Oh, there's actually this, like much easier way that like this idiot didn't think of."
- 1:34:15
So anything you think of probably works. Um,
- 1:34:21
here's a real-time API prompt. Yeah. Cool. Uh, any other, any other questions about anything? Um, it's been like an hour and forty minutes, so happy to, like, keep going with random content.
- 1:34:37
Um, there's a couple demos that I have that are like demoing some public repos that we have that you can use. Um,
- 1:34:47
yeah, but I wanna g- I, I wanna give anyone a chance to like speak up, say something if you want, ask questions.
- 1:34:54
Um, I, I don't see, I don't see a, a lot happening in... Like I can't, I c- you guys are tiny on my screen.
- 1:35:09
Uh, cool. Okay. Then I'll pose this to you. Sorry, what?
- 1:35:15
Oh, no, go ahead. It sounds like you're wrapping up. [laughs]
- 1:35:21
Yeah. The, the last thing we can do, I guess, is like, either I can, um, try and pull up a real-time, uh, demo that shows how to do the o1 stuff, uh, in a, in a slightly easier way.
- 1:35:35
Okay, cool. People wanna do that demo. [laughs] Okay, let's do it. So, um, the reason I'm a little iffy about it is because, yeah, it's real-time, uh, and those demos are tough to do publicly, but, um, there is this repo, openai, um, realtime-twiglio-demo,
- 1:35:58
um, that essentially sets up your whole, um, like phone calling assistant. Uh, and it's just works right out of the box. It walks you through the steps. Um, I may end up sharing a, a phone publicly.
- 1:36:12
Don't call it during this demo and don't share it, and I will delete it. But, um, yeah, I trust you guys. So let's see.
- 1:36:25
Okay, so looks like this is already running.
- 1:36:31
Oh, I do have it already running. Okay.
- 1:36:34
Um, oh, never mind. I, I did, uh, I did make it all dots. But yeah, essentially this, um, this little checklist is like live updates. So as you set up the account, as you set up the phone number, as you set up your local web server, et cetera, it'll like get checked.
- 1:36:50
So setting up Twilio and Groq and everything has been one of the most annoying things I've had to do multiple times, um, related to the real-time API. So this hopefully makes that process like a lot, a lot easier.
- 1:37:00
You can do most of it directly from here, and it lets you know like when things are done. So you don't actually have to do a lot from the Twilio console at all.
- 1:37:07
Um, anyway, so, um, I-- let me show you the back end. So there is a
- 1:37:18
handler function. There we go. Function handlers. So, um, you can either implement tools, uh, locally as a JSON schema, and we can, we can try that really quick. So this is the real-time playground, and I can say like, you know, make a-
- 1:37:36
Function to, you know, get the answer to answers to everything.
- 1:37:45
No params. [laughs] Maybe one param. Okay, so it gave, it gave us this nice little, uh, nice little function. Now we can just paste it in here.
- 1:38:01
Save it. And cool. So now we have this get universe answers. And I can save this configuration and call the phone number.
- 1:38:15
So let's see. Hello? Hello? Okay, this is why I love them.
- 1:38:31
Let me, let me try once more. Okay, let's go. Tool. Paste this. Save changes. Save config. Um.
- 1:38:53
Hello? Hello? I don't think it's happy. Is someone else-- No, I didn't show the number. Hello? Okay.
- 1:39:08
It's not working. That's fine. Sort of expected it. Um, essentially what you can do, like, when you, when you use this, um, it's cool because these tools you can either specify local ones with schema and they appear here, or you can define them in the back end.
- 1:39:24
Um, so if you define them in the back end, you can actually give, like, code to execute. Um, if not, they'll just show up here, uh, and you can enter in, like, what the response will be, uh, like a mock response.
- 1:39:34
Um, but you can also, like, set up back-end functions that will actually get handled in the back end. Uh, the sample one is, like, a real implemented get weather one.
- 1:39:43
But as you can see, there's like-- it's pretty straightforward to implement, like, the o1 one. Now, the main difference between the Realtime API here and, like, what we were doing before is the Realtime API does actually allow asynchronous functions natively.
- 1:39:59
So you can-- the model can call a function, um, get no response, and you can keep talking with the model, um, until a response is back. So that, that is behavior that, like, we specifically had to teach the real-time models, because unlike a chat conversation, you can't really enforce, um, like, you can't really halt the whole conversation
- 1:40:21
until the function response comes back. So we had to do that for, for, for launch. Um, because originally when we hadn't done that, you know, it-- we hadn't ever shown it how to do asynchronous functions, so it just couldn't, couldn't handle them correctly.
- 1:40:35
Anyway, sadly, uh, demo didn't quite work. Maybe I can try this one more time. We'll see.
- 1:40:40
Third time's a charm. Hello? Hello?
- 1:40:59
No. Okay, great. Anyway, um, I believe that is most of what I wanted to cover today.
- 1:41:11
Um, I'm gonna play with this hyper unsafe function-making agent a bit. But, um, yeah, thank you all for coming. I'll hang around for questions for a bit if anyone has any.
- 1:41:23
Otherwise, I think we're ending a little bit early. Um, like I said, yeah, mo-most of you can, can hang out or, or leave as you please. Um, but I'll-- maybe I'll call it wrapped up here.
- 1:41:36
Um, maybe what we can do is if you're interested, you can stay and, like, hack on some of the stuff that you saw. I'm happy to hang around, uh, and answer questions as well.
- 1:41:46
So thank you all for coming. Uh, hope you, hope you got something out of this.
- 1:41:58
Um, Anup is also here, uh, to help you folks.
- 1:42:03
Uh, otherwise, Ilan, I guess you can share your screen a little bit in case people have questions.
- 1:42:09
Mm-hmm. I'm actually gonna dump this, but this may be a really bad idea. But, um,
- 1:42:15
here you go. Here's the code for the super unsafe self, like, tool writing agent. [laughs]
- 1:42:34
And I'm probably gonna stop sharing my screen.
- 1:42:38
Um, yeah, I can, I can, I can share the repo and the slides. There's not a lot on the slides, but yeah, I'm happy to, happy to share them after.