AI Engineer World's Fair 2024
Building SOTA Open Weights Tool Use: The Command R Family
Read the talk
Building RAG and Multi-Step Tool Use with the Command R Family
Command R and R+ connect retrieval, grounded answers, and dependent tool calls, with an application toolkit that makes those model behaviors usable in a chat interface.
From a talk by Sandra Kublik
Before you start: Familiarity with language models, retrieval-augmented generation, and basic API calls will help you follow the discussion.
Open weights for retrieval and sequential reasoning
What if a model could handle advanced retrieval-augmented generation and sequential reasoning, compete with larger proprietary models, and still let you run its weights locally? That is the opening premise of Sandra Kublik’s presentation. She positions Cohere’s family against GPT-4 Turbo and Claude Opus, while emphasizing smaller models and local execution. Local availability is part of the proposition; the talk does not specify the hardware required.
Command R arrived on March 11, 2024, with open weights and a focus on scalable RAG. Command R+ followed on April 4, extending that emphasis to advanced retrieval and tool use. Open weights should not be confused with unrestricted licensing: the original Command R+ model card specifies CC-BY-NC licensing and an acceptable-use addendum. Kublik reports that R+ climbed the LMSYS Arena shortly after release, presenting community reception alongside the models’ intended specializations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Different applications, different kinds of grounding
Early adoption appeared across model platforms and applications. Kublik reports OpenRouter trending and 150,000 Hugging Face downloads within two weeks of release. HuggingChat’s tools integration used Command R+ as its underlying model, with tools including a document parser, image editor, and calculator. This means R+ powered the application’s tool behavior, rather than serving as a training base for a separate HuggingChat model. Kublik reports that almost half a million developers and researchers were using the R family at the time of the talk. These are historical adoption figures attributed to the presentation.
The examples move from broad adoption to concrete applications: Daniel San’s Coding Assistant and Complexity’s generative search, followed by two bots in Cohere’s Discord community. The bots illustrate two distinct ways to shape an application:
- Playful behavior: A fine-tuned bot demonstrates model capabilities through a more playful interaction style.
- Documentation help: A helpful bot grounds its answers in Cohere’s documentation and API information.
The distinction matters because adapting how a model behaves and supplying the evidence it should answer from solve different problems. The accompanying Discord examples show API documentation help beside a Dino Dash game concept.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why retrieved text is not enough
Building a reliable RAG application requires more than putting search results into a prompt. Kublik identifies three difficulties, beginning with prompt sensitivity. The model must know what information to look for, where to look, and how to distinguish retrieved evidence from the conversation it has already had with the user. A prompt that blurs those roles leaves the model to infer which text should govern its answer.
The second difficulty is document-position bias. Kublik describes models paying excessive attention to the beginning of a document instead of consistently finding the most relevant information wherever it appears. She points to retrieval evaluations, including needle-in-the-haystack tests, as ways this weakness becomes visible. Having the right information somewhere in the context does not ensure that the answer will use it.
The third is the conflict between pretrained knowledge and supplied evidence. The model’s parameters may contain an answer that no longer matches the world. Asking who the current president of the United States is illustrates the problem: the task calls for up-to-date information, not simply the name most strongly represented in training. RAG must let external evidence guide an answer when the stored knowledge is stale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train the retrieval decision, then make the answer inspectable
Cohere’s response is to shape the model’s retrieval behavior through post-training. The intended sequence is:
- Decide whether external information is needed. Some requests can be answered from pretrained knowledge.
- Operate the retrieval system. Form search queries and retrieve relevant information when the request requires it.
- Use the results as grounded context. Bring that evidence into the ongoing conversation when generating the answer.
This makes the decision to retrieve part of the model’s trained behavior, alongside the ability to use what retrieval returns.
Fine-grained citations make that grounding inspectable. A user needs to see where a particular claim came from and whether its source deserves trust. Kublik describes citation behavior as a major training focus and associates it with lower hallucination and more reliable use of context. Citations provide a route to checking an answer; their presence alone does not replace that check.
Kublik reports best-in-class performance for Command R and R+ on standard RAG datasets including KILT. The talk does not supply the tasks, scores, or evaluation configuration needed to interpret that comparison precisely. The broader design claim is qualitative: models small enough to be affordable, with good token efficiency, can cover use cases that would otherwise require a larger pipeline of LLMs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Toolkit supplies the application around the model
Even with retrieval and citation behavior in the model, an application still needs to expose those capabilities. A useful RAG chat interface must retain conversation history, display fine-grained citations, accept document uploads, and connect to tools. Cohere spent time building that interface and open-sourced it on April 24, 2024 as Cohere Toolkit. The repository was subsequently archived on May 14, 2026, so it is a historical application reference rather than an actively maintained starter today.
Toolkit separates the interface, model access, and retrieval so that an application can change one without rebuilding all the others.
| Component | Role | Choices described in the talk |
|---|---|---|
| Interface and storage | Chat UI and conversation history | Next.js application; small SQL database |
| Model access | Connect the application to Command R models | Cloud providers, Cohere platform, local access, Hugging Face |
| Retrieval | Connect tools and data sources | LangChain-based example with document uploads and web search |
The example retriever is a starting point. The retrieval component can be extended with other tools and data sources, while the model component determines where inference happens.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From retrieval to dependent tool calls
Training a model to operate vector databases and retrieval systems provides a path toward broader enterprise tool use. The next goal is to let the model work with tools beyond retrieval, ideally from their descriptions alone. Kublik presents zero-shot use of arbitrary tools as a direction of training, not a universally achieved capability.
The critical distinction is whether actions depend on one another.
| Tool-use pattern | Dependency | Example |
|---|---|---|
| Single-step | One action, or independent actions | Search for a document; send an email |
| Multi-step | Later actions use earlier results | Find → compare → summarize → email |
In the multi-step example, the system finds a document, compares it with another document, summarizes the comparison, and emails the result. It cannot produce the comparison summary before it has the comparison, and the email needs the resulting summary. The workflow’s dependencies determine the sequence.
That sequence need not be a fixed march toward completion. The system may retrieve from different sources repeatedly, inspect intermediate results, and correct errors. Kublik describes this as a loop and uses the term agents for software performing a sequence of actions in which each action builds on earlier steps. The relevant capability is not merely issuing several calls; it is using their results to decide what should happen next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Describe tools, inspect results, revise the plan
The multi-step API, described as released the week before the talk, exposes this behavior through tool descriptions. The developer supplies the available tools, what each does, and its parameters. A user request then gives the model a goal from which to construct a plan. This is the historical API presentation; the relative release date does not establish when multi-step capability first became available.
The execution loop connects planning to evidence:
- Select a tool call that advances the user’s request.
- Execute the call and return its result to the model, as illustrated by the Python loop on the slide.
- Inspect the returned content and decide whether the initial plan still fits.
- Continue or revise. If an API returns an error, retrying or changing the plan may be the next action.
In the document workflow, these returned results are what let the model progress from finding documents to comparing them and preparing the email. Retrying and replanning are intended behaviors, not guarantees that an unavailable API or another failure will be recovered from successfully.
The behavioral instructions live in a large multi-step preamble that Kublik points readers to on Hugging Face. It tells the model how to carry out the task, including how to proceed through intermediate results. The model’s trained prompt structure matters here: the model card warns that departing from its template can reduce performance. Open weights make these instructions and model behavior available for inspection, but do not make the surrounding orchestration irrelevant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tool provenance and production economics
Citations carry the grounding principle into tool use. Command R and R+ are trained to produce claims that users can verify through citations, making it possible to explain which tool supplied the information behind a response. For a workflow that crosses several data sources, that connection between a claim and its origin is a practical part of transparency.
Kublik claims Command R+ is competitive with Claude Opus and GPT-4 Turbo while being three to five times cheaper. The talk does not provide the pricing basis, workload, or exact model versions, so this is a historical cost comparison rather than a general deployment estimate. Kublik also reports Command R+ as close to or on par with GPT-4 Turbo on standard complex-reasoning benchmarks, without naming those benchmarks or supplying scores. The production proposition is that competitive capability at lower cost could make repeated tool-use steps more scalable.
The stated direction for upcoming releases is continued work on multi-step capabilities. No release dates or additional specific capabilities are promised: the focus remains improving the model’s ability to carry a task through dependent actions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The March 2024 announcement introducing Command R's retrieval-focused positioning.
Model specifications, licensing, inference examples, and grounding and tool-use guidance for the original Command R+.
An archived RAG application with a Next.js interface, conversation storage, model-provider configuration, and extensible tools.
The benchmark paper explaining shared Wikipedia grounding and evaluation of answers and supporting provenance.
The historical announcement of HuggingChat's tool integration using Command R+.
Further reading
- Multi-step tool use becomes the Chat API defaultDocumentation
The June 2024 changelog documenting the switch to multi-step tool use by default.
Read the complete timestamped transcript
- 0:00
[on-hold music] What if I told you that we have just handed you the keys to state-of-the-art model,
- 0:21
which excels at structured advanced RAG at sequential reasoning,
- 0:31
and you can run it locally on your machine.
- 0:35
It's competitive against GPT-4 Turbo, Claude Opus, and it's much smaller.
- 0:44
We've been really hard at work at Cohere, working on our family of models, and today, I'd like to talk to you about some of the stuff that we've done,
- 0:56
the decisions that we've made when it comes to the model design, and also what we're cooking when it comes to the future of the models.
- 1:08
So this year, we've been working really hard to push the boundaries of what's possible with LLMs, and
- 1:18
here's a quick look at our timeline. Three months ago, on March eleventh, we've released Command R. We opened the weights to the model.
- 1:29
Command R is a model optimized for retrieval-augmented generation, and it's scalable. It's small enough to be scale-friendly.
- 1:41
We followed it up with Command R+, and this model
- 1:48
is optimized for tool use, advanced retrieval-augmented generation, and has become a very popular model in the open source community. Within few days of the release, we've climbed the LMSYS Arena.
- 2:06
We're really proud of that. Really great achievement. Mm,
- 2:11
your response as a community using the model has been incredible.
- 2:18
Some of the zeitgeists. We started trending at OpenRouter.
- 2:23
Within two weeks of the release, the model has been downloaded a hundred and fifty thousand times from Hugging Face, which is wild.
- 2:34
Folks at Hugging Face actually like the model so much, especially when it comes to the tool use, that they decided to use it as a base model for HuggingChat.
- 2:44
So now you can play with HuggingChat. It has a doc parser, an image editor. It even has a calculator. It had it before the iPad. So today, almost, um, almost half a million of developers and researchers are using the R family.
- 3:06
We're really proud of that. It looks like you guys got really excited to get your hands on the model and to be play with the weights and look under the hood.
- 3:21
We keep hearing your feedback, and the love and support keeps pouring in. It really gets us going.
- 3:28
And I've seen some super cool stuff built with R+ since then. Some of my favorite ones I wanna shout out here are the Coding Assistant by Daniel San and a new generative search demo by Complexity.
- 3:44
I'll try to demo it later. We'll see how the tech goes, but I'll give you a sneak peek.
- 3:50
Another one that's my favorite is two Discord server bots that are powering, um, our Discord community. I invite you to go and check it out. One of them is fine-tuned to be playful and to demo the model capabilities, and the other one is made to be helpful.
- 4:11
It's grounded in our docs, and it's focused on the information coming from the API.
- 4:20
So I want to share the journey of building the R models, the decisions we've made along the way,
- 4:29
and to show you that we've committed ourselves to build the top RAG tools for AI builders.
- 4:40
We know firsthand that building RAG is excruciatingly hard. Tough word. When you set out to do that, um, you're going to face challenges, and
- 4:54
they are numerous. Challenge number one is that models are highly prompt sensitive, and when you, mm, want to use the model in the RAG context, you need to prompt it to not only look for the information
- 5:10
but also know where to look and know how to differentiate between the conversation history that the model has with the user and the retrieved information. It's not a trivial task.
- 5:23
Another problem is overcoming model's natural bias towards focusing on the beginning of the document. You've seen it with multiple RAG benchmarks and, um, evaluation tests, needle in the haystack and whatnot, that are really showing the problem of models not focusing on the most accurate information retrieval but rather
- 5:47
becoming a little bit lazy and focusing on the beginning mostly.
- 5:54
Another challenge is steering an ongoing battle that's happening within the model
- 6:01
between its pre-training knowledge and what it encounters in prompts.
- 6:08
For RAG use cases, you want the model to be able to tap into the knowledge that's not baked into the model parameters and
- 6:17
temporal information is a great example. When you're answering... When you're asking the model about who is the current president of the United States, you want the model to be able to tap into the up-to-date information.
- 6:33
So through pulse training, we've been able to optimize the model behavior to be able to address these and to decide when the external information is needed in the first place.
- 6:48
Sometimes it isn't. Sometimes the pre-trained knowledge is enough.
- 6:53
Then operate the retrieval system smoothly to be able to run search queries successfully, retrieve the information,
- 7:03
hopefully the most accurate one, and then use that information as a grounded context for the conversation that the model is having with the user.
- 7:13
We optimize all of this for you, the model behavior, so that you don't really have to think about it. It's really good at it out of the box, but it was hard work. [laughs]
- 7:25
Our major focus was working on citations. We're big on cite-citations. We believe that allowing the user to verify where the information comes from and whether it's trustworthy, it's really important.
- 7:39
So we're spending extra time to make these citations very fine-grained and thanks to that, you can experience low hallucination and reliable context use.
- 7:52
We tested Command R and R+ on some standard RAG datasets like KILT, and they exhibit best-in-class performance. They're small enough to be affordable, but powerful enough to cover a lot of your use cases.
- 8:09
They have a great balance of token efficiency. And
- 8:15
to achieve this level of performance, normally you would, you would have to line up a big pipeline of LLMs.
- 8:22
We've also heard from you that creating a UX and UI for RAG and tool use is super painful.
- 8:31
It's not a small feat, and we know it firsthand because we've spent considerable amount of time working on it ourselves.
- 8:40
We're really proud of it at the moment. I think it has everything a modern UI, modern chat UI needs to have. So you're able to have a conversation history, you're able to have fine-grained citations, you're able to upload documents there, you're able to plug it into different types of tools.
- 9:04
So spending so much time on it and knowing how much you're struggling either way, we decided that it's going to be a good idea to open source the UI, and that's what we did in April 24th.
- 9:17
I feel like not many people know about it, but our UI is out there and you can download it and start building with it.
- 9:26
So this is a Toolkit repo. That's how we call it. It has plug-and-play components and source code for an interface app that we've built with Next.js. It has a small SQL database for conversation history.
- 9:42
There is a model component which lets you customize how you're accessing Command R models. You can do it via cloud providers. You can do it via Cohere platform. You can do it locally.
- 9:54
You can do it via Hugging Face. Your pick.
- 9:57
Then there is retrieval component, and here you can acstomi-- you can customize access to tools and data sources. Out of the box, we've built an example data retriever buil-built off of LangChain.
- 10:11
It has document upload and it's using web search, but honestly, you can add support for any tools and any data sources that you're interested in.
- 10:23
Lately, we've been focused on optimizing tool use, particularly in the enterprise context. That's our game.
- 10:33
It's kind of extension of this RAG formula I mentioned earlier where we began by training the models to be really good with vector databases and retrieval systems, and then it naturally progressed into broader tool use.
- 10:49
Training the model to use any tools and ideally in a zero-shot context. That's kind of our ideal scenario that we're working towards.
- 11:02
Tool use comes in two flavors. There is single-step. It's really useful for situations where you have a single action to be performed or a set of independent actions. It could be searching for a document or sending out an email.
- 11:23
Multi-step, on the other hand, it's really good for scenarios where you have to carry out a sequence of actions with each action building on top of the previous ones.
- 11:35
So in the same example, it would be searching for that document, being able to compare it against another document, creating a summary of that comparison, and then sending it out via an email.
- 11:50
That's possible with multi-step tool use today. In sequential reasoning in multi-step, you want the system to be able to reflect and correct errors if there are any on the way, and we are teaching the models to retrieve the information many times over from these different data sources.
- 12:10
It's kind of a loop to be able to do that.
- 12:14
You know this behavior from the term agents. Most of the time when people use the term agents and multi-step, they mean the same thing. It's essentially a scenario where software is performing a sequence of actions with each action building on the previous steps.
- 12:33
Last week, we released multi-step API. Super hyped about it.
- 12:37
Um, we want it to be user-friendly and so all you need to do is you need to describe the tools that the model has on their hands, what these tools do, and then some parameters.
- 12:51
After user request is made, the model is going to create a plan, um, and it's going to figure out how to use these tools to fulfill the user request.
- 13:02
And once it calls each tool, it's going to reflect on the contents and it's going to adapt the initial plan if it's necessary. So for example, if the model is calling an API and it returns an error, it's going to automatically retry calling it again and coming up with a new plan.
- 13:21
We've outlined this behavior in this huge multi-step preamble. You can find it on Hugging Face. Um, essentially, it's a massive prompt that explains the model what it needs to do in order to get the job done.
- 13:37
Unique advantage, um, here is the transparency. We've trained Command R and R+ to generate claims that are verifiable through citations and again, big on citations. We really believe that when you can explain which tool has been used by the model for each response, it's going to make a difference and it's going to make the system
- 14:02
better. Command R+ has competitive, uh, performance to Claude Opus, GPT-4 Turbo, but it is three to five times cheaper, so that's a massive difference when it comes to scalability and being able to use it in production.
- 14:22
We test the R family on standard complex reasoning benchmarks and Command R+ is close to or on par with GPT-4 Turbo.
- 14:34
I'm super excited for the upcoming releases. We're gonna keep hammering on the multi-step, um, and yeah, stay tuned. Thanks a lot. [applause] [outro music]