AI Engineer World's Fair 2024
Build enterprise generative AI apps using Llama-3 at 1,000 tokens/s on the SambaNova AI platform
Read the talk
From fast Llama 3 inference to enterprise document Q&A
Build from a single SambaStudio inference call to a retrieval pipeline, following the workshop’s expert routing, memory hierarchy, prompt formatting and PDF search examples.
From a talk by Varun Badrinath Krishna, Petro Junior Milan and Rachelle Mattern
Before you start: To follow the exercises, you should be comfortable with Python, virtual environments and Jupyter notebooks, and have access to SambaStudio LLM and embedding endpoints.
What does fast inference make possible?
What can an enterprise application do with Llama 3 generating more than 1,000 tokens per second? That is the starting question for this workshop: first understand the platform delivering the inference, then connect it to Python and build a document question-answering application. Rachelle Mattern, director of solutions engineering, introduces the platform; principal AI engineer Petro Milan leads the workshop, with senior principal AI solutions engineer Varun Krishna supporting the hands-on work.
The progression is deliberately practical: platform orientation, project setup, a live build and questions along the way. Following the exercises requires a laptop with internet access, a Python environment and the ability to install packages with pip. During the workshop, Discord supplies files, endpoint information and API keys.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Integrating the path from hardware to models
SambaNova approaches enterprise AI as an integrated hardware and software problem. Founded in 2017 out of Stanford, the company brings together the chip-development and startup experience of founders introduced as Kunle, Chris and Rodrigo, including connections to Snorkel and Together AI. Mattern describes a fourth-generation chip and a stack spanning the chip, system and software layers, supporting pretraining, fine-tuning and inference. She also reports more than $1 billion in funding, naming BlackRock, Google Ventures, Intel and GIC among its backers.
For enterprise and government customers, including sovereign-AI deployments, the goal is to reduce the integration decisions between those layers. Mattern describes a platform intended to support more than a trillion parameters. Rather than compete with every component in the ecosystem, SambaNova integrates choices that otherwise have to be reconciled individually: which chip works with which compute system, which operating system supports it, and which models run on that combination. The application team gets a coordinated stack instead of assembling each dependency independently.
Samba-1 Turbo makes the inference side of that proposition visible through a public demonstration. Mattern points to Artificial Analysis comparisons and describes Llama 3 throughput around 1,000 tokens per second, ahead of competing providers in the presented results. These are historical comparisons; the presentation does not give the prompt lengths, concurrency or other workload conditions needed to reproduce the ranking. The live application later supplies a more concrete view of what the speed feels like.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combining broad capabilities with enterprise specialists
The model-management problem begins with two different ways of adopting generative AI. Broad models from OpenAI and model families such as Gemini and Claude offer general knowledge and straightforward integration. But an enterprise also has proprietary information, often accumulated in data lakes and kept separate from the public internet. Turning that information into useful model behavior introduces requirements around privacy, ownership and differentiation. Mattern adds a cost concern: with per-token billing, spending rises as more applications make more calls.
Smaller models such as Llama 3 8B offer another route: pretrain or fine-tune them for a particular enterprise task. That can improve the fit to proprietary data while giving the organization more control over the model. The operational burden then moves to managing many specialists.
| Approach | Useful property | Enterprise burden |
|---|---|---|
| Broad general model | Wide task coverage; easy integration | Proprietary-data fit, privacy and usage cost |
| Task-specific models | Adaptation and ownership | Orchestration, maintenance and repeated fine-tuning |
A legal model, an HR model and a coding model may each be useful, but they do not automatically form one manageable service.
Samba-1 uses a composition of experts, or CoE, to organize those separate models. Here, an expert means a model serving a particular set of tasks. Four controls make the collection usable:
- Single secure endpoint: Put legal, HR and coding models behind one application interface.
- Prompt routing: Inspect an incoming prompt and select the best-suited expert.
- Independent fine-tuning schedules: Retrain and swap models at their own cadence—for example, a finance model annually when policies change and a coding model quarterly as code evolves.
- Model-level RBAC: Grant an application, person or group access to only the models it should use.
The endpoint simplifies integration, while routing, update schedules and access controls preserve distinctions between the models behind it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing, evaluating and invoking experts
Organizations can choose a flexible CoE, selecting the models themselves, or begin with a precomposed Samba-1 collection. Mattern describes the precomposed catalog as 92 experts spanning languages, domains and enterprise tasks. She describes support across seven foundation-model architectures, naming Llama 2, Llama 3, Mistral, Falcon and BLOOM, and also mentions multimodal capabilities through LLaVA, CLIP and DePlot. These examples establish the breadth of the collection rather than a single uniform model architecture.
Mattern reports that SambaNova developed or co-developed 12 of the 92 experts. Examples include models for Thai, Japanese and Hungarian, a text-to-SQL model, and contributions to BLOOMChat for multilingual use. She describes BLOOMChat at the time as the second-largest open-source model. The precomposed catalog is a starting point: organizations can add experts to suit their own requirements.
Evaluation follows enterprise tasks rather than catalog size. SambaNova’s benchmark includes information extraction, text-to-SQL, coding and function calling. Mattern reports meeting or exceeding GPT-3.5 Turbo and GPT-4 on the enterprise comparisons presented. The talk does not establish the datasets, scoring or exact model versions behind those comparisons.
Applications also need more than one way to invoke the collection:
| Invocation mode | Application behavior |
|---|---|
| Routing | Submit a prompt and let the router select an expert |
| Direct selection | Call a known model for a known task |
| Model chaining | Connect model calls in an agentic workflow |
Routing handles selection when the application does not already know the right expert. Direct calls and chaining retain explicit control when it does.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping many models close to compute
The hardware supporting this collection is the SN40L reconfigurable dataflow unit, or RDU. Its memory hierarchy has three tiers: on-chip SRAM, high-bandwidth memory and a larger DDR capacity. The purpose is to keep many models available, then move the models needed for execution toward the faster tiers. Tight connections between the tiers make that movement part of the system’s execution strategy.
Mattern quotes 4 GB of SRAM, 512 GB of high-bandwidth memory and up to 6 TB of DDR. Those spoken totals should not be read as per-chip specifications: the SN40L architecture paper specifies 520 MiB of SRAM, 64 GiB of HBM and up to 1.5 TiB of DDR per socket, and describes an eight-socket node. The talk’s 6 TB DDR figure refers to a configuration whose scope is not established here.
The deployment comparison is about where a large model collection resides. In Mattern’s example, hosting 500 models on GPUs spreads them across systems, so applications must reach the system containing the required model. SambaNova instead uses its DDR tier to retain a large collection within one underlying system and move active models up the hierarchy. She claims capacity for up to five trillion parameters, but gives no precision or configuration assumptions for that total. The useful mechanism is the separation between capacity for stored experts and fast memory for active execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A workout plan, then responses while typing
The live demonstration opens Samba-1 Turbo from SambaNova’s website. Its model selector includes Llama 3 8B and Llama 3 70B, a CoE option, Mistral and SambaNova-trained models such as SambaLingo. Petro selects Llama 3 8B and asks it to create a three-day-a-week workout schedule for an intermediate fitness level. He repeats the request, then opens the performance insights.
For the demonstrated workout-plan response, Petro reports approximately 1,000 tokens/s, 0.09 seconds to the first token and 0.65 seconds total inference time. These three measurements describe different parts of the experience: generation throughput, the wait before output starts, and the duration of the response. He then submits the same prompt to ChatGPT for a visual speed comparison; the demonstration does not provide a matched model or controlled benchmark configuration.
The next mode changes the interaction itself. With real-time generation enabled, the response appears while the user is writing the prompt and changes as the prompt changes. An email-drafting example illustrates the benefit: revise the request and immediately inspect the resulting draft. The same feedback loop can support prompt engineering without repeatedly stopping to submit a finished prompt. The public demo is accessible from a laptop or phone.
An attendee asks whether generation parameters such as top-p can be adjusted. After checking the interface, the presenter qualifies the initial affirmative answer: those controls appear fixed in this UI. The API exercise is where model configuration becomes available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preparing the basic inference notebook
The hands-on work starts with loading environment variables, initializing an LLM and making one Python inference call. The second exercise builds a RAG question-answering application using LangChain, document loaders, E5-large-v2 embeddings, ChromaDB and the Llama 3 endpoint. Both exercises live in the SambaNova AI Starter Kit. The recording uses its 2024 SambaStudio workflow and Python 3.10; the repository’s current setup instructions differ.
For the first exercise, navigate to workshops/ai_engineer_2024/basic_examples. The setup sequence is:
- Clone the starter-kit repository and create
.envat its root. - Add the SambaStudio endpoint credentials supplied for the workshop.
- Enter
basic_examplesand create a Conda or Python virtual environment. - Install the exercise requirements and register the environment as a notebook kernel.
- Open the SambaStudio example notebook in Jupyter or VS Code and select that kernel.
The workshop provides two dedicated endpoints through Discord; participants can use either corresponding set of credentials. The demonstrated Conda environment is named basic_x, with Python 3.10 recommended.
With the environment active, pip install -r requirements.txt installs the basic exercise’s dependencies. A notebook can run in the browser through Jupyter or directly in VS Code. The demonstration uses VS Code, where the selected Python kernel and cell execution timing remain visible alongside the example.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Configuring inference and marking the assistant turn
The notebook imports its libraries, loads .env and initializes LangChain’s SambaStudio wrapper with the endpoint credentials and model configuration. do_sample=False disables sampling, which the presenter describes as deterministic generation; setting it to True enables probabilistic sampling. Temperature and the maximum number of generated tokens are also configurable. Because this is a shared CoE endpoint, the configuration selects Meta Llama 3 8B-Instruct as the expert.
The first inference call uses llm.invoke with the question What is the capital of France? It returns the answer, but also continues with material the user did not request. The next step adds the instruction-message format documented in Meta’s Llama 3 repository: mark the beginning of the text, identify the user turn, end that turn and open the assistant turn. In this raw-prompt workflow, those boundaries tell the model where the user’s request stops and its response should begin.
The same question can then become a reusable LangChain Expression Language, or LCEL, chain. A prompt template substitutes the country, the configured LLM generates the response, and an output parser returns text. Given the initialized llm, the composition is:
python
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate
def make_capital_chain(llm):
prompt = PromptTemplate.from_template(
"<|begin_of_text|>"
"<|start_header_id|>user<|end_header_id|>\n\n"
"What is the capital of {country}?<|eot_id|>"
"<|start_header_id|>assistant<|end_header_id|>\n\n"
)
return prompt | llm | StrOutputParser()
def ask_capital(llm, country: str) -> str:
chain = make_capital_chain(llm)
return chain.invoke({"country": country})
Calling ask_capital(llm, "France") preserves the original question while making the country an input rather than hard-coded prompt text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adding document retrieval before generation
The second exercise lives under workshops/ai_engineer_2024/ekr_rag, with notebook and Streamlit versions. Retrieval-augmented generation supplies the LLM with information from outside its training data. This is useful for proprietary documents or current information that should be available without retraining the model, and it can reduce hallucinations in some contexts.
The pipeline separates document preparation from answering a question:
- Load and parse documents. Use a loader suited to PDF, text or PowerPoint and extract text that downstream tools can process.
- Split the text. Create smaller chunks, with chunk size and overlap as configurable parameters.
- Embed the chunks. E5-large-v2 maps each chunk into a numerical vector.
- Store the index. Keep vectors together with their text and metadata in ChromaDB; Faiss is mentioned as another option.
- Retrieve for a query. Embed the question and find nearby chunk vectors using a similarity metric.
- Optionally rerank. Reorder retrieved chunks by relevance and remove unnecessary material.
- Generate the answer. Pass the question and selected context to the LLM.
The vector search supplies relevant source material; the final model call turns that material into an answer.
Execution crosses a clear boundary. Third-party tools handle loading, splitting and storage. The embedding model can run on the laptop’s CPU or through an endpoint on SambaNova hardware, while the LLM runs on SambaNova hardware. The recorded run uses remote embeddings; a CPU comparison is proposed but does not appear in the completed demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preparing the PDF-processing environment
Reuse the cloned repository, but deactivate the first exercise’s environment before setting up the RAG exercise. The root-level .env now needs two sets of connection details: the SambaStudio LLM endpoint and API key, plus the embedding endpoint and API key. Installation runs from the ekr_rag directory.
PDF processing adds system dependencies beyond Python packages. On macOS, the workshop installs Tesseract for OCR and Poppler outside the virtual environment. It then uses a Python 3.10 environment:
bash
brew install tesseract
brew install poppler
python3.10 -m venv .venv
source .venv/bin/activate
Run these environment commands from the exercise directory. If the shell cannot find Python 3.10, first install it and update the shell’s path, reloading .bashrc or .zshrc as appropriate. After activation, run the exercise’s installation script, install ipykernel and register the kernel for notebook use.
Some workshop laptops encountered NLTK or SSL-certificate issues, for which the presenter points to an additional repair script. Once the dependencies are ready, the notebook can open through Jupyter or VS Code. A Streamlit version is available in the exercise, but the recording proceeds with the notebook.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Indexing the SN40L paper and asking a question
The RAG notebook under ekr_rag/notebooks coordinates supporting modules, including document_retrieval.py and vector-database utilities. Its initial cells establish the kit directory as ekr_rag and the repository directory as ai-starter-kit. Loader selection comes from configuration: the demonstration starts with PyPDF/PyPDF2, with Unstructured available as an alternative.
The input document is the SN40L paper, which describes SambaNova’s hardware, stack and composition of experts. The exercise supplies it through GitHub and also allows a custom PDF. PyPDF loads the document content into a list, then LangChain’s recursive splitter creates the chunks. With chunk size 1,200 and overlap 240, the demonstrated 15-page PDF produces 89 chunks. The spoken configuration does not specify the size units, so 1,200 should not be interpreted as a token count.
Next, config.yaml selects SambaStudio as the embedding type for RDU execution. The endpoint accepts batches of 1 or 32 chunks; a batch size of 32 processes that many chunks together. This embedding service is a standalone endpoint, so its CoE setting is false. The resulting vectors are indexed in ChromaDB.
The presenter initially expects vectorization to take about ten seconds, then restarts the notebook and reruns the cells. After the restart, he reports four seconds to embed the demonstrated document. That is the timing reported for this run, rather than a CPU-versus-RDU comparison.
The QA chain now embeds an incoming question, retrieves the top three chunks and supplies them with the question to the LLM. This run does not use a reranker. Asking “What is a monolithic model?” produces a response the presenter describes as instantaneous; the notebook displays the answer alongside a source link and retrieved text. The visible result connects generation back to the document used to answer the question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The harder test: retrieving from a table
The last test moves beyond a definition to information in a table. The presenter opens the paper’s table relating operation intensity to fusion level and submits a prepared question to see whether the PyPDF-based pipeline can recover the relevant information. He judges the returned answer correct.
For more complicated PDF tables, the recommended next step is to change the configured loader from PyPDF to Unstructured. This is where the workshop ends its implementation: a working notebook question-answering flow, followed by a concrete adjustment to document extraction. Fast inference makes the answer appear quickly; the loader still determines what information reaches the retrieval pipeline in a usable form.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original paper explaining SN40L's memory hierarchy, operator fusion and composition-of-experts experiments.
Application examples and setup documentation for building with SambaNova. Current instructions differ from the recorded workshop.
English retrieval embedding model with examples for query and passage encoding, normalization and input-length handling.
Further reading
Contemporary vendor account of the Llama 3 8B milestone and its proposed uses in multi-call applications.
Meta's Llama 3 inference examples and instruction-message formatting guidance.
Read the complete timestamped transcript
- 0:00
[upbeat music] All right. Well, hi everyone. Thanks so much for joining us today.
- 0:17
Um, so today I get the great opportunity of introducing SambaNova and some of our capabilities around reaching over [REDACTED:password] thousand tokens per second using Llama 3. Today, I'm gonna spend [REDACTED:password] little bit of time getting you oriented around SambaNova, some of the capabilities that we provide as an AI platform, and also some of the underlying technologies that
- 0:40
are providing the means of achieving some of the accomplishments, like [REDACTED:password] thousand tokens per second. Before I start jumping into the content, I wanna take the opportunity to introduce some of my colleagues really quick.
- 0:52
So joining me today, I have Petro Milan, who is [REDACTED:password] principal AI engineer. He's gonna also be leading our workshop component and be here as we're getting hands-on with the technology.
- 1:03
I also have Varun Krishna, who is [REDACTED:password] senior pr- uh, senior principal AI solutions engineer, uh, joining me as well. And I'm Rachelle Mattern. I'm our director of solutions engineering.
- 1:15
I serve our global customer base at SambaNova.
- 1:19
So before jumping into our content, I just wanna cover what are we gonna be talking about today. So we're gonna start off with [REDACTED:password] little bit of housekeeping, uh, talk [REDACTED:password] little bit about the prerequisites.
- 1:28
We are gonna be getting hands-on today, and also introduce our Discord channel, which is where we're gonna be communicating with one another and also sharing some content and information that you're gonna need, such as files and, and other, uh, like API and s- uh, uh, keys and things like that.
- 1:43
Um, I'm gonna talk about SambaNova, just get you oriented around who are we and how are we achieving [REDACTED:password] thousand tokens per second. Um, then I'm gonna pass it off to Petro.
- 1:53
He's going to, uh, talk you through our workshop today, how we're gonna get hands-on, get you oriented around, uh, uh, how to get started, uh, and we're actually gonna go through [REDACTED:password] live build, and we'll support you along the way.
- 2:06
Uh, we'll also spend some time around questions, uh, just in case that, um, you have any questions about SambaNova, our technology, or anything that we're doing in the hands-on component.
- 2:15
So before we get started, I wanna talk [REDACTED:password] little bit about prerequisites. So first of all, we are gonna be using our laptops, so, uh, hopefully you have them today.
- 2:23
Uh, and we're also going to need internet access to get through our workshop. Uh, we're gonna be working in [REDACTED:password] Python environment, um, so hopefully we have Python set up and ready.
- 2:32
Um, and also we're gonna be needing to install some packages, uh, through pip. Um, we are gonna be working in Discord, so if you don't mind, uh, I'm gonna give everybody just [REDACTED:password] second to hopefully get on Discord and, um, and join our channel.
- 2:47
So I'll just give everyone [REDACTED:password] quick moment to, to get set up.
- 2:54
Once you're set up, just maybe give me [REDACTED:password] thumbs up so I'll know.
- 3:05
Question?
- 3:06
What is the Wi-Fi password?
- 3:09
The Wi-Fi password.
- 3:09
Uh, what, what is it?
- 3:10
TimeToBuild, [REDACTED:password] T-
- 3:14
What was the last-
- 3:15
... uh, [REDACTED:password] T, [REDACTED:password] [REDACTED:password]. The general one, not the one that says speaker.
- 3:18
Okay. Yeah.
- 3:19
AI engineer.
- 3:20
Okay. So just to repeat, uh, it's AI engineer. The password is [REDACTED:password] and [REDACTED:password] [REDACTED:password] [REDACTED:password].
- 3:28
No, no, [REDACTED:password] T, [REDACTED:password] T, [REDACTED:password] I.
- 3:30
Okay. All it-it's, it's all capitalized, each word, but concatenated.
- 3:34
The first letter.
- 3:35
Okay. Good? Um- Were you able to join? Yes. Okay. Awesome. Cool. Just wanna make sure there's no problems. Awesome. We'll give everyone [REDACTED:password] couple minutes who just got Wi-Fi access.
- 4:15
All right. Um, in case you haven't got [REDACTED:password] chance to set up, maybe just take [REDACTED:password] picture of this really quick. We'll also go back to it, um, before we kick off the hands-on component.
- 4:24
Um, but want to spend [REDACTED:password] little bit of time just getting you oriented around us at SambaNova. So SambaNova, we are [REDACTED:password] full stack AI platform, and we've existed since twenty seventeen.
- 4:37
Uh, we were founded out of Stanford University. So two of our, uh, co-founders are actually Stanford professors, uh, Kunle and also Chris. Um, both of them, and including Rodrigo, each bring [REDACTED:password] unique perspective to our founding team, uh, including previous, uh, startups that were, uh, acquired and had, uh, uh, various exits.
- 5:00
Uh, also building other AI startups that are pretty well known in the industry, such as Snorkel, Together AI. Um, and also [REDACTED:password] really, uh, depth of experience around building out hardware and chips.
- 5:11
Um, we are, uh, building the full stack from the ground up, so that means we build our own chip, um, I'll go into that [REDACTED:password] little bit layer- later, but also all the way through the system level and the software layer.
- 5:23
Um, we're on our fourth generation st- uh, chip. Um, and we have built [REDACTED:password] entire stack that allows you to both fine-tune models, pre-train models, and also deploy those models with really high performant inference.
- 5:38
Um, we have achieved over [REDACTED:password] billion dollars in funding from various well-known names like BlackRock or Google Ventures, Intel, GIC. Um, and so we are really well established to solve the challenge around [REDACTED:password]- building and deploying AI hardware.
- 5:57
So what exactly are we targeting and what exactly are we trying to solve for? Our customer base comes from [REDACTED:password] wide variety of enterprises and also to government organizations.
- 6:08
So we're really aiming to deliver capabilities that can help service the enterprise-grade AI capabilities that companies and governments require to deliver, um, unique and differentiated capabilities and also things like sovereign AI.
- 6:23
Um, our ca-- our underlying platform is delivering the means to actually achieve the scale of [REDACTED:password] trillion parameters, uh, plus. Um, and we're doing that through delivering full stack capabilities.
- 6:37
When I say full stack, um, m-many folks have probably utilized many of these technologies on, uh, this slide. We're not necessarily trying to compete with every single layer, uh, involved here.
- 6:47
But what we are trying to do is ease the process of getting started and ease the journey along the way. And so instead of having to make [REDACTED:password] decision at every single one of these layers, we are actually integrating things into [REDACTED:password] very seamless experience from deciding on what chip is going to work with, uh, what compute,
- 7:07
what compute and chip is going to work with what operation systems, what operation system works with what models. You don't have to actually make each of these decisions and know that they have to integrate with one another.
- 7:19
We actually create this very seamless experience along the way, um, where everything kind of orchestrates and works very nicely.
- 7:28
And what we're doing at the end of this is actually delivering the capability to not only fine-tune, but deliver really, really fast inference capabilities. So we're gonna demo [REDACTED:password] little bit of this later, but, uh, recently we released [REDACTED:password], [REDACTED:password], [REDACTED:password] demo, um, that you can actually go try live, and we'll do so later, called Samba-1 Turbo.
- 7:47
This is, uh, exceeding world records around, uh, speed of inference, especially when it comes to Llama Three. Um, and you can see that through some of the metrics that were re-recently published, um, through, uh, Artificial Analysis.
- 8:01
Artificial Analysis did [REDACTED:password], um, [REDACTED:password] benchmarking exercise to understand the different capabilities, uh, the speed at which they are able to deliver inference throughput, um, for [REDACTED:password] thousand tokens per second across various hardware providers.
- 8:17
And what you can see is that we are far exceeding as [REDACTED:password] platform, um, the th-the throughput capabilities compared to some of the oth-other providers out there.
- 8:28
And so I wanna kinda talk [REDACTED:password] little bit about how are we enabling such speed. Um, so when it comes to the underlying technology, many of us have experienced some of these trends in the industry.
- 8:39
Many of us got started with what you see on the right-hand side of the slide, which is the large monolithic model. This is the likes of like an OpenAI, for example, or [REDACTED:password] Gemini or [REDACTED:password] Claude.
- 8:51
And many of us started our LLM journey or our generative AI journey using some of these technologies. But along the way, many, uh, many other capabilities in the open source community started to pop up, specifically these smaller models.
- 9:04
And these smaller models allowed us to do things like fine-tuning and actually adapting some of these models to our enterprise data and our enterprise requirements. And so that really started to take off in the industry, and each of these started to see different pros and cons associated with them.
- 9:21
When it came to large monolithic models, when we actually started to put these into practice when it came to enterprise applications, one of the reasons many of us leaned into this is because of the broad capabilities that the likes of OpenAI brought, right?
- 9:34
And also the ease of integration in terms of OpenAI into the actual platform itself. It's super easy to manage, and it also was trained on the internet's data, and so it can handle [REDACTED:password] lot of different things.
- 9:46
But when it came to actual enterprise applications, enterprises have unique capabilities required to deliver on some of the use cases and challenges they're trying to solve for. For example, enterprises have unique data that they, they, you know, als-oftentimes segregate from the internet, or most of the time segregate from the internet, um, that's proprietary to them.
- 10:09
And oftentimes they have spent the last ten years trying to actually aggregate that data into the likes of data lakes and other, um, uh, kind of centralized capabilities. And so now how do you actually transform that into AI capabilities that you can leverage?
- 10:24
That was very difficult when it came to large monolithic models. The other challenges that we saw is that many started to become very concerned about security when it came to OpenAI.
- 10:33
They wanted to preserve, uh, data privacy. They wanted to also own and use the model as [REDACTED:password] differentiation for themselves. Um, and then also as you start to see more and more adoption, that cost just started to skyrocket.
- 10:47
Um, OpenAI and, and [REDACTED:password] lot of these closed source models charge on [REDACTED:password] per token rate. And so as you start to utilize more and more LLMs, uh, and utilize LLMs, uh, more heavily, the cost just starts to go up and up and up and up, and it's really, really hard to control.
- 11:03
On the other hand, when it came to adopting the smaller expert models or the smaller, uh, open source models like the likes of Llama 3-8B that we'll talk about later, um, we were able to address some of the enterprise accuracy concerns by actually pre-training, fine-tuning these models to adapt them to the enterprise requirements.
- 11:23
Um, while we were doing this at [REDACTED:password] smaller scale to address like different capabilities and tasks that we needed to solve for in the enterprise, they weren't trying to solve like [REDACTED:password] broad set of tasks and also adopt [REDACTED:password] broad set of general knowledge.
- 11:38
Um, thus manageability became [REDACTED:password] little bit challenging because now we have all of these kinda like micro models that we have to orchestrate and have them work together. Um, but we were able to solve for some things like security and model ownership.
- 11:51
Uh, data privacy and data ownership. Um, but again, because we had so many of these and we had to fine-tune each one of these, the cost also became very challenging.
- 12:00
So what were-- what are we trying to solve for, uh, through our capability of Samba-1? We're trying to bring the best of both of these, uh, paradigms together, um, to deliver the, the capabilities of each, um, in [REDACTED:password] very simplistic way.
- 12:18
And the way we actually deliver this is through four core capabilities. First of all, we take all of those expert models behind the scenes. So let's just say we fine-tuned [REDACTED:password] model for our legal purposes.
- 12:31
We fine-tuned [REDACTED:password] model for our HR purposes. We fine-tuned [REDACTED:password] model for coding capabilities. Each of those have different groups and tasks and use cases that are going to consume those.
- 12:44
But we want to really ease the experience of having to integrate those into the application. So we put all of those behind [REDACTED:password] secure single endpoint. And so you only have to interface with one endpoint to gain access to all these various models.
- 12:59
Now we need to determine how are we gonna actually use those or consume those various models behind the scenes. So now we need, uh, capabilities around orchestration. So one of the other capabilities we're delivering as [REDACTED:password] part of this is around, um, routing.
- 13:13
So we're delivering the ability to determine based off of an incoming prompt, what is the best-suited expert behind the scenes to solve that prompt? And we're doing so through [REDACTED:password] router.
- 13:25
We're also bringing the means of dynamically fine-tuning. Every expert is going to have [REDACTED:password] different cadence in which fine-tuning is going to make sense. Um, so maybe your finance model gets adjusted at annually when your policies are updated.
- 13:40
But maybe your coding model, because you're pushing code so regularly, needs to get updated on [REDACTED:password] quarterly basis. And so you wanna actually be able to schedule your fine-tuning jobs and adjust and swap these models at the rate at it-- which it makes sense to actually retrain these models.
- 13:57
And lastly, you have [REDACTED:password] bunch of models under the scenes. Not every application or group is going to or should be able to access each of those models. So now you need to figure out [REDACTED:password] way to, uh, manage the access controls for these.
- 14:10
So what we're also deliver-- delivering as [REDACTED:password] part of this capability is model-level RBAC. So you can actually determine this application or this person or this group of people should be able to access this set of models.
- 14:23
And it allows you [REDACTED:password] ton of efficiency from [REDACTED:password] computation and management operations standpoint with the security and fine-grained control that you need to actually manage acc-access to these different models and data under-underlying these models.
- 14:38
So within Samba-1, we have two ways of delivering this. We have something called [REDACTED:password] flexible CoE that allows you to kind of determine exactly what models lie under the hood, and we also have [REDACTED:password] pre-composed version of Samba-1 composition of experts.
- 14:53
So our pre-trained, um, or our pre-configured, I should say, or pre-composed version of this model has ninety-two underlying experts. And when I say experts, I'm really referring to [REDACTED:password], [REDACTED:password], [REDACTED:password] specific model, um, that can bring different capabilities, uh, associated with it.
- 15:10
And within those ninety-two experts, we have [REDACTED:password] broad range of languages that are covered within those models, [REDACTED:password] broad range of domains, and [REDACTED:password] diverse set of tasks that are very, very relevant to the enterprise.
- 15:24
All of these models are supported by seven different foundation model architectures, including Llama 2, Llama 3, Mistral Falcon, Bloom, and even some multimodal capabilities such as, like, Lava, Clip, Deep Lot, um, that is kind of starting to support some of the multimodal trends that are coming.
- 15:42
And of all these ninety-two models, we are actually delivering and partnering with organizations to, um, create and contribute back to the open source community. So out of the ninety-two experts, twelve of them are actually ones that we've helped, uh, e-either develop ourselves or co-develop with organizations out there, including some of the language capabilities that we've delivered, such
- 16:05
as models that can support things like Thai or Japanese, Hungarian. Um, and through that, we've developed [REDACTED:password] lot of experience on how to actually adapt models to different languages.
- 16:16
Um, we've also, uh, created [REDACTED:password] model for text-to-SQL capabilities and delivered, uh, really, really good results through that model for text-to-SQL. And lastly, we have contributed back to Bloom Chat.
- 16:30
Uh, it's the second-largest open source model, um, and, uh, it's also bringing [REDACTED:password] lot of the multilingual capabilities.
- 16:38
So organizations essentially, as they're constructing these different composition of experts, can add as many expert models as they need. As I mentioned before, while we have that pre-configured composition, this is really intended for you to be able to construct exactly what you need in terms of models under the hood.
- 16:56
So you can add as many as you need.
- 17:00
So our end goal with this is to really be able to bring the capabilities that enterprises need to, um, handle the diverse set of use cases and capabilities they need to solve their problems.
- 17:11
And so one of the things that we've created along the way to measure ourselves against this is an enterprise-grade AI benchmarking set. And this is really tailored to understand our capabilities against the best-in-industry models across various, uh, enterprise-specific tasks and domains that are needed.
- 17:30
Things like information extraction, that's where many, many enterprises are starting, um, but also [REDACTED:password] broad set of capabilities like text-to-SQL, coding, function calling. Um, and we're measuring ourselves against GPT-3.5 Turbo and GPT-4.
- 17:45
Um, and what we can see is along the way, um, we're meeting or exceeding, um, the capabilities that, that OpenAI is bringing.
- 17:55
Um, but alongside the capabilities, you also need to orchestrate these models. I talked [REDACTED:password] little bit about this before, but one of the deliverables as far as Samba-1, uh, from [REDACTED:password] product standpoint is we're bringing routing capabilities so that you can actually, uh, take [REDACTED:password] prompt, determine what is the bex-- best-suited expert, and then route to that.
- 18:13
Um, there's also scenarios where you may need to do something outside of routing. You may actually just wanna directly call [REDACTED:password] specific model or, in the case that is popping up really, really regularly now with agentic AI, you may need to do model chaining.
- 18:28
So that's another capability that we're bringing as [REDACTED:password] part of the product suite for Samba-1.
- 18:37
And so how are, how are we actually uniquely set up to deliver the composition of experts capability, um, and also the really fast inference speed that we're gonna see, uh, briefly?
- 18:48
So I mentioned earlier, but we are [REDACTED:password] full stack, uh, AI platform, but we also build our own chip. Our chip, we call that an RDU, or [REDACTED:password] reconfigurable data flow unit.
- 18:59
So instead of [REDACTED:password] GPU, we'll refer to our chip as an RDU. And our current version of that chip is called SN40L. And what is unique about SN40L is actually the memory structure of this chip.
- 19:12
So our chip supports [REDACTED:password], [REDACTED:password] three-tiered, uh, memory architecture. So we have our on-chip memory, and then we have our high-bandwidth memory, and then we have [REDACTED:password] huge, uh, memory capacity in DDR.
- 19:26
And this really allows us to store [REDACTED:password] ton of models, um, and have those models be swapped in and out of various tiers within the memory, um, to achieve really strong performance and also really, really efficient compute utilization.
- 19:45
So what does this look like? Um, as I mentioned before, we can store up to five trillion parameters on DDR. So that's like if we were to store like three to four OpenAI's on [REDACTED:password] single chip.
- 19:59
And then as we need to actually execute those models, they can move up the memory stack. Um, and this allows us to, one, take into consideration what models need to be used at what time, and then two, do so in [REDACTED:password] really performant way because the, the network connectivity between these three layers is really tight.
- 20:23
So just to kinda go into some of the specs, um, our on, on-chip SRAM, uh, has four gigs, uh, our high bandwidth memory has five hundred and twelve gigs, and our DDR has up to six terabytes.
- 20:35
So, um, lots of memory to work with. And when we think about how does this compare to what you experience in the GPU world, when you think about the number-- if you wanna host [REDACTED:password] really large amount of models, when you have to do so on [REDACTED:password] GPU, you're basically needing to work with the memory that, that
- 20:55
is, uh, the, the GPU has on chip. But with us, because we have the DDR component, we're able to store [REDACTED:password] huge amount of models in [REDACTED:password] single system and keep it coupled with the other memory, uh, tiers.
- 21:08
And so with GPUs, you're often-- if you wanted to host five hundred different models, you're gonna have to actually, uh, align those models to various systems, and you're gonna have to basically call the various system to actually access each of those models.
- 21:21
With us, it's gonna be one underlying system, um, because we're able to store, again, up to five trillion parameters.
- 21:32
So, um, I'm gonna hand it over to Petro. Uh, what we're gonna do next is actually, uh, get the chance to get hands-on.
- 21:38
Um, and yeah, thanks, uh, Rachelle, for the presentation. Um, so what we will be doing next is [REDACTED:password], uh, demo of our Llama 3 and Samba-1, um, Turbo. And after this, um, we move to the, uh, hands-on portion of the workshop.
- 21:54
Um, so if you wanna try out our Llama 3 endpoint, um, so you can go to our website, uh, sambanova.ai,
- 22:05
and then, uh, click on, uh, Samba-1 Turbo. So this is where you can access, um, our, um, chat infra. And here you have, um, options to select, um, various models, you know, Llama 3, um, 8B, 70B, our CoE, Mistral, and even some of our, um, in-house, um, models which we've trained ourselves like, you
- 22:30
know, [REDACTED:username] or others. Um, so yeah, we'll do [REDACTED:password] demo of Llama 3, um, 8B, and I'll ask it, uh, the following question. Um, so, you know, create [REDACTED:password] three-day-[REDACTED:password]-week workout schedule for intermediate fitness level.
- 22:46
Um, let, let me actually, uh, redo it again. You can see, you know, it gets, um, instant, uh, response. And then for the, uh, performance metrics, so you can see the, um, insights here.
- 22:59
Um, so few things, uh, to note. Um, so of course, we have [REDACTED:password] pretty high, uh, throughput, which is, uh, [REDACTED:password] thousand tokens, um, per second. But that's not only, um, the end of the story.
- 23:11
You know, we also have [REDACTED:password] pretty small, uh, time to first token, which is basically the, um, input inference time of point zero nine seconds. And we also have [REDACTED:password] pretty small, um, end-to-end, uh, total inference time of point six five seconds, right?
- 23:27
So with our full stack, um, platform, we can achieve high throughput and very small, um, inference time. And if just want to do [REDACTED:password] comparison, let's say with, um, ChatGPT, uh, for instance, um, if you ask, uh, the same question, just copy it.
- 23:47
Um, yeah, you can clearly see the, uh, difference in speed.
- 23:57
And the, uh, other, uh, cool thing which, uh, we built, so we have this, uh, real-time, uh, option where as you write your prompt, you can instantly see the model's response, and as you change the prompt, you can also see the change in the response.
- 24:16
Yeah. So let's say, I don't know, "Hi, I
- 24:22
want to write an email about blah, blah, blah," right? So the point here is that, you know, with this, um, real-time option, you know, you can do, um, real-time chatting, and this can be helpful, for instance, if you're drafting emails or even if you want to do some, like, real-time, um, prompt engineering, let's say.
- 24:45
So yeah, that's it for the, uh, Samba-1, uh, Turbo. I can give maybe [REDACTED:password] few minutes for folks to try it out before we move on to the, um, hands-on.
- 24:54
Um, so again, you go to, uh, our website, sambanova.ai, and then you click on Samba-1 Turbo. Um, so you can do it from your laptop or from your cell phone.
- 25:08
Yes. Yes, question. Can we tweak the generation parameters like top P- Yes, yes, yes, yeah. Uh,
- 25:20
let me see how to do it on this, uh, UI.
- 25:27
Okay. I think, uh, from this UI it seems it is fixed but, um, in the hands-on once, uh, you will be calling our endpoint, um, from the API key, uh, we will be changing some of the, um, configs as well, so.
- 25:42
Yeah. Any other, uh, question?
- 26:10
Yeah. Were you able to, uh, try it out? Uh, okay. Great.
- 26:14
Okay. So I think, um, yeah, we can move on to the, um, hands-on, uh, portion. Um, so what I'll do first is just, um, introduce what, uh, um, I'll be talking about in this part, and then, uh, we'll dive in, um, to the, uh, hands-on.
- 26:31
So yeah, we prepared two, uh, exercises, um, for you today. So the first one is [REDACTED:password] basic, um, example to get you started. Um, so here, um, I'll be showing how you can load, um, environment variables, set up the, uh, SambaNova API key, um, initialize the LLM, and do [REDACTED:password] simple, um, uh, inference call in Python.
- 26:55
And the, uh, second, um, example is [REDACTED:password] more practical one, so we will be, uh, building and deploying [REDACTED:password], uh, Q&[REDACTED:password] system, um, with RAG for, um, enterprise search, uh, using our platform.
- 27:09
And we will also be using, um, other libraries and packages like, you know, LangChain, um, various data loaders, uh, E5 large V2 embedding, um, ChromaDB vector store, and of course, the Llama 3, um, endpoint, which runs at the speed of one thousand tokens per second.
- 27:30
So yeah, let's start with the basic, um, example. And, um, if you wanna f-follow along, so you can, uh, yeah, go to Google, write AI Starter Kit, uh, SambaNova, and then, uh, click on this link, or you can just, um, write this, uh, URL.
- 27:52
So this is our, uh, starter kit, uh, repo. Um, we have [REDACTED:password] collection of open source, um, examples on, um, GenAI apps. And, um, yeah, once you get there, yeah, you can go to, uh, Workshops,
- 28:10
AI Engineer 2024, um, Basic Examples. So I'll go over the README and then do [REDACTED:password] live demo, um, of the work of this, um, exercise, and then I'll give you, uh, some time to, uh, try it out.
- 28:25
So, um, yeah, first, um, you'll need to clone, uh, this, uh, repo here, and then, uh, you'll need to, uh, create [REDACTED:password], um, .env file, um, in the repo root, uh, directory.
- 28:40
So this is where, um, we will be, uh, specifying the, uh, SambaStudio, um, API key. Um, yeah, so let me show you how this is done. Uh, it's gonna be...
- 28:54
Yeah, I guess with the two mic, uh, I have to find [REDACTED:password] way. [clears throat]
- 29:08
Yeah. Yeah, so I've already cloned, um, the repo. Yeah, the repo. And then, uh,
- 29:20
at this level, this is where you will need to create the, uh, .env file. So vi what... In your case, yeah, you'll have to do touch .env, and then,
- 29:35
yeah, you can add, uh, the- these information here. So for the first, um, hands-on, that's the, uh, um, only thing that you'll need. Um, and you can access or copy those from our, um, Discord, uh, channel.
- 29:52
So, yeah, if you go to Discord- Events.
- 30:09
Um, yeah, so you can copy either, um, of these keys. So we have two dedicated, you know, endpoints, um, for this workshop.
- 30:18
All right. And, uh, once we finish setting up the .env, uh, the third step is basically, you know, installing the, um, packages. So for this one, you can either do it with, you know, Conda or [REDACTED:password] Python environment.
- 30:33
Um, so first, yeah, we will go to the basic examples, um, repo. So just CD and then the repo path. Then you can create, um, [REDACTED:password] Conda, uh, environment.
- 30:46
So I would recommend using, uh, Python, um, three point ten. And then you activate your, uh, Conda environment, um, and here we name it basic_x. And then, yeah, you just, um, install the requirements with the, uh, pip install minus r, um, requirements.
- 31:04
And then you can use this, um, line to just, um, link the kernel to your, um, notebook. Okay?
- 31:13
So if I go to my terminal. Yeah, so we'll go to, uh, workshop,
- 31:27
AI engineer, basic examples. Okay, so this is where you can create the Conda. So I've already done it, um, beforehand, so I'm just going to activate the, uh, environment.
- 31:40
So... Yeah, and this is the, uh, requirements file.
- 31:59
So we only have like [REDACTED:password] few packages that is needed.
- 32:06
Yeah, and that's it, uh, for the installation. Um, so once this is done, yeah, you should be able to open, uh, the notebook. And again, you can do it, you know, from the terminal, which will route you to [REDACTED:password] browser or, um, uh, through VS Code.
- 32:22
So if you want to do it through the terminal, you just write Jupyter Notebook and then
- 32:30
the name of the notebook. So it's going to be, uh, example, uh, with, uh, SambaStudio.ipy, uh, oneB. I'm actually going to do the demo via VS Code just 'cause I can show you the, uh, timestamp.
- 32:46
So I already have this, uh, set up.
- 32:52
Okay, so you know, we are in the basic examples repo and then example with Samba Studio. And again, if you're doing it, uh, with VS Code, just make sure that you have the, uh, kernel set up.
- 33:02
Um, so this is [REDACTED:password] pretty basic, um, script. So we will first, uh, let me just restart it. Yeah. So yeah, we will be loading,
- 33:16
uh, the libraries. Um, so we [REDACTED:password]-actually have [REDACTED:password] wrapper with LangChain. So this is, um, where, uh, we will be, um, initializing and calling our endpoints. The second step is to load the, um, environment variables.
- 33:32
So these are actually the information which we added in the, uh, .env, uh, file.
- 33:38
And then, uh, we will initialize the, um, LLM. So yeah, we will be using the SambaStudio, um, wrapper, and, uh, we set the, um, SambaStudio API key, and then we specify the model config.
- 33:53
So I had [REDACTED:password] question earlier about, um, the model configs. Um, so this is where we can, uh, set this up and play with this. Um, again, I think most of you are familiar with these configs.
- 34:04
So, you know, do [REDACTED:password] sample. If you set it to false, this is basically [REDACTED:password], um, deterministic output. If you set it to true, then it becomes, uh, probabilistic.
- 34:13
You can change the temperature and also the max tokens, um, to generate. And also, this is basically our CoE endpoint, right? So we have one endpoint through which you can actually call, um, different models.
- 34:25
And in this case, uh, we set the expert to, uh, Meta Llama 3, um, 8B instruct. Okay, so I'll be running the, uh, the step. And yeah, we have now our model loaded, and now we're, uh, good to go.
- 34:40
So, um, I'll first show you how you can do an inference call using [REDACTED:password] simple, um, invoke method in LangChain. Um, so just write llm.invoke and then add your prompt.
- 34:54
And the prompt here is, um: What is the [REDACTED:password] of, uh, France?
- 35:01
And what you'll, uh, notice is it gives the right answer, but also, um, give you other stuff which you didn't ask for. And this is [REDACTED:password] common thing with open source, um, models because when you ask [REDACTED:password] prompt, you need to include the, uh, the special tags, right?
- 35:19
So in the case of, uh, Llama 3, you can get it from [REDACTED:password] Meta model card. So you'll have to actually, in the prompt, add these, uh, special tags or tokens.
- 35:30
Um, in particular, you have to let the LLM know that this is the beginning of text, and this is where you will insert the user query, right? So and then let the LLM know where it, uh, needs to, um, answer.
- 35:44
And once you have the special tags, um, inserted, um, now you should be able to get the right response. So this is the first way to do the, um, inference call.
- 35:53
The other way is to do it via [REDACTED:password], um, LCL in LangChain. So basically, um, you can use the LCL to connect, um, [REDACTED:password] prompt template with LLM and an output parser.
- 36:05
And LangChain has very Um, templates that you can use. So, um, we are asking the same prompt. It's just that the main difference here, we are adding the country as [REDACTED:password] placeholder, and then when you prompt the model, you can actually specify the value of this country, right?
- 36:24
Yeah. So that's it for the basic, um, example, and as I said, this is just to get you started. So, um, yeah, we can spend ten minutes for you guys to try it out.
- 36:36
Um, me, Varun, and Rachelle will be here to help and, um, yeah, then we can move on to the second, um, exercise.
- 36:47
I could use [REDACTED:password] hand if you're available. [laughs] Um, it might just be returning [REDACTED:password] string. Oh, that's it. That's it. It could
- 37:02
be... It could be, um, issues. Yeah, I'll move around if also people have questions. It could be our environment.
- 37:08
I think on the data loading question, it's open now. Never mind.
- 37:13
Do you think it's connected with
- 37:15
Oh.
- 37:17
Okay, cool. I think, uh, many of you were able to try out this simple, um, exercise. But yeah, we'll move now to the, uh, second, uh, one. Varun, do you wanna? [microphone feedback]
- 37:31
Um, yeah. Yeah. So, um, as I said earlier, this is going to be [REDACTED:password], uh, Q&[REDACTED:password], uh, system, um, with RAG.
- 37:51
And, uh, if you wanna follow along, um, again, from the same repo, uh, go to workshop, AI engineer twenty twenty-four, and then EQRAG. And like the previous exercise, I'll go through the README, do [REDACTED:password] live demo of the installation and the run setup, and then, uh, give you some time to, um, try it out.
- 38:13
And the, uh, app here, we have two versions of it, one with [REDACTED:password] Jupyter Notebook and the other one, um, with, uh, Streamlit, which is [REDACTED:password] UI-based. And before I, uh, jump into the hands-on, just wanted to give [REDACTED:password] brief, um, overview of what RAG is.
- 38:33
Although I'm sure many of you already know, uh, this concept, but just for, um, completeness. So yeah. So RAG is [REDACTED:password] technique that, um, we can use to supplement, um, LLM with, um, additional information from, um, various sources to improve the model's response.
- 38:51
And RAG is very helpful, um, if you want to use an off-the-shelf LLM to ask [REDACTED:password] question, um, beyond its, uh, training data, or if you want to, um, have the LLM access to up-to-date information without, um, retraining it.
- 39:08
Also in RAG can help reduce, um, hallucinations in some, uh, contexts. And [REDACTED:password] typical, uh, RAG workflow, um, consists of,
- 39:21
yeah, the, uh, following steps. So we first have, um, document loading and parsing. So this is where, um, we can use [REDACTED:password] data loader to actually load the data into [REDACTED:password], um, digital text that we can edit and format.
- 39:39
Um, and, you know, various data loaders are, um, available depending on the, um, extension of the file you- you're using, so in [REDACTED:password] PDF, text, um, PowerPoint. After this, we have [REDACTED:password] splitting step.
- 39:52
So this is where, um, we will be splitting the document into, um, smaller chunks. And, you know, the chunk size and the overlap, all of these are, um, hyperparameters.
- 40:04
And the, uh, next step is, um, vectorization. So this is where, um, we will be using [REDACTED:password], um, embedding model like E5 large V2 to map each, uh, chunk to [REDACTED:password], [REDACTED:password] numerical vector.
- 40:18
And, um, we can store, you know, the vectors along with the content and the metadata in [REDACTED:password] vector store like Faiss and ChromaDB. And today, um, we will be using, um, ChromaDB, which is, um, open source.
- 40:33
And again, the whole, uh, goal of this embedding is that it allows us to do, um, like semantic similarity and, uh, semantic search. And in the retrieval step, um, this is where we ask, uh, the question, which is going to also be embedded into the vector, um, space.
- 40:53
And then we have [REDACTED:password] retriever, which is going to, um, retrieve the closest, uh, chunk vectors to the query vector according to some, uh, similarity metric. And we can also add [REDACTED:password] re-ranker, which can re-rank the retrieved, uh, chunks, um, per, uh, relevance and also remove some of the, um, unnecessary chunks.
- 41:17
And the last step is basically Q&[REDACTED:password], um, generation. So this is where you provide the LLM, um, with the query and the final retrieved, uh, chunk to get the, uh, grounded response.
- 41:31
And, um, yeah, maybe also would like to precise that, um, in this exercise. So the... We will be using third-party tools for document loading, splitting, and, uh, storage. For the embedding model, um, you can either run it on CPU or on our hardware, and we'll be doing both to show the differences.
- 41:51
And for the LLM part, this is gonna be done on our, um, hardware.
- 41:57
So yeah, that's it for the, uh, overview, and I think, yeah, we can move on to the, uh, README. Um, yeah. So we first, uh, clone, uh, the repo.
- 42:07
I think if you've done the other, um, exercise, then you don't have to do, um, this step. Um, same thing, um, after this, we will set up the, um, environment, um, variable.
- 42:19
Um, so for this one, we will be using, uh, SambaStudio, um, API for the LLM, and also, uh, we will be using [REDACTED:password] embedding, um, API as well. So both are available on Discord.
- 42:36
Yeah. Let me show you, uh, in the terminal.
- 42:42
Yeah, and actually, I would also recommend to deactivate your previous, um, environment.
- 42:51
Okay, so we go to the EQRAG, uh, repo. Um, yeah, actually, for the dot and file, this one, you'll have to put it, um, at this level, the AI starter kit, okay?
- 43:03
So for this one, we will need the embed endpoint and API key and also the, uh, SambaStudio, um, endpoint and, and, and API key.
- 43:15
All right. Then we go back to the, uh,
- 43:20
EQRAG folder, and then we are ready to go with the installation. So here we will be needing more packages. Um, so first, um, Tesseract, uh, this is our OCR, um, data extractor.
- 43:34
So let's say you're using [REDACTED:password] Mac, um, just run brew install tesseract, and this one you can do it outside your local, um, environment. It should take you, you know, [REDACTED:password] few minute to install.
- 43:46
And you also need, um, Poppler, if you don't have it already. Um, you can just do brew install, um, poppler.
- 43:55
And then, yeah, we will need to set our, um, virtual environment. So since this is [REDACTED:password] more complicated, um, exercise, I would just recommend to use, um, the, you know, default option.
- 44:07
So the, uh, Python environment with [REDACTED:password] Python, um, three point ten. So if you don't have, uh, Python three point ten, um, let's say on [REDACTED:password] Mac, you can install it using this, uh, command here, and then you can add the path to your shell, like bashrc or, uh, zshrc, um, using this command here.
- 44:30
And then you can just source your, uh, shell file.
- 44:35
Okay? And then, yeah, we will, uh, go to the, uh, repo, if you haven't done already, um, create your Python environment. Um, so again, if you added this step here, then your laptop should recognize the Python three point ten, then minus m benv, and then the name of the environment.
- 44:56
You activate that environment, and then you run the install script. So this should take, I would say, you know, five minute if you have, um, good internet, and once this is done, you also need to install ipykernel and also, um, link your kernel to your, um, notebook.
- 45:17
So when we tested this on different laptops, um, you know, some folks were having also sometimes NLTK and SSL certificate. So you might also need to, uh, run this script here.
- 45:36
Yeah, it's in my bag, basically. Yeah. So let's activate the, uh, uh, Conda, the Python environment.
- 45:51
Yeah, and this is the, uh, requirements file. And as, and as, as you can see, right, we have, uh, more packages here.
- 45:59
And then, um, yeah, this is the file which you may need to also, um, run as well.
- 46:07
Yeah, and once this is set up, that's all you need to, uh, run. Yeah, that's fine. I can do it.
- 46:15
Yeah, uh, the, the notebook. And as I said earlier, right, so we, uh, have the app in [REDACTED:password] notebook and in [REDACTED:password] Streamlit. So for the, uh, notebook, um, again, right, you can open it from the terminal or, uh, from VS Code.
- 46:31
Yeah, let me do it, um, from VS Code.
- 46:40
Yeah, so you'll go to the EQRAG, uh, repo, notebooks, and then, um, rag_lcl.ipynb. And this is going to be our main, uh, script, which is using actually, you know, files and modules from other, um, files.
- 46:56
So in particular, uh, we will be using the, um,
- 47:02
the document retrieval dot py and also, uh, some files, uh, from the vector DB, which I'll explain in, in more details. All right, so, um, let's go maybe first over the structure of the notebook.
- 47:17
Um, so we first, um, import, uh, the libraries and set the required path. So you don't have to do, um, anything at this point, but, yeah, just know that the kit directory, this is the absolute path for your EQRAG, and then the repo directory, this is the absolute path for the, um, AI starter kit.
- 47:38
So let's run this, uh, repo. And then for the, uh, document, uh, loading and splitting, um... Yeah, so we added, um, you know, various, um, data loaders, you know, like PyPDF and Unstructured.
- 47:56
And which, uh, data loader you want, uh, you can set this up, um, in the config file, which is, um, here. So yeah, I'm just gonna do [REDACTED:password] test with PyPDF two for now, but we can switch to other, um, data loaders, um, afterwards.
- 48:15
And, um, yeah, for the experiment, I will be using the SN40L, um, paper. So this is an archive paper which we, um, recently, uh, submitted. So this contains, like, information about the stack, the hardware, and our, um, uh, CoE.
- 48:31
Um, I can show you, uh, the paper as well. Um, we also have it in, uh, GitHub, but you can also, um, upload your own, um, PDF.
- 48:40
And yeah, you'll have it to put it under Data, temp, and then...
- 48:50
Yeah, this is the, uh, paper that I'll be using, uh, uh, for the demo.
- 49:00
All right. And, uh, let's go back to our VS Code. So this is where, um, you know, we will be using PyPDF to actually load the content, um, into [REDACTED:password] list.
- 49:12
And then, um, we are using the, uh, recursive, uh, splitter from, um, LangChain. So what I'll do first is go into the notebook, and then we can go into the functions in more details if you are, um, interested.
- 49:25
Yeah, so let's run this step. Yeah, and, um, in the config, um, so I set the, uh, chunk size to twelve hundred, and then the, uh, chunk overlap to two forty, but again, you can change those configs.
- 49:48
So for this fifteen-page, um, PDF, we end up getting, um, eighty-nine chunks.
- 49:54
The next step is the, uh, vectorization and storage. So this is where, um, we will be using the embedding model to map each, uh, chunk to [REDACTED:password] embedding vector.
- 50:05
And as I said earlier, right? So we can actually run the embedding model either on CPU or on RDU. So RDU is basically our AI chip. So if you want to do it on RDU, then you'll have to go to the config.yaml file, and then, uh, set the type to, uh, SambaStudio.
- 50:24
And then, uh, batch size, so this one you can have it either one or, uh, thirty-two. So thirty-two means that we are actually processing thirty-two chunks at the same time.
- 50:34
And this is [REDACTED:password] standalone, um, endpoint, so yeah, the CoE is set to, um, false. And I'll show later, um, if you want to run the endpoint, um, for the embedding on your laptop, uh, how you can change the configs, um, for that.
- 50:49
All right. And then, yeah, let's run the, uh, vectorization. And after this, we're actually storing or indexing the embedding vectors, um, into the ChromaDB vector store.
- 51:02
Think, uh, this should take around, uh, ten seconds. Let me see what's happening.
- 51:11
I just restarted. Yeah, yeah. Yeah, always good to restart the notebooks. I'll just go over the steps again. [clears throat]
- 51:31
Okay. Yeah, so it took, you know, four seconds to embed the whole thing. And, uh, yeah, this is where, um, we will initialize our, um, QA chain.
- 51:47
Um, so again, we have different, uh, wrappers and classes, which I can go over it in details, um, afterwards. But for now, let's just execute, uh, the cell. Yeah, now you're ready to go, um, ask [REDACTED:password] question.
- 52:01
And then what happens is, right, through this QA chain, the question gets embedded to the vector, uh, space. We retrieve the, uh, top K chunks. So in this experiment, yeah, we have this set to three, since I won't be using [REDACTED:password] re-ranker, but I can also show you how to use the re-ranker.
- 52:21
And then, yeah, the question and the context are provided as, uh, co- basically, uh, context for the LLM to get the answer. So
- 52:32
yeah, what is [REDACTED:password] monolithic model? And you can see that, right, the response is, um, instantaneous.
- 52:38
Yeah, so that's it for the, um, uh, experiment. Uh, let's maybe try ask it [REDACTED:password] bit more complicated question. So if I open the PDF again... Uh.
- 52:58
I think, yeah, there was [REDACTED:password] table in the PDF. [clears throat]
- 53:06
Yeah, like this is [REDACTED:password] table, you know, showing, um, operation intensity versus fusion level. So yeah, let's see if PyPDF is able to, um, uh, retrieve, uh, some of the information from this table here.
- 53:20
So I already have the, uh, questions prepared. Let's try to access those.
- 53:28
Yeah, so yeah, it got the response, right? So four, ten point four, basically. Um, and again, if you end up having, like, more complicated, um, tables in PDF, um, in this case, I would recommend to switch to the Unstructured, um, data loader.
- 53:40
And for this, yeah, all you have to do is just go to the config file and then, um, change, uh, PyPDF to, um, Unstructured.
- 53:49
So yeah, that's it for, uh, this second, um, exercise. Um,
- 53:56
I can go into more details about each, uh, function if you're interested, um, or have you try it out first, and then we can maybe, uh, come back and then, um, go over the, the, uh, functions.
- 54:10
Um, yeah, so do you wanna maybe try it out first, I guess? Okay, great. [upbeat music]