- AI Engineer World's Fair 2024 Schedule

Attending with the VP Pass? Join your fellow leadership track attendees in an exclusive closed-door breakfast in the View Lounge on the 39th floor with breathtaking views of the city! PS - VP pass holders have exclusive access to the View Lounge from 8:00am - 3:00pm on the 26th and 27th, so feel free to stay during the main stage keynote or pop back in any time within those hours.

Benjamin Dunphy, co-founder of the AI Engineer World's Fair and managing partner at Software 3.0, LLC, welcomes attendees to day 2 and makes a few key announcements.

Today's leading generative AI applications have workloads that span high performance GPU compute, CPU preprocessing, data-loading, and orchestration — often spread across a combination of Python, C++/Rust, and CUDA C++ — which increases the complexity and slows down the cycle of innovation. This talk explores the capabilities and power of the Modular Mojo programming language and Modular Accelerated Xecution (MAX) platform, which unifies CPU and GPU programming into a single Pythonic programming model that is simple and extensible. This results in reduced complexity and improved developer productivity, and streamlines innovation. We'll walk through CPU and GPU support with real-world examples, providing details of how AI application developers can use MAX and Mojo to define an end-to-end AI pipeline and overcome the complexities.

In this keynote, Antje explores how generative AI is transforming the landscape of software development, enabling developers to innovate like never before. She will showcase the latest advancements in AI and demonstrate the powerful capabilities of generative AI tools available on AWS. You will learn how to harness these tools to accelerate your development processes, enhance creativity, and build robust, AI-driven applications.

Explore Anthropic's latest strides in large language models, emphasizing enhanced reasoning and multimodal capabilities. We'll showcase how these advancements translate into powerful developer tools, APIs, and best practices for building sophisticated, RSP-aligned AI applications.

Lets cut through the buzzwords and get to the point. In this talk we'll unveil the abilities of Gemma and Gemini models, and discuss how to design and build killer apps with LLMs. We'll cover the essential tools, techniques, and best practices you need to know, without the fluff. We'll explore how developers can use Google Cloud's Vertex AI platform to build safe, secure applications with open models. Using Google's Gemma model, we'll dive into an end-to-end example of model discovery, tuning, and deployment as part of real-world application.

A main issue with agents is that they are not reliable enough. It's easy to build a prototype of agents, but it's still a lot of engineering work to get them to perform reliably. Infrastructure (and tooling) can help make this easier - but thinking about and understanding your application's "cognitive architecture" will always be necessary. This talk will discuss new infrastructure (and tooling) that still gives you full control of the "cognitive architecture" of your application.

Evals & LLM Ops: Software development is rapidly evolving with the introduction of agentic workflows. However, the non-deterministic nature of these workflows makes ensuring reliability and predictability rather difficult. To effectively manage these challenges, teams need a specialized stack that guarantees precision, safety, and reliability.

This talk will explore how Chegg builds scalable, reliable, highly personalized agentic solutions for over 7.7 million students globally. We will dig into best practices for bringing solutions to production and how Chegg uses Galileo’s platform to improve quality, unlock new capabilities, and accelerate development.

The next frontier of LLMs is creating agents to take action and connect to external knowledge sources. However, it can be difficult ensuring that models respond reliably, intelligently and quickly. Fireworks founding PM Ray Thai walks through the benefits of open-source function calling models and provides a tutorial on Fireworks’ new, Llama 3-based Firefunction-v2 model. Firefunction-v2 offers fast, reliable function-calling for real-world assistant use cases by enhancing Llama 3’s function calling ability while preserving its chat and instruction-following capabilities.

CodeGen & Dev Tools: GitHub Copilot was introduced as “Your AI pair programmer” in January of 2021. Since then, we have been adding more and more capabilities to increase developer productivity and happiness. We’ve gone from simply leveraging AI to generate code to explaining existing code, generating unit tests, propose fixes for bugs, making code robust / secure, summarizing pull requests, having conversations tailored to your organization’s repositories, providing answers to your questions based on your organization’s knowledge base and so much more! Let's explore how to get started with GitHub Copilot, its ever growing list of capabilities, and how to make the most out of the tool.

As long as you have a web browser that can access public repositories on GitHub.com then you have all you need. To interact with GitHub Copilot we will be using GitHub Codespaces which you can connect to via a web browser.

https://gh.io/aifair-workshop

Programming languages like Java have been in the market for almost 30 years, resulting in hundreds of thousands of lines of legacy code that need maintenance and upgrades. In this talk, we'll explore how genAI can revolutionize the process of upgrading legacy code. You'll learn about 10 essential tools and tips to efficiently refactor, modernize, and enhance your existing Java codebase.

AI models enable some new and powerful capabilities, but we still need to think of them as functions in larger logical systems, rather than a sort of end-state for software. AI-integrated programs should be built just like any other program: by relating small semantic tasks to each other in order to automate a larger task. Right now, the main problem preventing developers from running 10-20 AI modules on a single user request is an infrastructure one, and that’s what we’re solving. Large companies like Google have built internal infrastructure and SDKs to enable this, but we’re offering this capability to any developer.

Ionic Commerce is on a mission to integrate AI agents into the economy, starting with ecommerce. We have created a connective tissue between commerce and large language models designed specifically for agents which allows agents to navigate the digital world effectively and execute complex transactions without navigating UX/UI designed for humans. We are further enhancing the data layer to support decision-making and bringing the transaction layer into the decision process. Ionic Commerce is laying the foundation for AI agents to seamlessly participate in the economy.

Discussions with VPs of AI

All rise! The honorable LLM Judge presiding. On the docket today, many cases of AI Engineers building with LLMs, without the ability to iterate, evaluate and improve their products.

The complexity and scarcity of deploying GPUs can bring AI development to a standstill. What if there was a better way to train, fine-tune, and serve models on accelerated compute infrastructure entirely in Python? See how during this session where we fine-tune 20 Llama models without doing any infrastructure work. Startups and enterprises can finally gain unprecedented speed and agility to build, iterate, and deploy anything that they can imagine, from multi-agent, multi-modal AI applications, to digital twins for real world simulation. What used to take weeks with dozens of best-in-class engineers can now be accomplished in hours from a single notebook.

Ask anything about the state of Retrieval Augmented Generation with the CEO of LLamaIndex.

Psst! Wanna learn how to build an AI model that punches way above its weight? Beats models 4 times its size and competes with the models from Meta, Google and OpenAI? In this talk I’ll spill the beans on how I pulled it off, and how you can too. I’ll share my unexpected journey that led to the creation of Moondream, a tiny open-source vision language model that kicks ass. I’ll share my journey and the technical hurdles that I faced along the way. I’ll also explain why small models are the future of AI. Join me for a story of accidental innovation, the democratization of AI, and how sometimes, thinking small can lead to big results.

Not Chatbots again! Not today, thank you!

Imagine a world where AI does more than just chat—it thinks, solves, and acts. Enter the realm of Minecraft, where virtual landscapes become the perfect proving ground for the next generation of autonomous agents.

This deep dive session takes you on an adventure beyond the ordinary, showcasing how we've engineered an AI entity capable of reasoning, problem-solving, and executing tasks in Minecraft.

Leveraging the power of Amazon's Bedrock, Lambda, SNS, and a suite of AWS serverless technologies, paired with the latest Claude 3 model from Anthropic, we reveal the full potential of AI beyond the confines of chatbots.

Discover how these serverless solutions empower our AI to interact with, adapt to, and transform its environment in real-time, providing a glimpse into the future of autonomous agents in both virtual and real-world applications. Join us as we navigate the intricacies of serverless AI architecture, demonstrating its practical implications, block by block.

Presentation will go over current and future hardware requirements for next generation frontier models

"Many failed AI products share a common root cause: a failure to create robust evaluation systems. Evaluation systems allow you to improve your AI quickly in a systematic way and unlock superpowers like the ability to curate data for fine-tuning. However, many practitioners struggle with how to construct evaluation systems that are specific to their problems.

In this talk, we will walk through a detailed example of how to construct domain-specific evaluation systems."

"everything is converging. ai models can now natively understand the same modalities as you. at the same time, AR technology is getting to a place where we are able to experience virtual worlds like never before. how did we get here, where is this going, and how do we build intuitive experiences in the age of unbounded products."

Agentic system design is a rapidly evolving and intellectually fascinating field, with huge potential for transforming how software is used across industries. Unlike traditional software, agentic systems rely on non-deterministic and oftentimes difficult to predict decision making. Taking inspiration from fields like robotics, cybernetics, and biology, we can start to develop intuitions around how to build systems that are ~more~ reliable than the sum of their individual stochastic parts.

It’s been 25 years since Intel broke the 1 Ghz speed barrier for a general purpose microprocessor. What does 1 Ghz AI inference mean? What application capabilities will this enable? How will we achieve it? Groq’s mission is to build the AI computer that will power the next generation of software.

Zapier is the #1 workflow automation platform for small and midsize businesses, connecting to more than 7,000 of the most popular work apps. We were also one of the first companies to build and ship AI features into our core products. We've had the opportunity to work with Braintrust since the early days of the product, which now powers the evaluation and observability infrastructure across our AI features.

We’ll walk through a couple of our projects – AI Zap Builder and Zapier Copilot – and how we built them from the ground up with evals and observability in mind. Hopefully, you can walk away with a few of our learnings from operating these projects at scale while systematically improving their performance.

What are the big breakthroughs required to bring realtime multimodal intelligence to every device in the world? This talk describes the work we're doing at Cartesia on bringing realtime models to life on an entirely new technology stack. I'll describe new research ideas that we developed over the last few years — state space models — that are enabling us to build audio models that are cheaper, faster and higher quality than state of the art approaches.

"This talk is about a simple idea: Everyone should be able to use AI to make cool stuff.

When I started making crewAI, a framework to build AI Agents, I didn't know just how much it would change the way I work. This isn't just a chat about building AI Agents; it's a story about how these tools helped us grow our business and how they can help you too.

I'll share our adventure with crewAI, showing how we used AI to solve problems, save time, and come up with new ideas. It's a journey of discovery and making things better with AI, showing everyone that they can use AI in their work and projects."

State-of-the-art machine learning models are increasingly using techniques like mixture of experts that enable larger-scale models to be trained more efficiently by distributing layers of the model across multiple neural networks. This sparse distribution of model state puts increasing pressure on cluster-level networking while training. At Crusoe Cloud, we’ve built a high-performance InfiniBand network that's designed to provide the highest possible performance for these state-of-the-art training techniques. We use a “rail-optimized” design, reducing the number of hops between any set of GPUs in our cluster, accelerating all2all performance, and reducing training time. Learn more about how to utilize Crusoe Cloud rail-optimized networks to accelerate your training workloads.

Executive briefing and private Q&A for VPs of AI

Attending with the VP Pass? Join your fellow leadership track attendees in an exclusive closed-door lunch in the View Lounge on the 39th floor with breathtaking views of the city! PS - VP pass holders have exclusive access to the View Lounge from 8:00am - 3:00pm on the 26th and 27th, so feel free to pop in any time within those hours.

A fresh look into the latest announcements in GitHub Copilot!

Multimodality: In this session, we'll do an in-depth exploration of how developers can use Gemini multimodal generative ai models to build custom applications and agents which can simplify and automate repetitive tasks. This session will dive into the latest advancements in Multimodal Generative AI, featuring Gemini 1.5 Flash and its capabilities to process and understand diverse data types including text, code, images, PDFs, audio, and video. Discover how to seamlessly integrate these multimodal models with external tools using Function Calling and Reasoning Engine, enabling the creation of sophisticated, end-to-end automated solutions. Whether you are looking to automate tasks or build innovative AI-driven applications, this session will equip you with the practical skills and knowledge to leverage the full potential of Google Cloud's cutting-edge AI technologies.

RAG & LLM Frameworks: The core web search infrastructure and algorithms haven’t changed in a decade. Exa is training foundational search models and embedding the web to change that. In the talk, we’ll discuss how we built and optimized a vector database from scratch to work with billions of vectors using SIMD, novel quantization methods, matryoshka embeddings, and new data formats for traditional filters. The combined system is a new type of database that achieves state of the art performance on complex queries, and we’re excited to present its inner workings.

With the launch of generative AI and GitHub Copilot, there are both new skills for developers to learn and workflows to adopt. As this space continues to grow, we want to explore where the future will take us and how to best position ourselves to take advantage of new technologies. Let's discuss what workloads are best offloaded to Copilot, how to get the most out of the tool, and the new features coming to GitHub Copilot.

GenAI models get larger every day and the infrastructure required to train them only gets more complex and harder to access. Navigating a sea of cloud providers or trying to rapidly scale up on-prem compute can be difficult at best -- just when you find the perfect solution at a fair price, you learn the lead time is too long or the hardware is not as reliable as advertised. If only there was a way to access exactly the amount of training compute power you need precisely when you need it. We are excited to introduce 1-Click Clusters, a new way to perform distributed training at scale. 1CC is a flexible way to access GPU clusters of all sizes for weeks or months at a time. Whether your project needs a quick burst of extra power or might scale up and down over time, you can reserve compute as simply as booking a hotel room. We will discuss features and how things work beneath the software to help you learn what to look out for and how to maximize your training efficiency.

GPU & Inference: Haven't you heard? GPUs are the new gold. If you're strapped for compute you're not alone. And if you already have some H100s on hand, you're probably looking for ways to get more bang for your buck. Enter OctoStack, the new Generative AI inference stack that you can deploy to any environment. In this talk, we'll tell you exactly how we got OctoStack to outperform DIY deployments, including VLLM, by 10X for high-concurrency use cases. We've spent a decade optimizing the heck out of the whole stack, and now we're ready spill all our secrets. If speculative decoding, custom kernels and CUDA graphs pique your interest, you're our kind of nerd. Come check it out!

RAG & LLM Frameworks: In this talk, we'll talk through how developers are getting the most out of Atlas Vector Search through intelligent data modeling of unstructured data. This enables both iterating on your search problem and operating your search system in a way that seamlessly syncs with your original source data. We will also discuss how to consider the capabilities of embedding and chat completion models so that you can effectively ingest data into MongoDB in a way that makes it easily searchable.

Join this session to learn more about how GitHub is incorporating GenAI across GitHub Advanced Security (GHAS) in addition to the core platform. Sarah Khalife, Principal Solutions Engineer, will provide an overview of key features including code scanning, secret scanning, and supply chain security, highlighting the latest AI-powered capabilities.

Training and fine-tuning models depends critically on how you construct your dataset. Part art, part science, we’ll share with you practical lessons in dataset construction at Character AI and how to build a data platform to support rapid iterative refinement of training data. For LLMs, data scale is much larger and workloads are more diverse. This is especially true for multimodal datasets. To deal with these challenges, we'll show you how LanceDB is used in production to solve many pain-points around the storage, management, and querying of large scale AI data.

The founders of Zeta Labs will provide an update on the progress in autonomous agent development, discussing its alignment with the timeline for replacing humans in their current jobs. Additionally, they will highlight how these agents represent a crucial milestone towards achieving Artificial General Intelligence (AGI)

Last set of discussions with VPs of AI

Simple instruction prompting? Few-shot examples? Fine-tuning? How do you decide when to do each? In this talk we’ll discuss the emerging concept of the GenAI maturity curve and define the steps along it. We’ll also discuss the concrete triggers you should watch for to indicate that you’re ready to move up to the next step.

At OpenPipe we’ve helped customers fine-tune thousands of models (and redirected thousands more who hadn’t hit the trigger points). Let’s talk about how to speed-run that journey!

Gemma is Google’s family of lightweight, state-of-the art open models built from the same research and technology used to create the Gemini models. Learn about the types of Gemma models available, and how to customize them to excel at new tasks.

Join us for a thought-provoking session on the future of AI evaluations, led by Sheila Gulati, founder of Tola Capital, and Nischal Nadhamuni, cofounder and CTO of Klarity. We'll delve into the importance of rethinking and innovating evaluations at this pivotal time, highlight the limitations of current methods, and offer practical steps for the AI community. As AI continues to play a significant role in our society, the need to develop the right evaluation tools is more important than ever. Learn how to create smarter, fairer, and more effective AI systems that align with our collective values and societal goals. Sheila will cover the thematic trends of this industry and Nischal will show how his company Klarity applied them as they have grown to serve the largest companies in the world and achieved their recent $70M Series B! This is a unique opportunity to engage with leading experts and shape the future of AI—don't miss out!

We will explore how we might use AI models to combine sounds, images, and videos and create new learning activities that support critical, creative and counter-factual thinking. Showcase demos using Generative AI (Vertex API, AI studio, LearnML) to detect misconceptions in math learning, support creative coding, create simulations to test scientific hypotheses, curate and sumarize research & identiy blind spots in one’s thinking. The goal of our demos will be to demonstrate how mights we use multimodal ai to support learning activities where the agency is balanced between learners and AI and we encourage people to discover the limitations of these technologies while developing a critical use of it, know if, when and how to use it when learning new skills.

Voice AI technology has evolved significantly in recent years, transitioning from simple command-response systems to more sophisticated natural conversational agents powered by Large Language Models (LLMs). This progression in voice AI is being driven by advances in core technologies such as foundational AI models, dramatically transforming interactions between humans and machines. Notable improvements include advanced automatic speech recognition and breakthroughs in human-like speech synthesis, all integrated with the deep language comprehension provided by LLMs. These developments have culminated in powerful, autonomous systems that interact through spoken language exclusively. During the session, Scott Stephenson, Founder and CEO of Deepgram, will explore the fundamental principles and best practices for crafting responsive, realistic, and captivating AI agents. He will delve into topics such as natural language processing and the design of multimodal interactions. Attendees will gain insights into the principal design challenges and key considerations involved in developing voice agents capable of managing complex conversations and providing context-sensitive responses on par with human speakers.

With nearly two-thirds of enterprise developers planning production deployments of large language models this year, LLM evaluation has never been more important. LLM evaluation is also an area where confusion reigns, starting with ambiguity around what “LLM evals” even means. Often, LLM model evaluation – quantifying general fitness (i.e. on the Hugging Face leaderboard) – is conflated with task-specific LLM system evaluation. And while many foundation model providers offer their own evals, AI engineers building LLM systems designed to plug into many models or tools need a way to objectively evaluate both different foundation models and their own systems with rigorous techniques. In this session, Arize AI founder Aparna Dhinakaran will release research onstage and walk attendees through real life examples of building an LLM Eval from scratch. This session will build on multiple research pieces that have garnered millions of views across social platforms, diving into techniques to build out robust LLM evals and ultimately gain a better understanding of the limits of LLM capabilities. Want to build your own LLM task evals for a specific use case leveraging open source tools? Want to see the latest research on which foundation models your company should be using for specific use cases? You won’t want to miss this session!

Generative AI powers the next generation of real time applications. The key to success of modern application development in the Gen AI era is secure, latency-sensitive and low cost LLM serving solution, which Firework’s enterprise grade deployment provides. Fireworks AI accelerates innovation through its SaaS platform of low latency inference and high quality fine-tuning of 100+ models, across the state of the art LLMs, image/video/audio generation, embedding and multimodality models. These advantages are delivered through Fireworks' proprietary FireAttention technology, 4x-15x faster than the OSS alternatives. To bring the totality of knowledge together, Fireworks tuned their own FireFunction model to integrate hundreds of models and API calling together. Fireworks' adoption is the fastest in the industry and it also enables a software stack capable of extracting the most across different hardware and deployment options.

Over the last 2 years the capabilities of LLM have been developing at a rapid pace and R&D organizations in almost every industry are expected to determine their "AI strategy" while balancing existing non-AI customer needs and a tighter financial environment. For some, recent AI developments might even pose a disruptive shift for their products and company, but how do you balance rapid prototyping and exploration of these disruptive technologies with catering to your current user base to avoid the Innovator's Dilemma? Should you add AI to every product team's roadmap or form a dedicated AI team? How do you experiment with entirely new AI endeavors without risking your current customer base? In other words, how do you cook with fire without burning down the kitchen?

In this discussion round, Dominik Kundel will share how Twilio has been tackling some of these challenges, their learnings and what changes/iterations they have been going through to address those learnings, followed by an open discussion for others to share their learnings.

How to build the world's fastest voice AI bot:

• Self-host speech-to-text, LLM inference, and text-to-speech all together in the same container/cluster.
• Route audio over the internet using WebRTC and edge networking.
• Configure timings for voice activity detection, phrase endpointing, and other parts of the pipeline to optimize for latency. (There are trade-offs to doing this!)

Here's a LLama 3 voice bot that has voice-to-voice response times of ~500ms.

We used @DeepgramAI 's STT and TTS for this bot, and everything is hosted on @cerebriumai 's serverless GPU infrastructure.

https://x.com/kwindla/status/1806129490411900940

Meet Circuitrix, the sassy robot who schedules meetings, Hootie McHootface, the whimsical owl who transcribes calls, and Zarplo, the sycophantic code-reviewing martian -- AI agents with personalities as unique as their workflows.

Is giving AI agents human-like traits merely a marketing gimmick? Or does it fundamentally transform and improve user adoption and interaction? We'll explore the impact of anthropomorphizing AI agents on user engagement, and share insights from letting end-users completely customize their own agents’ personalities.

Discover the engineering practices required to develop agents with unique personas, the technical challenges debugging a product with infinite(!) customizability, and why you may never say the phrase "works on my machine" ever again. Get ready to rethink the relationship between AI and users in this deep dive into the world of personality-driven agent design.

Join Alexander Kvamme and Arjun Bansal as they recount Echo AI’s journey rolling out its conversational intelligence platform to billion-dollar retail brands. They’ll discuss navigating LLM accuracy issues as well as what needed to happen at the application and infrastructure layers in order to successfully deploy at enterprise scale.

Meet Devin, a state-of-the-art AI software agent that helps developers save time and achieve more. Scott Wu, co-founder and CEO of Cognition AI, demos its capabilities and shares some of the lessons that he and his team have learned while building Devin.

description tba

RAG & LLM Frameworks: This panel will feature a 30-minute discussion featuring founders and engineering leaders from four companies leading the way in AI right now — MongoDB, Cohere, LlamaIndex, and Together AI. We will have 10 minutes toward the end for the audience to ask questions to the panelists. During the discussion, panelists will share insights into the following:

• Origins of the company, identifying the problem space
• What an ideal AI stack looks like today
• Main use cases of their products
• Favorite community projects that use their products
• Challenges with taking LLM applications from prototype to production
• What will the next few years in AI look like?</u>

If 2023 was the year of GenAI prototypes, 2024 is the year these apps go into production. In this session we’ll demo how Azure AI Search combines the best RAG capabilities for GenAI apps at any scale, without compromising cost or performance. We will share the latest product updates, including increased storage limits, advanced capabilities for detailed retrieval pipeline control, built-in support for content beyond text, and re-ranking improvements to boost quality of responses.

Generative AI is changing how work gets done. In this session we'll talk about coding assistants, and how they fit into the software development lifecycle. Coding assistants can teach new concepts, accelerate developer research tasks, improve existing code, automate boilerplate code creationg, and even help develop complex brand new features from scratch

Join us and hear from the Snorkel AI team about their experience and lessons learned using Azure AI infrastructure powered by NVIDIA GPUs. Learn about how Snorkel Researchers were able to run experiments quickly from small projects to large-scale distributed jobs on multiple GPUs reliably and with full monitoring mechanisms.

RAG & LLM Frameworks: This talk will share up-to-date learnings from the evolving field of knowledge graphs: from live case study examples at many of the world’s largest organizations, to the latest technologies & integration patterns, to how to think about knowledge graphs more generally and where they add value. You will come away with an understanding of how knowledge graphs can play the critical role of left brain to complement the statistically-derived LLM + Vector right brain. You will hear how these two techniques can work together to create more reliable and powerful solutions, in an enterprise landscape where many GenAI use cases call for the use of both hemispheres.

The future we are building towards.

Special double-feature closing keynote from the 6 authors of the hit O'Reilly article on Applied LLMs.

Join GitHub CEO Thomas Dohmke for the closing keynote with a deep dive on Copilot Workspace, and what’s ahead as he talks on AI’s coming agentic wave.

Evals & LLM Ops: LLM-powered autonomous agents combine (1) Tool calling, (2) Memory, and (3) Planning to autonomously perform tasks. While they hold tremendous promise, agent reliability has been a barrier for large-scale deployment and productionisation. We’ll cover ways to design and build reliable agents using LangGraph, which can support diverse self-corrective applications such as RAG and code generation. But, just as critically, we’ll cover ways to use LangSmith to test your agents, examining both agent's final response as well as agent tool use trajectories. Collectively, we’ll talk about three types of testing loops you can incorporate into your agent design process - at run time, pre-production, and for production monitoring.

Open Models: This workshop will be split into 3x one hour blocks:

1. How to analyze & fix LLMs - how to find and fix bugs in Gemma, Phi-3, Llama & tokenizers
2. Finetuning with Unsloth - continued pretraining, reward modelling, QLoRA & more
3. Deep dive into LLM technicals - hand deriving derivatives, SOTA finetuning tricks

It's recommended you have Python with Pytorch and Unsloth installed (or use online Google Colab / Kaggle). College level maths and programming would be helpful.

CodeGen & Dev Tools: In this hands-on workshop, learn how Large Language Models (LLMs) can significantly improve your productivity as a software developer. Drawing from three years of experience using LLMs in every aspect of his work as a principal engineer, the presenter will share practical insights and techniques that go beyond simple prompts and off-the-shelf tools.

Through a series of interactive exercises and real-world examples, participants will learn how to:

1. Combine classic design patterns, such as domain-specific languages and declarative programming, with transformer-based models
2. Identify the most effective ways to use LLMs for writing software, beyond simply having them generate code
3. Create systems that use LLMs to write software that instructs LLMs to write more software

This workshop is designed for engineers and professionals working in software-related fields who want to use LLMs to solve concrete problems and improve their workflow. Attendees will gain valuable insights and practical techniques that they can immediately apply to their projects, regardless of their current level of expertise with LLMs.

The presenter, a passionate advocate for "pragmatic programming" with LLMs, has made over 6,000 GitHub contributions using these models in the past year, achieving a high level of quality that demonstrates the technology's potential.

Participants need access to a LLM of their choice (recommended: gpt-4o and claude 3.5 sonnet). Please prepare a list of programming topics (projects you want to build, legacy code you might want to work on, frameworks and technologies you want to learn). I will provide an extensive list of ideas to try out during the talk.

Join the slack channel #llm-working-programmer for meeting other participants!

Repo: https://github.com/go-go-golems/go-go-workshop

CodeGen & Dev Tools: GitHub Copilot was introduced as “Your AI pair programmer” in January of 2021. Since then, we have been adding more and more capabilities to increase developer productivity and happiness. We’ve gone from simply leveraging AI to generate code to explaining existing code, generating unit tests, propose fixes for bugs, making code robust / secure, summarizing pull requests, having conversations tailored to your organization’s repositories, providing answers to your questions based on your organization’s knowledge base and so much more! Let's explore how to get started with GitHub Copilot, its ever growing list of capabilities, and how to make the most out of the tool.

As long as you have a web browser that can access public repositories on GitHub.com then you have all you need. To interact with GitHub Copilot we will be using GitHub Codespaces which you can connect to via a web browser.

https://gh.io/aifair-workshop

Multimodality: AI Agents are becoming more powerful at a rapid pace! In this talk you will learn best practices and considerations to think about when building your AI Agent Swarm. All aspects from low latency Speech to Text, Large Language Model RAG and Fine Tuning and Text to Speech. I will share open source code for how to build an AI Agent you can talk to with sub second latency end to end! You will learn how you can scale your AI Agents to handle 1000s or millions of concurrent conversations.

Deepgram Account

• https://console.deepgram.com/signup

Requirements

• Browser
• Microphone
• Speaker / Headphones

Development Tools

• NodeJS (http-server)
• JavaScript, CSS, HTML

Repos

• https://github.com/DamienDeepgram/deepgram-workshop-client (Required)
• https://github.com/DamienDeepgram/deepgram-workshop-server (Optional)

Building and deploying generative AI solutions can be challenging and time-consuming, especially for startups with limited resources and expertise. In this workshop, you will learn how to use AI templates and GitHub to quickly prototype and deploy generative AI applications in minutes. AI templates are ready-made solutions that leverage Microsoft Azure Services like Azure OpenAI and GitHub features like GitHub Codespaces.

https://aka.ms/aie-workshop

Learn to build generative AI applications on AWS using PartyRock and Amazon Bedrock. You will gain skills in prompt engineering and using the Bedrock API. We will also explore how to 'chat with your documents' through knowledge bases, retrieval augmented generation (RAG), embeddings, and agents. We will also use Amazon Q Developer to assist in coding and debugging. You will be provided an AWS account to go through this workshop. Some familiarity with writing Python code is helpful.

Workshop link will be given during session (it comes with a pre configured AWS account) Everything will work in browser, some Python experience would be helpful.

CodeGen & Dev Tools: GitHub Copilot was introduced as “Your AI pair programmer” in January of 2021. Since then, we have been adding more and more capabilities to increase developer productivity and happiness. We’ve gone from simply leveraging AI to generate code to explaining existing code, generating unit tests, propose fixes for bugs, making code robust / secure, summarizing pull requests, having conversations tailored to your organization’s repositories, providing answers to your questions based on your organization’s knowledge base and so much more! Let's explore how to get started with GitHub Copilot, its ever growing list of capabilities, and how to make the most out of the tool.

As long as you have a web browser that can access public repositories on GitHub.com then you have all you need. To interact with GitHub Copilot we will be using GitHub Codespaces which you can connect to via a web browser.

https://gh.io/aifair-workshop

In this workshop, I'll show you how to add secure AI code execution that supports any LLM in your AI app using E2B. AI-powered code execution improves reasoning of LLMs and allows you to build AI-based dashboards where users can chat with their data, generative UI, or coding agents like Devin.

We'll be using a prepared repository with Next.js app from E2B's cookbook: https://github.com/e2b-dev/e2b-cookbook/tree/main/examples/anthropic-power-artifacts

Tech stack:

• Anthropic's Sonnet 3.5 (get your API key here - https://console.anthropic.com/)
• E2B's Code Interpreter SDK (get your API key here - https://e2b.dev/docs/getting-started/api-key)
• Next.js
• Vercel's AI SDK

GPUs & Inference: TensorRT-LLM is the highest-performance model serving framework, but it can have a steep learning curve when you’re just getting started. We run TensorRT and TensorRT-LLM in production and have seen both the incredible performance gains it offers and the hurdles to overcome in getting it up and running. In this workshop, participants will learn how to start using TensorRT-LLM, including selecting a model to optimize, building an engine for it with TensorRT-LLM, setting batch sizes and sequence lengths, and running it on a cloud GPU.

https://github.com/basetenlabs/Workshop-TRT-LLM

BRING YOUR PROMPTS -- we will workshop them live! No yapping, no slop, just writing, testing, and editing prompts. Let's see how many we can get through!

Got prompts that aren't working quite the way you want them to? Bring 'em here and we'll see what we can do! You'll want to also bring a couple specific examples they misbehave on. Don't expect many slides -- this will be one hour of nothing but writing/editing prompts.

Intro Speaker - Jamie Neuwirth, Head of Startup Sales, who will share notes on building with Claude and working with Anthropic.

Building generative AI applications for production requires a paradigm shift to LLM Ops, with new tools, platforms and processes for orchestrating end-to-end development workflows. In this session, you’ll learn to build, evaluate and deploy an enterprise copilot application end-to-end, using a code-first approach with Azure AI Studio and Prompt flow. From ideating your scenario with your data, to evaluating it for quality locally and on Azure, then deploying it for real-world usage.

Requirements: laptop, GitHub account Helpful: Familiarity with VS Code, Python, Jupyter Notebooks, bash GitHub repository: https://github.com/Azure-Samples/contoso-chat

Open Models: In this 2 hour workshop, we will build an AI research agent that can search for research papers, summarize them, and answer questions on topics based on past research. We will use MongoDB as the agent's memory provider and knowledge store, open-source LLMs as the agent’s “brain”, and LangChain to orchestrate the end-to-end agentic workflow.

Attendees will be provided with all the resources required to successfully execute the hands-on portions of the workshop, including a GitHub repository consisting of notebook templates with pseudocode. Attendees will replace the pseudocode with their own code during the workshop.

For this workshop, attendees will need basic to intermediate knowledge of Python, and a laptop with the latest version of Python installed.

The preferred environment for running the labs is Google Colab, but for those who would prefer a local setup, instructions can be found here: https://mongodb-developer.github.io/ai-agents-lab/docs/dev-env/dev-setup#local-setup

RAG & LLM Frameworks: Learn about practical graph design patterns and retrieval strategies to more effectively customize GenAI for real-world applications. While GenAI offers great potential, it faces challenges with hallucination and lack of domain knowledge. Graph-powered retrieval augmented generation (GraphRAG) helps overcome these challenges by integrating vector search with knowledge graphs and data science techniques to improve context, semantic understanding, and personalization while facilitating real-time updates. You'll receive detailed coded examples to begin your journey with GenAI and graphs, leaving with practical skills to apply immediately to your own projects.

Prerequisites: This is a hands-on workshop where you can follow along with Jupyter notebooks in Colab. We will provide links to notebooks at the beginning of the workshop. To follow along please:

• Bring your laptop
• Create a Google account ahead of time if you don’t have one already so you can run Colab notebooks.
• [Prefered] Bring a working openAI API key. We will provide a key for those that do not have one yet. To make a new key, create an OpenAI account or sign in. Next, navigate to the API key page and "Create new secret key". Optionally naming the key. Save this somewhere safe, and do not share it with others. We recommend testing your key to make sure it works - see the OpenAI quickstart tutorial for more details.

Building and deploying generative AI solutions can be challenging and time-consuming, especially for startups with limited resources and expertise. In this workshop, you will learn how to use AI templates and GitHub to quickly prototype and deploy generative AI applications in minutes. AI templates are ready-made solutions that leverage Microsoft Azure Services like Azure OpenAI and GitHub features like GitHub Codespaces.

In this workshop, you will learn how to build LLM-based apps, such as a question-answering system with RAG, in LangChain using Llama-3 at 1,000 tokens per second on the SambaNova AI Platform.

Level: Intermediate

Abstract: SambaNova delivers generative AI capabilities to the enterprise. In this workshop, you will learn:

● About SambaNova’s full-stack generative AI platform, powered by the SN40L AI chip and delivering unparalleled performance for training and inference

● Samba-1, a trillion parameter composition of experts (CoE) model, and how it can be used for enterprise settings

● How to build and deploy a question-answering app end-to-end with retrieval augmented generation (RAG) for enterprise search using the following suite: LangChain as framework, Unstructured for pre-processing text documents, E5-large-v2 embedding, ChromaDB vector store, and Llama-3-8B-Instruct running at speed record of 1,000 tokens per second via SambaNova.

This workshop is designed for tech professionals, engineers, and anyone interested in enterprise generative AI applications.

Prerequisites: Experience programming, ideally in Python, a Github account, and laptop

Assets: We will provide a link to the Github repo with step-by-step instructions on how to install the required libraries and how to run the Jupyter notebooks and Streamlit apps. We will also provide SambaNova API keys for the CoE and Llama-3 endpoints.

GitHub Repo: https://github.com/sambanova/ai-starter-kit/tree/main/workshops/ai_engineer_2024/ Dev Setup for Exercise 1: https://github.com/sambanova/ai-starter-kit/blob/main/workshops/ai_engineer_2024/basic_examples/README.md Dev Setup for Exercise 2: https://github.com/sambanova/ai-starter-kit/blob/main/workshops/ai_engineer_2024/ekr_rag/README.md

This talk explores how MongoDB's Aggregation Framework transcends limitations of "classic RAG". While RAG utilizes vector representations and semantic relevance for data retrieval, it often relies on pre-defined knowledge bases, limiting the scope of analysis. The Aggregation Framework, on the other hand, empowers you to analyze entire collections directly within MongoDB. Furthermore, the Aggregation Framework offers unparalleled flexibility for complex calculations and data transformation.

Speakers and VP Pass attendees - head to the incredible View Lounge on the 39th floor for an exclusive closed-door welcome reception from 6:00pm - 9:00pm!

Get acquainted with fellow VPs of AI.

Multimodality: The rise of AI music generation tools has opened up a world of creative possibilities for musicians and non-musicians alike. This workshop will demystify these tools, providing a hands-on introduction to their capabilities and potential. Participants will learn how to use AI to generate musical ideas, create custom loops and samples, and even compose full songs.

In this workshop we will:

• Understand the different types of AI music generation tools and their use cases.
• Get hands-on experience with popular AI music platforms like Suno and Udio.
• Explore prompt engineering techniques for generating different musical styles and genres.
• Learn how to refine and customize AI-generated music to your liking.
• Discuss the ethical and legal considerations surrounding AI music creation.

Takeaway: By the end of the workshop, attendees will have a practical understanding of AI music generation tools and how they can be used to enhance creativity and musical expression. Whether you're a seasoned musician or just starting out, this workshop will equip you with the knowledge and skills to harness the power of AI for making music.

Prerequisites:

• An account with Suno and Udio (free or paid)
• No prior musical experience or coding knowledge required (but it helps)

Skill level: Beginner

Assets:

• https://aietalk.com/music/

RAG & LLM Frameworks: In this workshop, we'll use a variety of technologies to build a better AI assistant. This workshop leverages a Convex vector database to establish a knowledge base, Google Cloud speech-to-text, GPT-4 API, text-to-speech, Twilio and audio streaming. By the end, we'll have built an AI assistant that can take basic requests by voice, interpret what to do, make a phone call and interact with a human on the other end, then come back with some action to do (like adding an event to a calendar, or sending you an SMS update!).

This project is written entirely in Typescript, with Node, Express, and Next frameworks. To get started, clone this repo: https://github.com/get-convex/ai-world-fair.git

To follow along with full functionality, you will need keys for the following services. (Don't worry if you don't have them all! You can still follow along the presentation, and substitute the keys in later, or use entirely different services altogether.)

You will also need a free ngrok account if you wish to try it out locally.

See /server/.env.template for the required keys:

OpenAI required to manage the conversation, and for text-to-speech APIs

OPEN_AI_KEY=

Google Cloud credential file (often called service_account.json). Required for speech-to-text translation services. OAUTH client only required for integrating calendar & email. Not required for the workshop.

GOOGLE_APPLICATION_CREDENTIALS=./service_account.json GOOGLE_OAUTH_CLIENT_ID=

Twilio credentials. Required to make and stream phone calls

TWILIO_ACCOUNT_SID= TWILIO_AUTH_TOKEN=

There has been a ton of hype about AI's applications in software development, but leaders must look beyond the hype to assess the ROI for their organizations. We'll cover some quantitative and qualitative frameworks that have proven useful for our customers, highlight the benefits and drawbacks of popular metrics in use, and share the themes that have emerged as important pillars of the value prop for AI dev tools. I'll also share a vision for where the next 18 months of AI will lead in software development.

AI is poised to add $15.7 trillion to the global economy by 2030, with generative AI at the forefront of this revolution, marking a transformative shift across sectors. In his talk, Lukas Biewald will unpack the impact and potential of generative AI models and share practical insights learned from the best ML teams in the world who're building and implementing AI in production. He will share specific insights on deploying GenAI models into real-world applications, emphasizing LLM evaluation, dataset management, model experimentation and optimization. This session is a call to action for ML teams looking to leverage AI's full potential responsibly, and looking to expedite putting AI into production.

Drowning in unstructured text data? The Neo4j LLM Knowledge Graph Builder is your lifeline. This powerful application transforms your PDFs, web pages, and YouTube videos into structured knowledge graphs, enabling you to uncover hidden insights and make data-driven decisions quickly. By leveraging advanced ML models and Neo4j's graph database, the Knowledge Graph Builder streamlines the extraction of entities and relationships from your unstructured data. Connect, upload, and visualize your data as a comprehensive knowledge graph in just a few clicks.

Sendbird is building highly scalable generative AI chatbot in Amazon Bedrock that enterprise customers can use with confidence. In this session, we will reveal the lesson learned and secret sources which developed an LLM-based global chatbot service for 300+ million monthly active chat users.

Benjamin Dunphy, co-founder of the AI Engineer World's Fair and managing partner at Software 3.0, LLC, welcomes attendees and makes a few key announcements.

Previously, on AI Engineer...

Mozilla's Llamafile open source project democratizes access to AI not only by making open models easier to use, but also by making them run fast on consumer CPUs. Lead developer Justine Tunney will share the insights, tricks, and hacks that she and the project community are using to deliver these performance breakthroughs, and project leader Stephen Hood will discuss Mozilla's approach to supporting open source AI.

What if ChatGPT could talk to your realtime data? Your product database? Your salesforce data?

→ Write an email for my top paying customer quoting recent movies they rented →→ What are the genres of the movies here?

→→→ Make it funny using a reference from one of the movies.

As we hurtle towards AI ubiquity, AI connected to realtime data will become the default user expectation.

Join me as I unveil Pacha DDN, an AI powered realtime data access layer that securely connects data & business logic spread across multiple sources to your LLM. Pacha DDN provides autonomous explainable multi-step query planning to retrieve data from structured & unstructured data sources and APIs instantly.

Hyperspace is launching a new kind of agentic planning model, alongwith a breakthrough new AI application for power users. This is meant to run on the Hyperspace peer-to-peer AI network: the world’s largest and most advanced Uber-like network of open source models running on consumer devices (with over 15,000 nodes registered so far). It’s time to put a dent in the universe.

Come hear from the founders of Nixtla and how they partner with Microsoft in growing their business!

Building chat bots is fun! Building command line applications in Rust is even more fun! So join me in this session as we combine the two, and we look into the ways how you can create command line applications that use LLMs on Amazon Bedrock with Rust

RAG is the go to approach for AI to interact with data that it is not trained on. But beyond simple PoCs, writing your own RAG pipeline can be frustratingly complex. Why? Because users expect AI to answer their broad ranged natural language queries, but the data infrastructure is still playing catch-up. Traditional data architectures, designed with apps and APIs in mind, are now being pushed to their limits by the demands of advanced AI applications.

In this talk, we go back to first principles and present the requirements your data strategy should have to fully utilize the potential of AI. Instead of spending time setting up complex RAG pipelines per use-case, developers should be spending time preparing quality data and allowing a broad range of use-cases to just work. The heavy lifting of query planning, data retrieval and LLM orchestration should be done agentically. We will show a quick demo of Pacha, the world’s first AI agent that connects your LLM with private data.

In this talk, we'll talk through how developers are getting the most out of Atlas Vector Search through intelligent data modeling of unstructured data. This enables both iterating on your search problem and operating your search system in a way that seamlessly syncs with your original source data. We will also discuss how to consider the capabilities of embedding and chat completion models so that you can effectively ingest data into MongoDB in a way that makes it easily searchable.

In this talk, Devendra Singh Chaplot, Research Scientist at Mistral AI will explore the building blocks and training strategies that power Mistral AI’s large language models. It will feature Mistral AI's open-source models, Mixtral 8x7B and Mixtral 8x22B, which are based on a mixture-of-experts (MoE) architecture and released under the Apache 2.0 license. The presentation will also provide guidance on utilizing Mistral "La Plateforme" API endpoints and offer a preview of upcoming features.

In this brief product launch, you'll hear the story behind Plumb, a low code AI pipeline builder, and the challenges we faced in creating our own AI-powered product. The solution was to invent a different way to build AI products and features—one that lets everyone play. Bio:

Aaron Dignan is the cofounder and CEO of Plumb—a low code tool for building, testing, and deploying complex AI pipelines. He is a 5x founder and the author of Brave New Work (Portfolio 2019).

Discussions with VPs of AI.

In this session, we'll talk about current explorations by GitHub Next including Copilot Workspace a copilot native dev environment for getting started with everyday tasks.

"Discover best practices for rapid evaluation and iteration on LLM apps in large-scale applications, with a first-hand account from Discord's engineering team. This talk covers development workflow and evaluation methodology in order to measure model & prompt improvements, mitigate risks, and speed up development. We'll discuss the best practices that we refined and implemented internally, the tooling and automation that got us shipping improvements consistently, and some of the strange and wonderful things that happen with LLMs in the wild!"

"LLM inference is not your normal deep learning model deployment nor is it trivial when it comes to managing scale, performance and COST. Understanding how to effectively size a production grade LLM deployment requires understanding of the model(s), the compute hardware, quantization and parallelization methods, KV Cache budgets, input and output token length predictions, model adapter management and much more.

If you want to deeply understand these topics and their effects on LLM inference cost and performance you will enjoy this talk.

This talk will cover the following topics:

• Why LLM inference is different to standard deep learning inference
• Current and future NVIDIA GPU overview - which GPU(s) for which models and why
• Understanding the importance of building inference engines
• Deep recap on the attention mechanism along with different types of popular attention mechanisms used in production
• Deep dive on KV Cache and managing KV Cache budgets to increase throughput per model deployment
• Parallelism (reducing latency) - mainly tensor parallelism but data, sequence, pipeline and expert parallelism will be highlighted
• Quantization methods on weights, activations, KV Cache to reduce engine sizes for more effective GPU utilization
• Increasing throughput with inflight batching and other techniques
• Detailed performance analysis of LLM deployments looking at Time to first token, inter-token latencies, llm deployment characterizations, and more that can help reduce deployment costs

The main inference engine referenced in the talk with TRT-LLM and the open-source inference serve NVIDIA Triton."

In this talk, we will explore the cutting-edge techniques for Retrieval Augmented Generation with MongoDB. We will focus on leveraging Vector Search, specifically Atlas Vector Search, over MongoDB data to improve information retrieval and generation processes.

We will show how to build a RAG system using a Parent Child Retrieval Strategy to enable more efficient and accurate retrieval of relevant information. Additionally, we will show how all of this can be done within the MongoDB document model rather than relying on implementing these relationships in the application layer. And finally, we will introduce the concept of Search Nodes which enable you to serve vector search workloads at scale.

This talk is aimed at developers, ML engineers, and data scientists interested in building AI powered experiences with RAG. By the end of the session, attendees will have a solid understanding of how Retrieval Augmented Generation, Vector Search, and MongoDB can be leveraged to build innovative and scalable AI-powered applications.

JSON, prompt formatting, hallucinations. If you feel uncomfortable, you have probably had to implement complex solutions to circumvent these problems. What if there was a way to increase the reliability of Large Language Models at no cost? Enter structured generation. In this talk we will explore how the output of models can be steered at no extra cost, why this improves their efficiency and accuracy significantly and makes generation less sensitive to the specifics of the prompt. By the end of the talk, we'll understand how we can start benefiting from this breakthrough today by using the open source library Outlines.

Codeium is trailblazing the next frontier in retrieval and hint: it’s not just embeddings. Learn what the next generation of retrieval looks like and how 1M+ developers are already leveraging this superpower using the Codeium IDE plugin for AI autocomplete, chat, and search. We’ll dive deep into how existing benchmarks are failing us, what it takes to serve our custom models at scale, and what the future of AI-assisted software development looks like.

Harassment, Hate Speech, Pig Butchering Scams, and Underage users. These are just some of the possible categories of the (very) long tail of harm on Tinder. How can we possibly train, serve, and maintain models for all of these, at global, real-time scale? We build off of pre-trained models and an increasingly mature open-source ecosystem. In this talk, we'll cover how we've dramatically accelerated our modeling pipeline with (1) human-AI hybrid dataset generation for different harm vectors, (2) automated parameter-efficient fine-tuning of open-source large language and multimodal models for violation detection, and (3) serving fine-tuned adapters efficiently in real-time and at scale using LoRAX and cascade classification.

Retrieval Augmented Generation (RAG) has become a cornerstone for integrating domain-specific content and addressing hallucinations in AI applications. As the adoption of RAG solutions intensifies across industries, a pressing challenge emerges: understanding and identifying where within the complex RAG framework changes and improvements can be made. This talk delves into the methodology of extracting crucial indicators from your RAG pipeline, empowering informed decision-making during experimentation. Join us as we navigate an end-to-end process for RAG experimentation and evaluation, offering insights into optimizing performance and addressing hurdles along the RAG implementation journey.

When your mindblowing prototype meets the real world, GenAI projects can get stuck on their way to production. Whether the sticking points have to do with accuracy, explainability, compliance, cost, privacy, or something else, depends a lot on what’s at stake. In this round table session, we will explore “getting unstuck” and how that's different depending on project stakes... where "stakes" refers to what you have to gain or lose from good & bad answers. Let’s explore the bar for “getting to production” together through this lens--considering things like dollar impact, brand & reputational impact, privacy, bias, health & human safety, regulatory, etc. Bring your examples and questions and learn from each other’s successes and challenges to break the AI production code.

We opened model weights for Command R and R+, and the response was incredible. This talk will showcase the community's innovative projects and our journey to building SOTA multi-step tool use proficiency with the R family. We will also share the design decisions that make the R family unique and effective.

Cursor is an applied research lab building extraordinarily productive human-AI systems. To start, we're building the engineer of the future: a human-AI programmer that's an order of magnitude more effective than any one engineer.

In this talk, we’ll explore what went into building the Cursor of today, and how we’re building for the future.

In the insurance industry, LLMs promise efficiency but often get bogged down by manual tuning for optimal performance. DSPy changes the game.

Traditional LLM deployment is a high-effort, error-prone process, demanding extensive prompt engineering and fine-tuning across multiple steps.

Imagine deploying LLMs where manual optimizations are replaced by DSPy's automated, efficient prompt and weight optimization

Retrieval Augmented Generation is such a hack. Why would an embedding of your prompt coincide with the documents needed to answer it? Meanwhile Transformers already have a key/query mechanism built in! In this talk, we'll introduce Extended Mind Transformers, a new flavor of transformer that allows the model to select and attend to the most relevant information at each generation step. We demonstrate EMT's state-of-the-art performance and discuss important design decisions for long context applications.

I'll share practical insights I've learned from AI leaders at Duolingo, Gusto, Vanta, Filevine, Ironclad and Sourcegraph who have succeeded with LLMs in production. We'll cover the skills your team needs, tips for RAG in production, how to choose evals and what it takes to succeed with agents.

Explore GPT-4, multi-modality, and demos integrating sight and language with Dall-E and Whisper. Learn about developer tools, AI assistants, scalable applications, and customization. Focus on responsible AI, data privacy, and security with Azure. Featuring interactive demos and stories, this session is perfect for developers and innovators.

Attending with the VP Pass? Join your fellow leadership track attendees in an exclusive closed-door lunch in the View Lounge on the 39th floor with breathtaking views of the city! PS - VP pass holders have exclusive access to the View Lounge from 8:00am - 3:00pm on the 26th and 27th, so feel free to pop in any time within those hours.

RAG & LLM Frameworks: This in-depth technical session delves into real-world use cases and design patterns for building generative AI applications on top of your databases. We’ll cover evolving beyond semantic search, and using agentic patterns to implement dynamic grounding using various RAG techniques, giving you valuable insights on implementing enterprise-grade gen AI solutions.

Evals & LLM Ops: We all want faster, cheaper and more accurate tool calling abilities, but what are our options for improving accuracy and reliability of tool calling with Open Source LLMs? A systematic approach suggests a robust evaluation system is required to properly understand how every tiny design decision influences total system performance. When working with open source LLMs like Llama 3, the need to rigorously evaluate is crucial as many more adjustments can be made that influence the Tool Calling process.

During this talk you will learn what a baseline evaluation looks like and how a tradeoff surface emerges as you decide on implementation details like fine tuning, prompt/encoding choices, streaming, retries, and more.

Custom copilots are evolving from simple LLM+data retrieval augmented generation (RAG) solutions into more complex solutions with multiple interoperating APIs, models, metaprompts and grounding data. In this session, we’ll talk about these emerging patterns and show how Azure AI Studio and Promptflow empowers developers to build multi-agent copilot solutions. Discover advanced features such as debugging tools and security measures for developing intelligent, scalable, and safe AI solutions

Lunch & Learn: In this workshop you’ll build a demo that augments an app with AI for classification, natural language search, summarization, and outlier detection. Then you’ll learn how to iteratively improve before and after shipping to prod. We’ll use Hypermode to move fast and reduce infra overhead. You’ll walk away with a greater intuition for when & how to integrate AI without refactoring. Also, we have swag.

Lunch & Learn: In this bootcamp we demonstrate how smaller open source models, fine-tuned to excel at specialized tasks (e.g. classification, function calling etc.) can deliver comparable if not improved quality over proprietary state of the art models, resulting in significant cost reduction (over 60x) in your GenAI usage costs.

We’ll learn how to (1) collect and curate a fine tuning dataset, (2) fine tune an open source model (Llama 3), and (3) deploy the model fine tune into production with OctoAI and OpenPipe.

Follow the prerequisite instructions contained within the colab notebook (takes about 2 mins of setup): https://colab.research.google.com/drive/1DVw6vfEtzYV7QfcVXhmmjTVBOiJiJ02b#scrollTo=fUT1waisoXFs

RAG & LLM Frameworks: "RAG with vector databases has become a standard in GenAI applications, but the simple approach doesn't always produce the most relevant results. Contextualized Late Interaction over BERT (ColBERT) can overcome the shortcomings of traditional RAG, but how do you implement ColBERT at scale?

This talk introduces ColBERT, how it improves upon simple vector search, and how you can use it with the open-source RAGStack. Through live demos, you'll see how you can implement ColBERT embeddings, index them in Astra DB and build better retrieval for your GenAI application."

Unlock the power of Agent-E, a state-of-the-art open-source web automation agent designed for complex workflows. Discover how Agent-E leverages hierarchical planning and skill harvesting to execute multi-step, multi-site tasks across the web. Join this interactive workshop to delve into Agent-E's architecture and challenges, explore enterprise applications, and gain exclusive insights into the upcoming Emergence Agent Orchestrator.

Learn how the Azure AI model catalog helps you discover and operationalize flagship LLMs and SLMs on Azure and unlocks access to a wide collection of models with enterprise data privacy, content safety compliance, and security built in. Experience live demos of deploying popular LLMs like Mistral, Phi3 & Llama3 and finally, take a deep dive into how the model catalog has enabled organizations to drive innovation, efficiency, and strategic decision-making with real-world examples.

RAG & LLM Frameworks: It's long been said that "data is the new oil". With the recent rise of generative AI technologies, we have a new event more precious digital resource fueling the digital economy: knowledge. Retrieval-augmented generation (RAG) has presented a first generative AI use case for organizations to use their knowledge resources, like documentation, knowledge bases, and code repositories. This talk explores various other ways that organizations can use the knowledge resources they ingest for RAG to fuel growth and productivity. Use cases for knowledge resources covered in this presentation include: traditional natural language processing, fine-tuning models, and creating knowledge APIs for AI agents to consume. Attendees will come away from this talk with actionable guidance on how they can leverage their knowledge resources for maximum efficacy.

The challenge with financial agents successfully completing complex workflows like tabular reasoning or sentiment analysis often comes down to the reliability of executing numerous chained tasks together. Establishing the p99s necessary has to happen at the model level, yet most finance domain-specific LLMs are either only pre-training (BloombergGPT) or using supervised fine-tuning (FinBERT).

This presentation reveals how we transformed an open-source model into Albatross (https://huggingface.co/gradientai/v-alpha-tross), capable of performing at the top of the leaderboard on chat as well as domain-specific tasks. Our journey involved an intensive data pipeline and training regiment, incorporating a combination of continual pre-training, fine-tuning, and preference optimization, to customize the model for the intricacies of financial tasks. We'll share our insights on overcoming the execution hurdle, which is often the downfall of AI projects in specialized domains.

95% of RAG hallucinations are generated by the RAG, not the LLM. But why? In this talk, we'll discuss the hard data engineering problems of building highly accurate RAG systems and how to fix them. You'll see how companies like Air France are getting 95% accuracy or better.

Discussions with VPs of AI

Despite the big potential productivity increases proven from several years in the market, the embarrassing truth is that most developers in the world never or rarely use AI while coding. With hundreds of billions of dollars in upstream AI investments betting on insatiable inference demand, this is a big risk—but also a big opportunity that's not yet priced in, if you believe adoption will accelerate. Quinn Slack, the CEO/cofounder of Sourcegraph, presents this problem and shares solutions for other AI application builders and foundation model companies from lessons learned at Sourcegraph building Cody, the AI coding assistant that is #2 in revenue.

In the past years, we’ve witnessed a whirlwind of AI breakthroughs powered by extremely large and resource-demanding models. And now, faced with actually deploying these models at scale, AI engineers and builders are left to pick up the pieces on how to improve latency and resource consumption practically. In parallel, the on-device AI movement is heating up, imposing even more physical constraints on AI model deployment. In this talk, we will first identify key sources of inefficiency in AI models. Then, we will discuss techniques and practical tools to improve efficiency, from model architecture selection, to quantization, to prompt optimization.

"AI engineers are great, but to scale it out to the organisation you need an AI Platform team.

Similar to introducing Agile and DevOps companies have pilot projects to release their first genAI features. They would bring in people that have an affinity for both AI and applications together to form the first change agent in a company. Once you have a few teams, you notice that there is shared AI infrastructure, you need enablement and governance across. This pattern has been used to introduce Cloud, Security and Developer Experience.

In this talk we highlight:

• the shared components of the AI stack: proxies, caching, testing, feedback collection, guardrails, ...
• the steps (and struggles) to enable this across the whole engineering (hackathons, training, abstractions)
• how it fits in the existing SDLC workflow and processes (testing , versioning, observability , security)
• how we can leverage the knowledge of all platform teams together (cloudops, secops , developer experience , data platform and ai platform) for dealing with security , permissions and performance"

Dr. Sarah Buchner, Founder & CEO of Trunk Tools, envisions a future for construction where an army of AI agents works on behalf of our users. We are currently deploying an agent every 45 days: our Q&A agent, TrunkText, is already saving field professionals 1-2 hours every day. We believe that ~$2.5 Trillion of the construction industry can be automated with Trunk Tools.

A famous poet once said "Natural language is most powerful when it can draw from a rich context." Ok fine, I said that. But that's true of both poetry, and of LLMs! Well, Knowledge Graphs excel at capturing context. How can combining Knowledge Graphs with RAG – an emerging technique known as GraphRAG – give context to your RAG application, and lead to more accurate and complete results, accelerated development, and explainable AI decisions? This talk will go deep on the why and how of GraphRAG, and where best to apply it. You’ll get concepts, examples, and specifics on how you can get started. You’ll walk away with an understanding of how GraphRAG can improve the context you pass to the LLM and the performance of your AI applications.

The story behind our 8 bug fixes for Gemma, multiple tokenization fixes for Llama 3, a sliding window bug fix and Mistral-fying Phi-3, and learn about how we analyse and find and fix bugs in open source models. Learn also how we make finetuning 2x faster for all these models

AI agents show incredible promise, but have a hard time dealing with existing large-scale codebases. This talk will show examples of how LLMs fail when not given the proper tools and practical demonstrations of how pre-LLM tools already make developers superhuman. Explore a concrete set of challenges around user interfaces for AI and a vision for what superintelligent developer assistants might look like.

"Do you remember that feeling when you realized who was Jon Snow's mother? Or who was the Batman really? Those 'aha' moments define scientific reasoning: of many steps and non-obvious.

Even though the scientific discovery using LLMs is becoming more popular recently, there is little if any discussion about high level reasoning process behind breakthroughs in science.

Some of the discoveries like RNA interference required connecting of distanced areas of biology, described in different journals and with distinct vocabulary. In this talk we would like to discuss if LLMs would be able to spot those novel relationships behind phenomenas from distanced areas? What would it take: RAG, Agentic RAG or fully fledged AI agent?

We will first try to classify problems tackled by RAGs. Then we would define a new family of NLP problems related to the scientific discovery and propose a template for benchmarking of thereof. We would discuss them on the examples of Nobel Prize discoveries. Then the question would be if and how RAGs or more general AI Agents might help us in tackling those problems, so we will try to entertain you with some attempts to solve it."

Despite their trillions of tokens of training data, GPT4 and others models can’t answer a single question about us as unique individuals. The personal AI assistant we all want surely has to know us as individuals, right? Today’s AI assistants either give us generic information, or require significant input from us before providing useful personalized assistance. Isn’t there a better path? Of course there is. In this talk, we’ll discuss our implementation of a personal, private, proactive AI assistant that begins its relationship with you by downloading your transaction logs from major B2C sites (think of Google, Facebook, Amazon, Doordash, Strava, Uber and dozens of others) to your local device. By building a RAG system using this data, our personal AI assistant becomes useful immediately, without requiring significant user input. Learn how a modest amount of very powerful personal data can unlock the power and potential of personal AI assistants.

"In this presentation, I will explore Khan Academy's journey to become an AI-first organization, focusing on our AI Tutor and Teacher Assistant, Khanmigo. This case study will cover the strategic evolution from conception, through to launch, and continuous iteration and refinement.

Key Highlights:

• Strategic partnership: The collaboration with OpenAI to launch Khanmigo alongside GPT4, going from prototype to fully launched product in 3 months time. I will delve into the why and how behind our decision to partner with OpenAI, and to leverage generative AI in our product.
• Cultural shift in product development: The transition from our previous, more traditional product development approach to a highly experimental, prototype-driven culture. This section will highlight how we embedded AI across all teams to enhance our mission, moving beyond isolated AI projects to fully integrate AI into our operational fabric.
• Khanmigo in action: An in-depth look at Khanmigo's functionality and its solution to real-world challenges faced by students and teachers, coupled with it's impact in real classrooms around the US.
• Ongoing technical challenges and scaling: An overview of the technical obstacles encountered in scaling and maintaining quality as we continued to add more users.
• Ethical considerations and AI safety: How we thought of AI safety in the context of the classroom, and delivered on an experience that parents and educators trust.
• Pilot programs and iterative learning: Insights from our pilot programs across various school districts, focusing on how we've used feedback and data to refine and enhance Khanmigo's capabilities."

Fine-tuning LLMs is a fundamental technique for companies to customize models for their specific needs. In this talk, we will cover when fine-tuning is appropriate, popular libraries for efficient fine-tuning, and key techniques. We will explore both supervised fine-tuning (LoRA, QLoRA) and preference alignment (PPO, DPO, KTO) methods.

Learn how AI can enable code to self-improve in readability and security. This session explores the integration of AI into CI/CD pipelines with innovative prompting techniques for improving variable names, automating code comments, adding unit tests, and autonomously identifying and fixing security vulnerabilities. Discover practical methods for integrating AI with developer workflows, resulting in code that evolves and improves with minimal manual intervention. This approach makes software development more efficient and secure. Attendees will walk away with a plug-and-play CI template with a Bring-Your-Own-LLM option that can be integrated into their own CI pipelines.

"Large Language Models (LLMs) have become essential in advancing AI, enabling remarkable capabilities in natural language processing and understanding. However, the efficient deployment of LLMs in production environments reveals a landscape of challenges and technical debt.

Ethically, LLMs face issues such as bias amplification, where they might perpetuate existing stereotypes in their outputs. Misinformation is another concern, with the potential misuse of LLMs to create convincing yet false narratives. Privacy risks emerge from LLMs possibly memorizing and revealing personal data. Moreover, societal challenges include the impact on employment, as LLMs could automate tasks but also lead to job displacement. These challenges highlight the need for careful management and ethical considerations in the deployment of LLMs.

In this talk, Ahmed will highlight the key challenges and technical debt associated with LLMs' deployment, which demands customization and sophisticated engineering solutions not readily available in broad-use machine learning libraries or inference engines."

PLEASE ONLY RETURN JSON

It's been a year after Jason's most popular talk at the AI Engineering Summit. I wanted to come back with all the learnings I've had in the past year. How I've learned that Pydantic is still all you need, and bring more applications and use cases that I've seen in businesses and my thoughts on where the space will be heading in the future.

An executive briefing on OpenAI's Enterprise offerings and roundtable discussion for VPs of AI.

Come hear from the founders of Synthesized AI and how they partner with Microsoft in growing their business!

RAG & LLM Frameworks: This will be a 30-minute panel discussion between one MongoDB representative and three of our customers who are actively building AI applications. We will have 15 minutes towards the end for the audience to ask questions to the panelists. During the discussion, panelists will share insights into the following: -How their role at the company has changed in the past year -Their preferred AI stack -What AI features they are currently working on -Challenges with taking LLM applications from prototype to production -Most common user pain points, how they collect and incorporate user feedback -Metrics to measure success and monitoring

CodeGen & Dev Tools: "If you’ve ever tried any AI coding assistant tool, you may have noticed: they kind of suck. ChatGPT doesn’t know anything about your codebase, and even tools that are supposed to be able to answer questions about it often fail.

Starting from simply being able to talk about a single file, to being able to include specific files, then incorporating code search, coding assistants have evolved a lot over the past year or so.

However, most coding assistants are still not able to incorporate the full context of your codebase as well as things like Linear issues, design docs on Notion, and more.

In this talk, I’ll take you on a journey of how we fixed this problem by leveraging new and interesting strategies to incorporate just the right context without blowing up your context window."

RAG & LLM Frameworks: Learn about the power of GraphRAG (Graph Retrieval Augmented Generation) to improve the accuracy, relevance, and quality of LLM responses. While LLMs offer great potential, they can face challenges with lack of domain knowledge and hallucination. GraphRAG helps overcome these challenges by integrating vector search with knowledge graphs and data science techniques to improve context, semantic understanding, and personalization while facilitating real-time updates. In this session, we will cover GraphRAG through a real-world practical example, from creating a starter knowledge graph with a vector index to identifying and implementing useful GraphRAG patterns in a GenAI application.

"The term LLMOps has been around for just under 2 years. In the meantime, we have gone from using fine-tuned foundation models to running complex Agentic AI systems in production. Many breakthroughs are yet to be made and hardware limitations to be overcome. In this talk I will walk you through the LLMOps pipeline as if we were building an AI system from scratch. From building foundation models and fine-tuning them to observing complex Agentic AI systems in production. I will emphasize the different levels of scalability requirements for observability infrastructure and tooling at each step and why you might want to invest into it."

The field of Adversarial ML has been active since at least 2013 and despite over a decade of attempts to make models more robust to imperceptible changes in the input, attack methods still outpace the ability to defend neural networks and other machine learning models. In this talk, we'll get into why adversarial examples are becoming increasingly relevant with the advent of agentic multimodal LLMs and what we can do to defend these models.

Tired of chatbots that can barely answer basic questions? This talk dives deep into Vertex AI's agentic workflows. We'll walk you through the nuts and bolts of designing and building agents that can handle complex tasks, make decisions, and interact with your systems like a pro. You'll leave with the skills and knowledge to automate the boring stuff and unlock new levels of efficiency.

So you've built an AI assistant, but how do you gain the confidence to deploy it to production? In this live demo we introduce BotDojo, a platform that helps you build and improve LLM applications. We'll demonstrate how it identifies gaps in your assistant's knowledge through targeted evaluations. Then, we'll show how to generate synthetic data from existing documentation and support interactions to improve performance.

AI agents are poised to revolutionize software systems and devices, promising unprecedented automation and efficiency for enterprises. However, the road to this future is riddled with challenges such as inefficiency, non-determinism, high costs, discoverability, and rapid technological evolution. At Emergence, we are tackling these challenges head-on to transform the vision of useful AI agents into reality. Join us as we unveil our Orchestrator, which is a meta-agent designed to intelligently route tasks, integrate diverse AI agents, and future-proof enterprise systems. We will also show work from our R&D labs where we are advancing the science of agents and developing a web agent in the open source to automate complex enterprise workflows. Discover how our experience in building scalable AI solutions and our commitment to open-source development are paving the way for a seamlessly automated world."

[description will be updated after talk]

The struggle to grasp the inner workings of AI models can leave even experienced engineers from non-ML backgrounds feeling lost in a sea of terminology and new concepts. What if the key to understanding the intricate mechanics of LLMs didn't require a Ph.D.? This session offers an innovative approach, employing spreadsheets to dissect and demystify the architecture of decoder-based LLMs using a fully working implementation of GPT-2 entirely in Excel. Attendees will tour through GPT-2's architecture from tokenization, embeddings, attention, multi-layer perceptron, all translated into the accessible format of spreadsheets with minimal abstractions to get in the way. By the end, you'll gain unparalleled insights into AI's backbone, transforming abstract concepts into tangible, understandable processes, without ever touching code.

One year into the Rise of the AI Engineer, what do we think we know, and are we so sure?