RAG & LLM Frameworks: In this workshop, we'll use a variety of technologies to build a better AI assistant. This workshop leverages a Convex vector database to establish a knowledge base, Google Cloud speech-to-text, GPT-4 API, text-to-speech, Twilio and audio streaming. By the end, we'll have built an AI assistant that can take basic requests by voice, interpret what to do, make a phone call and interact with a human on the other end, then come back with some action to do (like adding an event to a calendar, or sending you an SMS update!).

This project is written entirely in Typescript, with Node, Express, and Next frameworks. To get started, clone this repo: https://github.com/get-convex/ai-world-fair.git

To follow along with full functionality, you will need keys for the following services. (Don't worry if you don't have them all! You can still follow along the presentation, and substitute the keys in later, or use entirely different services altogether.)

You will also need a free ngrok account if you wish to try it out locally.

See /server/.env.template for the required keys:

OpenAI required to manage the conversation, and for text-to-speech APIs

OPEN_AI_KEY=

Google Cloud credential file (often called service_account.json). Required for speech-to-text translation services. OAUTH client only required for integrating calendar & email. Not required for the workshop.

GOOGLE_APPLICATION_CREDENTIALS=./service_account.json GOOGLE_OAUTH_CLIENT_ID=

Twilio credentials. Required to make and stream phone calls

TWILIO_ACCOUNT_SID= TWILIO_AUTH_TOKEN=