AI Engineer Summit 2023
Shift Left: How to Become an AI Engineer from a Full-Stack Background
Read the talk
Shift Left: From Full-Stack Development to AI Engineering
A practical learning sequence for full-stack developers: understand language models, learn prompting, compose applications, evaluate changes, and fine-tune when the task warrants it.
From a talk by Reid Mayo
Before you start: You should be comfortable building infrastructure, databases, backend services, and client applications; no prior AI or machine-learning experience is assumed.
What do you need beyond full-stack skills?
If you can build an application from infrastructure through its client interface, what else do you need to build useful AI products? Reid Mayo’s curriculum starts with that question. It assumes you already understand infrastructure, databases and persistence, backend services, and client applications, but have no AI or machine-learning background. The proposed path is a boot camp: a sequence of concepts and practical tutorials aimed at professional AI engineering.
The opportunity comes from foundation models. Shawn “swyx” Wang’s The Rise of the AI Engineer, which inspired the talk’s name, describes how developers can now build useful AI solutions without first acquiring the traditional ML expertise or making the substantial upfront investment in data collection those solutions once required. Full-stack experience is a starting asset: the new work extends an existing ability to turn components into usable products.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a foundation without getting distracted
Before choosing tools, establish a way to learn. Drawing on The Art of Learning, Mayo recommends limiting distractions and avoiding low-information material with diminishing returns. Spend that attention on fundamental building blocks: understanding them thoroughly makes it possible to compose more sophisticated products later.
Use ChatGPT as a private tutor while reading. With Socratic questioning, an unfamiliar term becomes the beginning of a conversation: ask what it means, why it works, and how it relates to concepts you already understand. Mayo invokes a January 2022 knowledge cutoff to emphasize how much foundational material predates the tool; that is his historical reference, not a current cutoff specification. The recording supplies the sequence and the reasons for studying each topic; the assigned materials supply the detailed practice.
Begin with a bounded overview of what large language models are and how they work. Mayo selects Cohere’s educational documentation, noting its connection through a founder to the creation of the transformer architecture. Read module one in its entirety, rather than following every possible branch of the documentation, and continue questioning your tutor as concepts arise. This first pass should give subsequent experiments a conceptual foundation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn prompting before fine-tuning
Prompt engineering can feel strange to someone accustomed to symbolic code and explicit control flow. Why should phrasing a request politely change the quality of a result? Mayo uses that counterintuitive example to introduce the difference between programming conventional logic and interacting with a neural model. The wording of an input can affect output quality, so learning how to construct that input is part of engineering the system.
Skipping directly to fine-tuning misses two benefits. First, the best-performing prompts can become part of the training data used for fine-tuning. Second, hands-on prompting exposes what a model can do and where it fails. Prompting is both an optimization technique and a way to investigate model behavior.
Work through the materials in sequence:
- Watch Elvis Saravia’s overview, then read the Prompt Engineering Guide in full.
- Pay particular attention to the graduate job classification case study. Mayo highlights how successive prompt modifications improve aggregate output quality; the useful lesson is to examine the accumulated effect across examples, rather than one appealing response.
- Read LearnPrompting.org in full. Mayo describes it as favored by OpenAI and recommends the overlap as reinforcement, alongside the additional concepts it introduces.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Get to working experiments with OpenAI
OpenAI is the next stop because it combines capable models with accessible interfaces. That combination lets you explore what is possible and begin building quickly. Read the documentation and API reference thoroughly; practical limitations of depending on one provider come later in the curriculum, after you have learned how to use it.
Then briefly survey the hands-on examples in the OpenAI Cookbook. The goal at this point is familiarity with the available patterns, not mastery of every notebook. Return to individual examples when a project needs them. The linked Cookbook and the Evals repository introduced later are evolving collections, so their present contents should not be treated as frozen copies of the materials assigned in the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose an application with LangChain
A model call is only one component of an AI application. LangChain enters the syllabus as a framework for organizing the surrounding pieces into a maintainable, modular system. Those pieces include models, prompts, long- and short-term memory, retrieval-augmented generation, and conversations. Its integration points also allow unsupported or proprietary components to join the application.
That breadth gives the framework a second role: it becomes a map of the AI application ecosystem. Learning how its components fit together exposes more of the engineering practice than studying a model API alone. Start with CommandBar’s nontechnical executive summary, then the plain-English technical guide Mayo recommends for a few basic building blocks. Move from that initial picture of composition into the documentation and implementation.
Read both the Python and JavaScript/TypeScript documentation. At the time of the talk, their coverage differed, so each version offered concepts absent from the other; the repetition also reinforced the shared ideas. As you encounter a documented feature, inspect its implementation on GitHub. This connects the framework’s public abstractions to the code that makes them work.
Finish with Mayo Oshin’s LangChain beginner tutorial to see the fundamentals used in an application. His other walkthroughs then apply those same fundamentals to more complex tasks. The progression is deliberate: understand the pieces, inspect their implementation, and watch them compose into a working system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Establish the test before changing the model
For a full-stack developer, evaluations are the analogue of software tests. Before fine-tuning a black-box model, establish a repeatable way to tell whether a change improves behavior or causes regressions. Without that feedback loop, a different output is easy to mistake for a better one.
A classification task makes the basic structure concrete. The following TypeScript example compares baseline and candidate predictions against the same labeled cases. The records are a teaching example: the candidate fixes one case and breaks another, so an unchanged aggregate score would conceal a meaningful change in behavior.
typescript
type Label = "graduate" | "not-graduate";
type EvalCase = {
id: string;
expected: Label;
baseline: Label;
candidate: Label;
};
const cases: EvalCase[] = [
{
id: "entry-level-developer",
expected: "graduate",
baseline: "not-graduate",
candidate: "graduate",
},
{
id: "senior-engineer",
expected: "not-graduate",
baseline: "not-graduate",
candidate: "graduate",
},
];
const results = cases.map((item) => {
const before = item.baseline === item.expected;
const after = item.candidate === item.expected;
return {
id: item.id,
baselineCorrect: before,
candidateCorrect: after,
change: before === after
? "unchanged"
: after ? "improvement" : "regression",
};
});
console.table(results);
For less constrained outputs, deciding what counts as success requires more care than exact label matching. Mayo explicitly calls out the creativity needed to design effective AI evaluations.
Use the Cookbook’s example evaluations to learn how to write them, then review OpenAI Evals, which provides an evaluation framework, a suite of evaluations, and support for custom ones. This is a focused pass through the materials, but it comes before fine-tuning for a reason: the measurement process needs to exist before the model changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ship, collect data, then test specialization
Now work step by step through OpenAI’s hands-on fine-tuning Cookbook material. Knowing how to adapt its models can take a product a long way, but relying on a hosted provider also introduces production constraints: cost, latency, rate limits, privacy, and control. These constraints supply a practical reason to investigate other models.
Mayo proposes a conditional deployment sequence:
- Prototype and ship with OpenAI’s models to get a useful solution into use quickly.
- Collect usage and training data from that solution.
- When scaling warrants it, evaluate a smaller model fine-tuned for the target use case. Test whether it can match or outperform the original model while costing less.
The smaller model is a candidate to evaluate, not an automatic replacement. The target is performance on the product’s actual task.
After completing the OpenAI material, move to Anyscale’s tutorial on fine-tuning Meta’s Llama 2. Its fine-tuning case study illustrates the task-specific nature of this strategy: fine-tuned models exceeded GPT-4 on some structured tasks, but still trailed it on GSM8k mathematics. Specialization can close a gap without establishing general model parity.
Mayo then cites an OpenPipe case study in which a smaller fine-tuned Llama 2 matched the reported results of OpenAI’s state-of-the-art model for $19 versus approximately $24,000 on the same example task. He describes the example as not cherry-picked, but the talk does not specify the task, quality metric, workload, OpenAI model, or cost accounting. Treat those figures as his reported case-study comparison, not as a cost estimate for a new application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Practice before going deeper
At this point, pause the curriculum and use the skills in the real world. Mayo recommends deploying applications before moving into advanced study. The boot camp has covered enough to build, measure, and adapt an AI system; further reading can follow experience with those responsibilities.
When you are ready for deeper theory, continue with fast.ai’s practical deep learning course and Hugging Face’s NLP course and documentation. These extend the path beyond application composition into a richer understanding of deep learning, more advanced fine-tuning, and eventually training models from scratch. Mayo closes by pointing to the companion syllabus displayed beside him—the path continues, but the next step is to put the fundamentals to work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Swyx's 2023 essay about the engineering discipline emerging around foundation models and AI APIs.
Guides, papers and notebooks for studying prompting techniques and applications.
Practical examples for building with OpenAI models; the collection has evolved since the talk.
An open-source framework and benchmark registry for evaluating language models and building custom evaluations.
A 2023 fine-tuning study comparing Llama 2 on functional representations, SQL generation and grade-school mathematics.
Further reading
A study of prompting language models to identify job postings suitable for graduate or entry-level applicants.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, I'm Reid Mayo, founder of re:ma AI.
- 0:16
Welcome to Shift Left: How to Become an AI Engineer from a Full-Stack Background. In this talk, we'll provide and review a syllabus that walks you step by step through a defined process with practical tutorials that teach you the comprehensive best practice skills and knowledge required to launch a professional AI engineering career.
- 0:35
Think of it in a way as an AI engineering boot camp. So this talk assumes that you have a strong full-stack engineering background. You should be comfortable building modern tech from the ground up across all of the different layers:
- 0:46
infrastructure, database and persistence, and applications both on the back end and on the client side. However, this talk assumes zero background with, uh, any AI or machine learning. We're gonna start from scratch there.
- 0:59
So why, why would you be interested in becoming an AI engineer? For an in-detail summary, I'd encourage you to read The Rise of the AI Engineer by Shawn Swyx Wang that inspired the name of this talk.
- 1:10
One critical takeaway from this essay is when Swyx identifies that full-stack engineers can now deploy a wide variety of legitimately useful AI solutions by leveraging new foundational models. Previously, such solutions would've required substantial experience in traditional ML techniques and costly investment in upfront data collection.
- 1:31
So let's go ahead and move forward. So before we dive into the syllabus itself, I want you to follow a few techniques from the book The Art of Learning that this course was designed around.
- 1:41
This is gonna make your learning more effective and efficient. First, stay focused and limit distractions. There's a lot of low information out there with diminished returns. Stay focused on the important topics.
- 1:52
Speaking of important topics, we're gonna invest heavily in the fundamentals in this course. By understanding fundamental building blocks well, we'll be able to build sophisticated AI products through composition of those blocks.
- 2:03
Lastly, as you go through the syllabus, use ChatGPT as a private tutor. Any time you come across a new concept, use the Socratic method with ChatGPT to unfurl the topic until you understand it thoroughly.
- 2:15
You'd be surprised how many concepts predate ChatGPT's January 22nd knowledge cutoff date. So regarding the syllabus itself, as we go through each section, I'll be spending most of our limited time talking about the why.
- 2:28
We'll summarize what you will learn and why it is important. Let's go ahead and dive in. Section one, overview to large language models. Before we start working with large language models, it's useful to start with a short but respectably thorough overview of what they are and how they work at a high level.
- 2:44
Cohere is a company founded by one of the creators of the transformer architecture, and they've got a great overview of these core concepts in their educational docs, so we'll start there.
- 2:53
Remember, stay focused. Only review module one in, in its entirety, and keep pairing the Socratic method with ChatGPT to flesh out your knowledge as you go along.
- 3:04
Okay, moving forward. Section two, prompt engineering. So on its face, prompt eng- prompt engineering feels like a bunch of voodoo mumbo jumbo. Uh, it feels absurd really because we're used to working with symbolic architectures based on code logic, so it's strange to imagine getting higher quality output by prompting an AI model politely.
- 3:23
Uh, but [laughs] the language models are neural architectures. They're inspired by our brains, so different techniques are required. The bottom line is that prompt engineering objectively increases the quality of neural architectures' output, uh, such as language models.
- 3:37
So now you might be tempted to say, "All right. I'm gonna skip all this prompt engineering stuff and get straight to fine-tuning models," but fine-tuning quality is often increased by starting with the best performing prompts and using those prompts in your fine-tuning training data.
- 3:51
Lastly, it's important to really sink your hands into the prompt engineering clay to see what language models are capable of and also to probe their limitations. So regarding course materials, start out by watching the overview video from Prompt Engineering Guide founder Elvis Saravia, then dive directly into the guide itself.
- 4:09
Read it cover to cover, and pay special attention to the graduate job classification case study that shows how layering on prompt engineering techniques iteratively increases quality of output in aggregate.
- 4:21
Next, read the Learn Prompting org docs favored by OpenAI cover to cover. The redundant concepts in this second guide are useful to review to really lock in these critical concepts, and also this guide does cover additional concepts as well.
- 4:36
All right, moving on. Section three, OpenAI. OpenAI does two things incredibly well. One, they provide state-of-the-art AI models, and two, they make them incredibly accessible. By learning OpenAI, you can understand the art of what's possible today.
- 4:52
You can also start building and experimenting with AI engineering quickly. However, there are some practical limitations to consider that we will address further on. So regarding course material, we're gonna read the OpenAI docs and API reference cover to cover.
- 5:06
Then I would encourage you to quickly review the practical hands-on examples in their cookbook. Don't spend too much time there. You can come back later, and we want to keep marching.
- 5:16
Okay, moving on. Section four, LangChain. LangChain is the applications framework that allows you to put AI tech together in an organized and well-architected way so it is highly maintainable, modular, and scalable.
- 5:29
So LangChain integrates all the different parts and pieces required for a modern AI system, models, prompts, long and short-term memory for retrieval-augmented generation and conversations, practically everything. Furthermore, for any components that aren't supported yet, LangChain is flexible enough to allow straightforward integration of these new components, including your proprietary needs.
- 5:51
Lastly, and this is very important in the context of the syllabus, because LangChain is the glue layer for most everything else in the AI ecosystem, you will learn a lot about the comprehensive practice of AI engineering by building a comprehensive understanding of LangChain.
- 6:07
Now, onto the course materials. So building AI apps is a new paradigm. There's a lot to absorb. So we're going to prime you with a non-technical comprehensive executive summary by CommandBar first, then we'll follow up with a simple plain English technical guide that covers only some basic LangChain building blocks, so you can begin to quickly grok how
- 6:27
a more complex AI system can be built up modularly with this framework. So as you might imagine, the meat and potatoes of this section will be the LangChain docs and code base.
- 6:37
LangChain's documentation is highly thorough, so take full advantage of it. I encourage reading both the Python and the JavaScript/TypeScript d- uh, docs cover to cover, as the review helps lock in your knowledge, and there are important concepts in each version that aren't yet in the other.
- 6:53
As you read through the docs, pop over to GitHub and stick your head under the code base, uh, hood to see how LangChain implements the features and functionality that the documentation covers.
- 7:03
This will give you in-depth practical knowledge on how to build AI tech the right way. Lastly, for real-world LangChain app tutorials, Mayo Ocean has great video walkthroughs. Specifically, I would encourage reviewing his LangChain beginner's tutorial, as it covers the fundamentals.
- 7:20
His other videos take these fundamentals and apply them towards more complex tasks. All right, moving on. Section five, evaluating AI models. Coming from a full stack background, evals are basically your software tests.
- 7:34
Before we start fine-tuning black box AI models, we need a scientific process that can evaluate our changes iteratively. Otherwise, how do we know we're making improvements and not regressions, right?
- 7:45
So regarding the course materials, OpenAI has a great cookbook that walks you through writing some example evals. Note that the nature of AI output often means you're going to have to be a little bit creative when writing effective evals.
- 7:57
Furthermore, OpenAI also provides a framework that includes a robust eval suite and allows for writing your custom evals as well. Review these mat- materials quickly. All right, moving on.
- 8:10
Section six, fine-tuning. By this point, you've already gained some exposure into fine-tuning OpenAI's models. We're going to take that further by going step by step through their fine-tuning cookbook.
- 8:20
So knowledge of fi- of how to fine-tune OpenAI models will take you a long way. However, there are practical limitations to relying on OpenAI alone. For example, it can be cost-prohibitive, and you can run into latency or rate-limiting issues in production.
- 8:33
This is in addition to standard privacy and control concerns. Because of this, an efficient pattern is to prototype and ship a solution quickly using OpenAI's models, start gathering usage and training data.
- 8:45
Then, if the solution needs to start scaling, see if you can fine-tune a smaller and cheaper open source model to match or out-compete OpenAI's model on your target use case.
- 8:56
So regarding course materials, first, completely go through the OpenAI fine-tuning hands-on cookbook. After that, we'll walk through Any S- Anyscale's tutorial that demonstrates how to fine-tune an open source model, Meta's Llama 2, such that it can match or even beat OpenAI's models and target tasks.
- 9:14
Finally, we're going to skim OpenPipe's cost savings case study that shows how on our example task, and it's not cherry-picked, a smaller fine-tuned Llama 2 model at a cost of $19 can match results from OpenAI's state-of-the-art model, which would cost around $24,000 for the same task.
- 9:36
Final section, advanced study. So by this point, you've completed the boot camp section of the syllabus. I'd encourage you to start deploying your AI engineering skills in the real world before moving on to these advanced studies.
- 9:47
However, once you're ready to take your skills well beyond the basics, fast.ai's practical deep learning course and Hugging Face's NLP course and their docs will give you a rich understanding of deep learning theory.
- 9:58
In addition to learning fine-tuning further, you will also be able to train models from scratch. All right, so we've reached the end. So the syllabus is linked to my left.
- 10:07
Thanks for joining me today. And for any questions, please reach out to me on LinkedIn. Bye. [outro music]