No-code Fine-tuning: Mark Hennings

Talk transcript

1. 0:00

   [upbeat music] Hey, my name is Mark Hennings.

2. 0:22

   I'm a serial entrepreneur, and I'm super excited to talk to you about fine-tuning large language models today without any code. So let's begin. For our purposes today, fine-tuning is training a foundation model for a specialized task.

3. 0:38

   Some examples of these specialized tasks are writing any kind of copy, emails, blog articles, product descriptions. It could be scrubbing fake emails from a list, extracting or normalizing data, translating, paraphrasing, rewriting, qualifying a sales lead, ranking priority of support issues, detecting fraud, or flagging inappropriate content.

4. 0:59

   These are very common tasks that businesses do every day, and something they have in common is that traditional programming or rule-based approaches do not work well for them. But large language models are great at them.

5. 1:12

   They perform them easily, and they can capture the nuance in the text that you're working with. So why should we fine-tune? I mean, prompt engineering is great, right? You can do almost all of these things with a prompt.

6. 1:23

   Well, I'll tell you. Fine-tuning is awesome. It's faster and cheaper because you can train a lighter model to match the quality of what you were doing with a prompt.

7. 1:33

   It reduces the size of your prompts, allowing for longer completions. Training examples allow you to cover edge cases and collaborate better as a team,

8. 1:42

   and it's naturally resistant to prompt injection attacks. So let's dive into some of these. How much faster is it really? Well, if you take GPT-4 and its response time per token, it's about a hundred and ninety-six milliseconds, give or take, from the OpenAI API.

9. 2:00

   On the same API, GPT-3.5 is seventy-three milliseconds. That's three times faster.

10. 2:07

    How much cheaper is it? Well, taking an example with GPT-4 versus GPT-3.5 fine-tuned, you can save eighty-eight point six percent. Well, then how much shorter do the prompts actually get?

11. 2:19

    Well, I'll give you one example because it's gonna vary depending on your prompt. But here's what a typical engineered prompt might look like. It has some instructions, um, saying that, you know, write a blog post on this topic, how to write it, what tone to use, what to do, what not to do.

12. 2:36

    Well, with a fine-tune model, it learns how we write, so we don't need all of those instructions. It learns from our training examples. So we're just giving it the one thing that's unique about this prompt versus another prompt, which is the topic that we want to write on.

13. 2:51

    And in this very conservative example, it's ninety percent shorter.

14. 2:57

    Now let's talk about collaborating as a team, right? Because none of us work in a vacuum. We work with other people. Imagine a GitHub repo. You have one file.

15. 3:06

    Your whole code base is just one file. That's like your epic prompt. Well, with fine-tuning, now you can have multiple files like we're used to, where developers can work on this section of code or that section of code.

16. 3:17

    But we're not talking about code, we're talking about training examples. So your training data is this layer that your team can work on and add to and edit and improve, and then that feeds into the fine-tune model.

17. 3:31

    So the main point is if you can get equal or better output, why wouldn't you fine-tune a model? Now, fine-tuning is kind of a dev job right now, okay?

18. 3:42

    Let's be real. If you go online and you look up how to do fine-tuning, you're gonna find articles that talk about how to spin up GPU servers for training and inference, and you gotta format your data with these ad hoc Python scripts and configure these parameters and then make API calls.

19. 3:57

    It just looks like a dev job. But if you really break it down, why can't we just automate all of that with a user interface? Is that possible?

20. 4:08

    It is possible, and the bar is lower than most people think to get started doing this. If you can get twenty examples of what you want your fine-tune model to do, you can fine-tune a model.

21. 4:19

    This is not traditional machine learning, where you need thousands of examples to get started, and the dataset is this impossible barrier to get past. No, this is something that you could handwrite these if you want to.

22. 4:32

    One way to think about this is as an extension to few-shot learning. Let's say you can have five examples of what you want a model to do in your prompt.

23. 4:41

    Well, with fine-tuning, your training example dataset can be as long as you want. So instead of five examples, you can now have twenty or a hundred. So it seems intuitive that with more examples, the model would be able to do closer to what we want it to do.

24. 4:56

    So here's what I propose for a dev life cycle for large language models. We start with prompt engineering. Prompt engineering is a powerful tool. It allows us to create a prototype, to validate the concept, and we can also use it to create our initial datasets for fine-tuning.

25. 5:12

    Once we have those datasets, we should fine-tune a model, and we should evaluate it to make sure that it actually is better than the prompt engineered version. And then we can test which models we can get to perform at the same level.

26. 5:25

    Then the fine-tune model can go into production. And from production, we can capture feedback from our users, and we can log the examples. And with those examples, we can continuously improve our fine-tune model because now all of a sudden we have, um, the real examples that we can add back into our dataset.

27. 5:42

    So in terms of roles, I think that there's a huge opportunity for people to get into prompt engineering and fine-tuning who are not developers. Yes, if you're a developer, you can fine-tune.

28. 5:52

    Absolutely. But you shouldn't have to be the only person that can fine-tune. I'm a co-founder at Entry Point, and we have built the modern tooling to make this easy.

29. 6:02

    Let's take a look at how it works. Here we are on the dashboard, and I'm gonna open the Press Release Writer project. Let's take a look at my 20 examples.

30. 6:11

    Um, the way I created these 20 examples for a press release generator was I went online and I found 20 press releases that looked really good. They came from blog articles about the best press releases that you can write.

31. 6:24

    However, I didn't have input data, so my dataset was incomplete, but I used ChatGPT-4 to take the press release and then write a list of facts that would be needed to actually have a professional writer write such a press release.

32. 6:40

    You know, large language models aren't great at facts, so providing it the facts as the input makes sense to me, that I want to give it a list of facts and then have it write something that's really polished that would be a really good first draft of a press release.

33. 6:52

    With this user interface, I have a lot of visibility into the data that I'm actually putting into my fine-tuned model, which I think is really important. And the way this works is that we have a structured data approach.

34. 7:03

    So when you import, like, a CSV into Entry Point, each column becomes a field. Here I have the facts, and here I have the press release. And these fields you can use in a template.

35. 7:15

    Just like you were writing a mass email and you wanted to insert somebody's first name or personalize the emails with information about a contact record, you can use references to these fields with the Handlebars templating language.

36. 7:28

    Um, it provides a really intuitive way to easily format your output, your input. And GPT-3.5 Turbo, when you fine-tune it, you can actually use the system prompt, uh, which is where you can include instructions as well, which creates this really interesting hybrid between prompt engineering and fine-tuning, where you can have a small dataset for fine-tuning, but you

37. 7:48

    can also give it some instructions to help. Once we have a dataset like this, we can go and we can go to our fine-tunes, press the Add button, um, select the model, the platform, because this is cross-platform, and then we count your tokens and estimate your cost for you.

38. 8:04

    This is gonna be a whole dollar, so [inhales]

39. 8:07

    hold on tight. Press Start, and that will get started. But I have some here that are already trained, um, so let's go into one and use Entry Point Playground and see if we can actually generate a press release with our fine-tune model.

40. 8:21

    The list of facts here I actually wrote about, um, the AI Engineer Summit, and we'll see if we can make a press release for the AI Engineer Summit.

41. 8:30

    Let's go. All right. So this fine-tune model created a title here, and it made it look like a press release. What I found to be a really cool workflow is to actually create a list of facts and then generate an article, read the article, and then get ideas from it, and go back to my list of facts

42. 8:49

    and refine those, and then that actually becomes an iterative process to get really cool results. So I really enjoy fine-tuning. It takes a lot of the boilerplate out of the prompt, and you can just focus on what's important for the results you want, um, and the rest is taken care of by your training data.

43. 9:04

    Entry Point has a lot of other cool features, like data synthesis and tools to compare the performance of your fine-tune models. Unfortunately, we don't have time to go into all of that today, but I hope you will check it out.

44. 9:15

    It's entrypointai.com, and it was a pleasure speaking to you. [upbeat music]