Pragmatic AI With TypeChat

Talk transcript

1. 0:00

   [upbeat music] Good afternoon, my AI engineering friends.

2. 0:17

   How are we all feeling today? Great. [cheering] There we go. We got some energy even post-lunch. All right, you heard. I'm Daniel Rosenwasser. I'm the program manager on TypeScript, as well as a new little experimental library I'm here to talk about today called TypeChat.

3. 0:32

   Now, this is an AI engineering conference. Everybody here has used something like ChatGPT, right? We use it for this continuous flow of information. We've been able to prototype things with it, just get useful answers just by having this adorable little chat interface, right?

4. 0:49

   Um, but that's just one end of the spectrum. And on the other end of the spectrum, we have our traditional apps. These apps that are looking for this more precise sort of data to work with.

5. 0:59

   So the question is, how do we make all of the new AI tools, all these language models that are so powerful, accessible to every engineer out there? And so just to start things off, um, what if we had this cute, you know, this little app right here.

6. 1:14

   You have some basic user input at the very top, followed by these items, and each of these items has a venue name and description. So this just helps me figure out what I need to do on a rainy day in Seattle, because this is every day in Seattle for me.

7. 1:27

   Um, a lot of weather apps at this conference. But

8. 1:31

   the, the problem that you may find with trying to bridge together these language models and these traditional apps is that you find that you need to sort of massage the data.

9. 1:42

   You need to sort of like really, really, really pamper the models to give you what you're looking for. And even after all that's said and done, by default, these apps will give you natural language, which is great for people, but it's not great for code.

10. 1:59

    So if you just prototype this in, you know, something like a chat view, maybe you'd actually use the playground to do this. You would find yourself saying certain things to pamper, like keep it short and do this, and put everything on its different line and do whatever.

11. 2:13

    Um, you might find that you're starting to glom onto the patterns of what the language model gives you because you've seen it in a certain way, right? And you've noticed, oh, well, it gives me this format.

12. 2:23

    Each of these things is on its own line. Um, each of the lines has a leading number. They are always separating the venue name by the des-- and the description by a colon.

13. 2:31

    So I'll just do some basic parsing, split by new line, remove the trailing, the leading numbers, and then, uh, split on the colon. Um, that is a disaster [chuckles] waiting to happen because you can't rely on the language model to always do this.

14. 2:46

    And you can't know whether or not you're going to have something in the middle of that input that is going to just sort of wreck your parsing strategy, right?

15. 2:54

    Parsing natural language is extremely hard, if not a fool's errand for most people.

16. 3:01

    The thing that many people at this conference and elsewhere have discovered is you can say, "Pretty, pretty please give me some JSON." And it works pretty well, right? You know, just, here's an example of what I'm expecting.

17. 3:15

    Please respond with the answer. And voila, it comes right back. But there's two issues with this. One is just doing that on its own is not enough to guarantee that your app is actually gonna get the data it's looking for, um, because maybe there's an extra property that doesn't seem to align.

18. 3:34

    Maybe there's not enough data in the actual response, so you need to do some level of validation. Um, but not just that. You can't comprehensively describe all of the things that you want practically.

19. 3:47

    In this case, I have really, really simple schema or really, really simple like example. All the objects are uniform. They all have the same properties. End of story, right?

20. 3:58

    But what if something is optional? What if something is required but needs to be null in some cases? What if this could be a string or a number, but never something else?

21. 4:07

    I don't know. So you will not be able to get that far for more complex examples because you end up with this combinatorial explosion. So what we found is that you can use types.

22. 4:20

    Types are this great description to actually guide the model. Here, I'm just using type definitions in TypeScript. These are just plain interfaces. All I want is a thing with a list, and the list has these objects, and the objects have these two properties that are both strings on them.

23. 4:36

    And the beauty of these type definitions is that the types can guide the model, right? So you can actually use these types to tell a model, "Hey, here's some user input.

24. 4:48

    Here's a user intent. Now use this with the types that I'm g- actually gonna use in my application. Throw it through your cool AI service," whatever that is. That may be OpenAI, Cohere, uh, Anthropic.

25. 5:01

    Maybe it's a local model. Maybe it's Llama code. I don't know. But the, the point is, what we found is that if you use an, a language model that is sufficiently trained on both human prose, natural language, and code, this actually bridges the two worlds together.

26. 5:18

    But like I said, the guidance is not-- it's only half of the problem, right? You need to be able to actually validate what you're getting. And that's the key insight is that the types can also validate the results.

27. 5:30

    And so what we found is, in our experience, you know, we're using TypeScript. TypeScript's great for JSON because it's a superset of JavaScript, which is a superset of JSON, which means that you can actually construct a miniature little program that underneath the hood, the TypeScript compiler is using to do that validation.

28. 5:48

    And if that all goes well, then great. You have well-typed data from your language model. And if it doesn't go well, well, underneath the covers, what we actually end up with is an error message, right?

29. 6:01

    Because it's actually using the TypeScript compiler under the hood.

30. 6:04

    That error message can be used to perform a repair when you are reaching out to a language model to say, "No, no, no, no, no, that's not what I wanted.

31. 6:13

    Try again." And so the key insight is types are all you need. Types can actually guide and validate, and it becomes a very powerful model because... Whoops.

32. 6:27

    Well, yes, actually. [laughing] That's a key insight that we have with TypeChat. It's an n- it's a library on npm right now. It's a TypeScript library at the moment. Um, and basically we've bundled this all together and make it easy to just guide a language model, perform these, uh, queries, and actually, like, make sure that you're actually getting

33. 6:46

    well-typed, um, data from the language models. And so you can actually use much more complex examples as well. You might say, like, "I have a coffee shop, and the coffee shop has this schema, these types."

34. 6:58

    You define them like this. And basically you can use that to combine that with a user intent and input, and you get well-typed output. And I'll actually demo that right now.

35. 7:12

    Um, what I have here is my, you know, the TypeChat repository cloned, npm installed, everything's set up, and we have an examples directory. And I think if you're just curious to get started with, um, TypeChat, the examples directory gets you started.

36. 7:29

    We have a table. If you look at the README, we have a table of all of our examples. They kind of increase in, in complexity and difficulty, and the first one is, like, a sentiment thing where we say if something is positive, negative, or neutral.

37. 7:42

    Um, but that's so basic, it's like our hello world. I actually wanna go back to that coffee shop example that I showed you just now. So

38. 7:51

    we have this coffee shop schema, and this is just a bunch of types, right? You probably have something similar in your preferred language as well.

39. 8:01

    Um, and what I can do here is I'm just gonna run

40. 8:06

    our entry point, and from the command prompt, I actually have a little prompt, and I can actually just make orders here. So I can say, "One latte with foam, please."

41. 8:17

    Ta-da. Right? I- [cheering] Yeah. [clapping] So, you know, it's, it's, it's... This is the key thing is that it's actually so simple, and it actually just works very well in a surprising way.

42. 8:35

    Um, now that's... I could, I could just tell you about that and I could walk off, and that's not really good enough, I know. Um, what happens if I say one latte and a medium purple gor- purple gorilla named Bonsai?

43. 8:54

    So what actually happened here is technically when we ran this prompt, this thing succeeded. But even though we got a successful result, we were able to do this sort of recovery here.

44. 9:08

    We actually in our app are able to say, "I didn't understand the following, a medium purple name, gorilla named Bonsai." And that actually showed up in the JSON. And the reason that it did is because we have this thing called unknown text.

45. 9:21

    So we've started to see these patterns in that instead of doing this sort of prompt engineering, you're doing schema engineering. You're able to sort of thread through these results, uh, into your app, because if you actually, you know, remove this stuff...

46. 9:35

    And let me show you what this actually looks like. If you look at the coffee shop example, this is under 40 lines of code, right? The magic here actually comes from, we create a model, we infer it based on your environment settings, um, and then the actual magic is that we have this JSON translator.

47. 9:52

    You give us the contents of your types, you select the type that you're expecting, and then every single time you need to translate the user intent, you just run this translate function.

48. 10:02

    Now I'm getting type errors because I removed the type and it's telling me, like, this will never happen. Whoops. Not that.

49. 10:08

    So if I rerun this thing and I say, "One cappuccino."

50. 10:17

    Cappuccino? I can't spell anything today. And a purple

51. 10:22

    gorilla named Bonsai. I wanna be precise here. [laughs]

52. 10:30

    So I got a bagel with butter because I asked for Bonsai, and the thing is that the lang- what's gonna happen is that the language model really doesn't wanna dis- disappoint you.

53. 10:41

    It really wants to make sure you're getting what you want. So

54. 10:45

    this is the, this is the thing is you can actually define a schema that is rich enough to encoun- you know, anticipate failure, gives you a chance to recover, show that to the user, say, "I got this and this and this and that.

55. 10:58

    It wasn't so clear on that." And that's kind of the beauty of this approach. It's very simple, and it's really just about defining types, which you're gonna use in your application anyway.

56. 11:09

    Now, there's this other thing that we started encountering when we s- showed this off to teams internally. Um, people said, "Well, that's all cool. You're turning coffee into code."

57. 11:21

    Um, I do too. How do I actually do something more rich like commands? What if I wanna actually script my application in some way?

58. 11:32

    Well, this approach that I just showed you actually works for very simple stuff as well, right? You can, you can imagine something where you say, "Schedule an appointment for me," and that turns into a specific command for a calendar app.

59. 11:44

    In fact, in our examples, we actually have that. Um, what if you wanna string together multiple things? Hey, that's just a list of commands, right? Um,

60. 11:55

    kinda. [laughs] What's the... The problem with this is if I want these to kinda thread through to each other, this is a simple example, so it's just going input, you know, run the command, get the output, go to the input, et cetera, et cetera, et cetera, et cetera.

61. 12:10

    Um, what if you have something that expects multiple arguments? What if you wanna reuse a result? Sure seems like you need variables and other things like that here. Um, so we asked ourselves, is there a thing here where you can imagine you can just generate code and

62. 12:29

    just take the same approach where types are all you need? So what if you could just define, here's all the methods that I want you to be able to call.

63. 12:36

    Come back with some code that only calls those methods, and then generates a program like this.

64. 12:43

    The problem is that you really wanna have some sort of sandboxing and safety constraints in place, right? And so you might start saying, "I need availability. I can't just endlessly loop here.

65. 12:55

    Um, so I'm not gonna allow loops. I'm not gonna allow lambdas and whatever." And the, the problem is that even if you decide, I'm gonna pick a subset of a language like JavaScript or Python or whatever you have, um, the language models have seen so much of that code that they're gonna draw outside the lines, and then

66. 13:12

    you'll hit this failure case, and then you just won't get a result. You won't get a bad result. You just won't get a result that conforms to what you're expecting.

67. 13:19

    And then you still have to worry about sandboxing, and then there's all these questions about synchronous versus asynchronous APIs and all this other stuff too, that language models don't tend to understand because I guess most people don't either.

68. 13:31

    Um [laughing] So what we actually [laughs] have been trying is, uh, we generate a fake language. Uh, we have the language models generate a fake language still based on the types,

69. 13:43

    but it's in the form of JSON, actually. And so you have things like refs, and refs are just pri- you know, references to prior results. And [laughs] if you're familiar with like, you know, if you're a compiler, this will look, may look like SSA.

70. 13:54

    It might look like an AST, whatever. Um,

71. 13:58

    but we use that to construct a fake TypeScript program in memory as well, and use that to make sure that not just are you calling all the, only the methods that are available to you, that you can only do certain actions, but also that the inputs from prior steps, um, matches up with the types that you're defining

72. 14:18

    from your API. And so that kinda comes back to types are all you need. We have another really simple example for...

73. 14:28

    We have a math schema. This is basically a calculator in sheep's clothing. So if you go back and we run this here,

74. 14:38

    we have another prompt that's an abacus. That's the closest thing to a calculator I could get. Um, if we could say something like add one to forty-one and then divide by seven.

75. 14:51

    Now, basically what happened here is we made a language model good at math. So we've also solved a whole other set of problem, right? [clapping] Yeah. Um, more seriously, though, uh, so at each of these steps, we're actually performing, not having the language model call a method, perform an operation.

76. 15:14

    And if you actually look at the code here, um, math main.

77. 15:20

    This is all under fifty lines of code. We are able to do the same sort of translation. We have a separate thing called a program translator. And in that program translator, when you are successfully able to validate your results, you know, you say, "If this thing is a success or not a success, just jump out, otherwise do

78. 15:38

    some stuff with it." We have this evaluate function, and this evaluate function takes a callback, and that callback is just sort of like this instruction interpreter. And so you can do this with objects, you can do this with a function, with a switch case or whatever.

79. 15:52

    Um, but the point is that this actually allows you to do some richer tasks. Um, now, there are other approaches for many of these things, and they overlap with what TypeChat does.

80. 16:04

    But the cool thing is that TypeChat is able to actually give you this level of validation for both JSON and programs. Um, and it's something that we're also experimenting with, with other languages too.

81. 16:15

    So for example, people at this conference have been saying, "Yeah, I, you know, TypeScript is very cool," and I agree with them because I work on TypeScript. Um, but how would I wor- make this work with Python?

82. 16:27

    And so we have been experimenting with this, and we've been getting fairly good results. I'm able to do something like the coffee shop with a very similar approach using types.

83. 16:37

    Um, I'm able to do something similar with the calculator app, just defining methods on a class with comments and all this other stuff that helps the model do a little bit better, and it works really well.

84. 16:50

    Um, we can even do more complex examples too. Like we have this CSV example. Um, maybe I want to be able to... Well, I'm not gonna get into... Oh, pipenv.

85. 17:04

    The demos, demo gods are gonna kill me here. [laughs]

86. 17:10

    That. Brutal. Okay. I can just create a program that does this now. I have this entire API that grabs columns and is able to perform certain operations, and then do joins that do filtering and joining and all this other stuff as well because it just sort of does this selection based on Booleans.

87. 17:34

    So read a CSV, find all the values that equal N/A, and then drop the rows. And so this becomes this sort of powerful approach, and this is just a prototype of the Python stuff that we've been working on as well.

88. 17:45

    Um, it's not primetime, and if you wanna talk to me about it, I'm definitely game.

89. 17:52

    So what I want from you all is to try TypeChat out. Reach out. What I'm here at this conference for is to learn about what you're all trying to build, trying to help bridge the gap as well between what we're all learning on the cutting edge and making that more accessible to everyday engineers who have been at

90. 18:10

    this more precise end of the spectrum, bringing the power of these language models that are so rich to the traditional apps. Thank you very much. Come see me at the Microsoft booth.

91. 18:19

    I'll be hanging out for a little bit. And thank you. [clapping] [upbeat music]