AI Engineer Summit 2023
Pragmatic AI With TypeChat
Read the talk
Pragmatic AI With TypeChat
TypeChat connects natural-language requests to application code by using types to guide generation, validate results, and describe programs whose operations remain under application control.
From a talk by Daniel Rosenwasser
Before you start: Basic familiarity with TypeScript interfaces, JSON, and function calls will help you follow the examples.
A rainy-day recommendation needs a dependable shape
What should an app do with a request for places to visit on a rainy day in Seattle? Its interface might show a map and a list of recommendations, each with a venue name and description. A conversational answer is useful to a person, but the application needs those pieces as separate fields. This is the gap Daniel Rosenwasser, program manager on TypeScript, introduces: connecting the freedom of a chat interface to the precise data expected by traditional software.
A tempting first implementation asks the model to keep its answer short and put each recommendation on a separate line. After seeing a few responses, a developer notices a pattern: every line begins with a number, followed by a venue name, a colon, and a description. The parser practically writes itself: split on newlines, remove the leading numbers, then split on the colon.
But an observed formatting habit is not an interface contract. The model can change its formatting, and the content itself can contain a delimiter that breaks the parser. More instructions may make the pattern more common; they do not make parsing arbitrary prose dependable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
JSON supplies syntax; types supply the contract
Asking for JSON improves the situation immediately. Give the model an example object containing venue names and descriptions, and it can return something the application can parse without interpreting prose. Yet a parseable response can still contain unexpected properties or omit information the app needs. Valid JSON is only the beginning of validation.
An example also becomes an awkward way to describe a richer contract. Uniform objects are easy to illustrate; optional properties, required properties that sometimes contain null, and values that can be either strings or numbers introduce combinations that examples must somehow cover. Types express these distinctions directly:
| Requirement | TypeScript form |
|---|---|
| Property may be absent | description?: string |
| Property must exist but may be null | description: string | null |
| Value accepts two alternatives | value: string | number |
These declarations describe the allowed cases without enumerating every possible object.
For the Seattle app, the contract can remain small: a list of objects, each with two string fields. An interface expressing that shape looks like this:
typescript
interface Venue {
name: string;
description: string;
}
interface Recommendations {
venues: Venue[];
}
The model receives the user's intent together with the type definitions the application expects to consume. Those types become guidance for producing the answer. Rosenwasser's qualification is that the model needs sufficient training on both natural language and code; naming a provider or a local model does not establish that every model will perform equally well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the same types to validate and repair
Guiding generation solves only half the problem. The returned data must also satisfy the contract. TypeScript provides a useful route because JSON values can be represented inside a TypeScript program. Under the hood, the library constructs a miniature program that lets the compiler check the candidate against the requested type.
That produces a straightforward validation and repair procedure:
- Check the generated data against the expected type using the TypeScript compiler.
- If the check succeeds, return the well-typed data.
- If it fails, use the compiler diagnostic to ask the model for a repaired answer.
The same contract therefore guides generation and supplies concrete feedback when generation goes wrong.
TypeChat packages this process into a TypeScript library available through npm. Its usefulness grows with schemas richer than the recommendation list: a coffee order, for example, can describe several kinds of items and the choices available for each. User intent and those types go into translation; structured, checked output comes back.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give unsupported requests somewhere to go
The live demonstration starts in a cloned TypeChat repository with dependencies installed. Its README orders the examples by increasing complexity, beginning with positive, negative, or neutral sentiment classification. The coffee shop moves beyond that introductory task: its schema describes orders, and its command-line prompt accepts a request for one latte with foam. That request translates successfully.
The next request combines a latte with a medium purple gorilla named Bonsai. Translation still succeeds, but success does not mean the coffee shop has acquired a gorilla. The JSON carries the unsupported phrase through an unknown-text alternative, allowing the application to tell the user which part it did not understand. The schema can represent both the recognized order and the unresolved part of the request.
This shifts some of the work from prompt engineering to schema engineering. The schema does more than list acceptable products: it defines how the application can receive and handle an incomplete interpretation. An explicit unknown case gives the model a way to preserve unsupported intent instead of forcing every phrase into a menu item.
Rosenwasser describes the shown coffee application code as under 40 lines. Its core wiring creates a model from environment settings, constructs a JSON translator with the type-definition contents and expected type name, then calls translate for each request. The recording uses the original createJsonTranslator(model, schema, typeName) construction; TypeChat 0.1.0 later changed this to createJsonTranslator(model, validator) and moved program helpers to typechat/ts. The line-count claim concerns the shown application wiring, not its separate schema or the library underneath it.
Removing the unknown-input alternative has an immediate effect in the editor: the application code that handles that case now produces type errors. After adjusting the example, Rosenwasser requests a cappuccino and a purple gorilla named Bonsai. This time, the result includes a bagel with butter. The altered schema has left the model without an appropriate way to express that part of the request.
Type correctness does not establish that an answer faithfully represents the user's intent. A bagel can be a valid menu item while being the wrong interpretation. A schema that anticipates misunderstanding gives the application a recovery path: show what was recognized, expose what remains unclear, and let the user resolve it. The unknown case supports that interaction; it is not a guarantee against every mistaken interpretation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A list of commands eventually needs dataflow
Structured data can also describe actions. A request to schedule an appointment can translate into a calendar command, and several requested actions can become a list of commands. This is a natural extension of the coffee example: the output now tells the application what to do.
A simple chain works when one operation's output becomes the next operation's input. The scripting slide illustrates a sequence of readFile, trimText, and writeFile. But what happens when an operation needs several arguments, or two later operations need the same earlier result? The representation needs a way to name and reuse values. A command list is becoming a program.
One approach is to declare the methods the model may call and ask it to generate code using only that API. Execution constraints quickly complicate the idea. To protect availability, the application might forbid loops; it might also exclude lambdas and other constructs. A model trained on full JavaScript or Python can still generate code outside that permitted subset, leaving the application with a rejected program instead of a usable result. Sandboxing and the distinction between synchronous and asynchronous APIs add further concerns.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Represent the program as JSON
TypeChat's experimental approach is to generate a small artificial language encoded as JSON. The API types still guide generation, but the model produces a structured program rather than unrestricted source code. References identify results from prior steps, making those results available as arguments to later operations. Rosenwasser compares this representation to an abstract syntax tree and to static single assignment form: both analogies help explain explicit operations and dependencies.
The JSON program is then used to construct a synthetic TypeScript program in memory. Compiler validation checks both the calls and the connections between them: an operation must be available in the declared API, and a referenced result must have a type accepted by the consuming operation. This goes beyond checking that a command name exists. It checks whether the proposed program fits the API's type relationships. Those checks constrain the program; they do not by themselves establish a complete execution sandbox.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let application code perform the arithmetic
The calculator demonstration makes the dependency concrete: add one to forty-one, then divide by seven. The addition produces the value needed by the division. The model's job is to describe those operations and their relationship; application code performs the arithmetic. The resulting calculation is (1 + 41) / 7 = 6, not evidence that the model itself has become a more accurate calculator.
Rosenwasser describes the shown calculator application code as under 50 lines. It uses a program translator, stops if translation and validation fail, and otherwise passes the validated program to an evaluator. The math application's evaluator callback acts as an instruction interpreter, dispatching each operation to application code. That dispatch can use an object, a function, or a switch.
For the two operations in this request, an application-owned dispatcher can be as small as:
typescript
type MathOperation = "add" | "divide";
function dispatch(
operation: MathOperation,
left: number,
right: number,
): number {
switch (operation) {
case "add":
return left + right;
case "divide":
return left / right;
}
}
const sum = dispatch("add", 1, 41);
const result = dispatch("divide", sum, 7);
In the translated workflow, the evaluator resolves the reference to the addition's result before invoking the division callback. This separates interpreting the request from executing its operations, while allowing tasks richer than a single command.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry the typed approach into Python
Other approaches overlap with parts of this design. The capability Rosenwasser emphasizes is validation of both structured JSON results and programs. The next question is whether the same idea can work beyond TypeScript. In the Python experiments, a coffee-order schema again guides interpretation, while a calculator API describes operations as class methods with typed arguments and explanatory comments. Rosenwasser reports encouraging results, without presenting a quantitative benchmark.
The Python code view shows calculator methods and their docstrings, illustrating how an API can communicate both permissible operations and their intended meaning. The next example extends that idea from arithmetic to a richer CSV-processing API.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose a data-cleaning request from available operations
The CSV API exposes column access and operations that support filtering and joins, with Boolean selections identifying rows. The closing request is to read a CSV, find values equal to N/A, and drop the corresponding rows. The same program-generation idea now connects several data-processing operations: obtain the data, compute a selection, and use that selection to remove rows.
The Python work shown here is explicitly a prototype, not ready for prime time. Even so, it extends the central mechanism from menu items to arithmetic and tabular data: describe the application's vocabulary with types, translate intent into that vocabulary, and validate the result before the application uses it.
Rosenwasser closes by asking engineers to try TypeChat and share what they are building. The practical goal is to bring language models into the precise world of everyday applications, giving developers a familiar contract through which free-form requests can become usable data and executable plans.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Library and examples for translating natural-language requests into validated structured data.
Application code that translates requests into programs and dispatches arithmetic operations through an evaluator callback.
Further reading
- Introducing TypeChatArticle
The original announcement explains type-guided generation, compiler validation and repair.
Setup instructions and an ordered path through sentiment, coffee, calendar and program-generation examples.
- Schema engineering techniquesDocumentation
Practical schema guidance, including explicit ways to represent requests outside an application's domain.
- PypechatRepository
Daniel Rosenwasser's Python prototype with coffee, math and CSV examples.
Updates since the talk
- TypeChat 0.1.0 API changesArticle
Explains validator-based translator construction, Zod support and changed import locations introduced after the talk.
Read the complete timestamped transcript
- 0:00
[upbeat music] Good afternoon, my AI engineering friends.
- 0:17
How are we all feeling today? Great. [cheering] There we go. We got some energy even post-lunch. All right, you heard. I'm Daniel Rosenwasser. I'm the program manager on TypeScript, as well as a new little experimental library I'm here to talk about today called TypeChat.
- 0:32
Now, this is an AI engineering conference. Everybody here has used something like ChatGPT, right? We use it for this continuous flow of information. We've been able to prototype things with it, just get useful answers just by having this adorable little chat interface, right?
- 0:49
Um, but that's just one end of the spectrum. And on the other end of the spectrum, we have our traditional apps. These apps that are looking for this more precise sort of data to work with.
- 0:59
So the question is, how do we make all of the new AI tools, all these language models that are so powerful, accessible to every engineer out there? And so just to start things off, um, what if we had this cute, you know, this little app right here.
- 1:14
You have some basic user input at the very top, followed by these items, and each of these items has a venue name and description. So this just helps me figure out what I need to do on a rainy day in Seattle, because this is every day in Seattle for me.
- 1:27
Um, a lot of weather apps at this conference. But
- 1:31
the, the problem that you may find with trying to bridge together these language models and these traditional apps is that you find that you need to sort of massage the data.
- 1:42
You need to sort of like really, really, really pamper the models to give you what you're looking for. And even after all that's said and done, by default, these apps will give you natural language, which is great for people, but it's not great for code.
- 1:59
So if you just prototype this in, you know, something like a chat view, maybe you'd actually use the playground to do this. You would find yourself saying certain things to pamper, like keep it short and do this, and put everything on its different line and do whatever.
- 2:13
Um, you might find that you're starting to glom onto the patterns of what the language model gives you because you've seen it in a certain way, right? And you've noticed, oh, well, it gives me this format.
- 2:23
Each of these things is on its own line. Um, each of the lines has a leading number. They are always separating the venue name by the des-- and the description by a colon.
- 2:31
So I'll just do some basic parsing, split by new line, remove the trailing, the leading numbers, and then, uh, split on the colon. Um, that is a disaster [chuckles] waiting to happen because you can't rely on the language model to always do this.
- 2:46
And you can't know whether or not you're going to have something in the middle of that input that is going to just sort of wreck your parsing strategy, right?
- 2:54
Parsing natural language is extremely hard, if not a fool's errand for most people.
- 3:01
The thing that many people at this conference and elsewhere have discovered is you can say, "Pretty, pretty please give me some JSON." And it works pretty well, right? You know, just, here's an example of what I'm expecting.
- 3:15
Please respond with the answer. And voila, it comes right back. But there's two issues with this. One is just doing that on its own is not enough to guarantee that your app is actually gonna get the data it's looking for, um, because maybe there's an extra property that doesn't seem to align.
- 3:34
Maybe there's not enough data in the actual response, so you need to do some level of validation. Um, but not just that. You can't comprehensively describe all of the things that you want practically.
- 3:47
In this case, I have really, really simple schema or really, really simple like example. All the objects are uniform. They all have the same properties. End of story, right?
- 3:58
But what if something is optional? What if something is required but needs to be null in some cases? What if this could be a string or a number, but never something else?
- 4:07
I don't know. So you will not be able to get that far for more complex examples because you end up with this combinatorial explosion. So what we found is that you can use types.
- 4:20
Types are this great description to actually guide the model. Here, I'm just using type definitions in TypeScript. These are just plain interfaces. All I want is a thing with a list, and the list has these objects, and the objects have these two properties that are both strings on them.
- 4:36
And the beauty of these type definitions is that the types can guide the model, right? So you can actually use these types to tell a model, "Hey, here's some user input.
- 4:48
Here's a user intent. Now use this with the types that I'm g- actually gonna use in my application. Throw it through your cool AI service," whatever that is. That may be OpenAI, Cohere, uh, Anthropic.
- 5:01
Maybe it's a local model. Maybe it's Llama code. I don't know. But the, the point is, what we found is that if you use an, a language model that is sufficiently trained on both human prose, natural language, and code, this actually bridges the two worlds together.
- 5:18
But like I said, the guidance is not-- it's only half of the problem, right? You need to be able to actually validate what you're getting. And that's the key insight is that the types can also validate the results.
- 5:30
And so what we found is, in our experience, you know, we're using TypeScript. TypeScript's great for JSON because it's a superset of JavaScript, which is a superset of JSON, which means that you can actually construct a miniature little program that underneath the hood, the TypeScript compiler is using to do that validation.
- 5:48
And if that all goes well, then great. You have well-typed data from your language model. And if it doesn't go well, well, underneath the covers, what we actually end up with is an error message, right?
- 6:01
Because it's actually using the TypeScript compiler under the hood.
- 6:04
That error message can be used to perform a repair when you are reaching out to a language model to say, "No, no, no, no, no, that's not what I wanted.
- 6:13
Try again." And so the key insight is types are all you need. Types can actually guide and validate, and it becomes a very powerful model because... Whoops.
- 6:27
Well, yes, actually. [laughing] That's a key insight that we have with TypeChat. It's an n- it's a library on npm right now. It's a TypeScript library at the moment. Um, and basically we've bundled this all together and make it easy to just guide a language model, perform these, uh, queries, and actually, like, make sure that you're actually getting
- 6:46
well-typed, um, data from the language models. And so you can actually use much more complex examples as well. You might say, like, "I have a coffee shop, and the coffee shop has this schema, these types."
- 6:58
You define them like this. And basically you can use that to combine that with a user intent and input, and you get well-typed output. And I'll actually demo that right now.
- 7:12
Um, what I have here is my, you know, the TypeChat repository cloned, npm installed, everything's set up, and we have an examples directory. And I think if you're just curious to get started with, um, TypeChat, the examples directory gets you started.
- 7:29
We have a table. If you look at the README, we have a table of all of our examples. They kind of increase in, in complexity and difficulty, and the first one is, like, a sentiment thing where we say if something is positive, negative, or neutral.
- 7:42
Um, but that's so basic, it's like our hello world. I actually wanna go back to that coffee shop example that I showed you just now. So
- 7:51
we have this coffee shop schema, and this is just a bunch of types, right? You probably have something similar in your preferred language as well.
- 8:01
Um, and what I can do here is I'm just gonna run
- 8:06
our entry point, and from the command prompt, I actually have a little prompt, and I can actually just make orders here. So I can say, "One latte with foam, please."
- 8:17
Ta-da. Right? I- [cheering] Yeah. [clapping] So, you know, it's, it's, it's... This is the key thing is that it's actually so simple, and it actually just works very well in a surprising way.
- 8:35
Um, now that's... I could, I could just tell you about that and I could walk off, and that's not really good enough, I know. Um, what happens if I say one latte and a medium purple gor- purple gorilla named Bonsai?
- 8:54
So what actually happened here is technically when we ran this prompt, this thing succeeded. But even though we got a successful result, we were able to do this sort of recovery here.
- 9:08
We actually in our app are able to say, "I didn't understand the following, a medium purple name, gorilla named Bonsai." And that actually showed up in the JSON. And the reason that it did is because we have this thing called unknown text.
- 9:21
So we've started to see these patterns in that instead of doing this sort of prompt engineering, you're doing schema engineering. You're able to sort of thread through these results, uh, into your app, because if you actually, you know, remove this stuff...
- 9:35
And let me show you what this actually looks like. If you look at the coffee shop example, this is under 40 lines of code, right? The magic here actually comes from, we create a model, we infer it based on your environment settings, um, and then the actual magic is that we have this JSON translator.
- 9:52
You give us the contents of your types, you select the type that you're expecting, and then every single time you need to translate the user intent, you just run this translate function.
- 10:02
Now I'm getting type errors because I removed the type and it's telling me, like, this will never happen. Whoops. Not that.
- 10:08
So if I rerun this thing and I say, "One cappuccino."
- 10:17
Cappuccino? I can't spell anything today. And a purple
- 10:22
gorilla named Bonsai. I wanna be precise here. [laughs]
- 10:30
So I got a bagel with butter because I asked for Bonsai, and the thing is that the lang- what's gonna happen is that the language model really doesn't wanna dis- disappoint you.
- 10:41
It really wants to make sure you're getting what you want. So
- 10:45
this is the, this is the thing is you can actually define a schema that is rich enough to encoun- you know, anticipate failure, gives you a chance to recover, show that to the user, say, "I got this and this and this and that.
- 10:58
It wasn't so clear on that." And that's kind of the beauty of this approach. It's very simple, and it's really just about defining types, which you're gonna use in your application anyway.
- 11:09
Now, there's this other thing that we started encountering when we s- showed this off to teams internally. Um, people said, "Well, that's all cool. You're turning coffee into code."
- 11:21
Um, I do too. How do I actually do something more rich like commands? What if I wanna actually script my application in some way?
- 11:32
Well, this approach that I just showed you actually works for very simple stuff as well, right? You can, you can imagine something where you say, "Schedule an appointment for me," and that turns into a specific command for a calendar app.
- 11:44
In fact, in our examples, we actually have that. Um, what if you wanna string together multiple things? Hey, that's just a list of commands, right? Um,
- 11:55
kinda. [laughs] What's the... The problem with this is if I want these to kinda thread through to each other, this is a simple example, so it's just going input, you know, run the command, get the output, go to the input, et cetera, et cetera, et cetera, et cetera.
- 12:10
Um, what if you have something that expects multiple arguments? What if you wanna reuse a result? Sure seems like you need variables and other things like that here. Um, so we asked ourselves, is there a thing here where you can imagine you can just generate code and
- 12:29
just take the same approach where types are all you need? So what if you could just define, here's all the methods that I want you to be able to call.
- 12:36
Come back with some code that only calls those methods, and then generates a program like this.
- 12:43
The problem is that you really wanna have some sort of sandboxing and safety constraints in place, right? And so you might start saying, "I need availability. I can't just endlessly loop here.
- 12:55
Um, so I'm not gonna allow loops. I'm not gonna allow lambdas and whatever." And the, the problem is that even if you decide, I'm gonna pick a subset of a language like JavaScript or Python or whatever you have, um, the language models have seen so much of that code that they're gonna draw outside the lines, and then
- 13:12
you'll hit this failure case, and then you just won't get a result. You won't get a bad result. You just won't get a result that conforms to what you're expecting.
- 13:19
And then you still have to worry about sandboxing, and then there's all these questions about synchronous versus asynchronous APIs and all this other stuff too, that language models don't tend to understand because I guess most people don't either.
- 13:31
Um [laughing] So what we actually [laughs] have been trying is, uh, we generate a fake language. Uh, we have the language models generate a fake language still based on the types,
- 13:43
but it's in the form of JSON, actually. And so you have things like refs, and refs are just pri- you know, references to prior results. And [laughs] if you're familiar with like, you know, if you're a compiler, this will look, may look like SSA.
- 13:54
It might look like an AST, whatever. Um,
- 13:58
but we use that to construct a fake TypeScript program in memory as well, and use that to make sure that not just are you calling all the, only the methods that are available to you, that you can only do certain actions, but also that the inputs from prior steps, um, matches up with the types that you're defining
- 14:18
from your API. And so that kinda comes back to types are all you need. We have another really simple example for...
- 14:28
We have a math schema. This is basically a calculator in sheep's clothing. So if you go back and we run this here,
- 14:38
we have another prompt that's an abacus. That's the closest thing to a calculator I could get. Um, if we could say something like add one to forty-one and then divide by seven.
- 14:51
Now, basically what happened here is we made a language model good at math. So we've also solved a whole other set of problem, right? [clapping] Yeah. Um, more seriously, though, uh, so at each of these steps, we're actually performing, not having the language model call a method, perform an operation.
- 15:14
And if you actually look at the code here, um, math main.
- 15:20
This is all under fifty lines of code. We are able to do the same sort of translation. We have a separate thing called a program translator. And in that program translator, when you are successfully able to validate your results, you know, you say, "If this thing is a success or not a success, just jump out, otherwise do
- 15:38
some stuff with it." We have this evaluate function, and this evaluate function takes a callback, and that callback is just sort of like this instruction interpreter. And so you can do this with objects, you can do this with a function, with a switch case or whatever.
- 15:52
Um, but the point is that this actually allows you to do some richer tasks. Um, now, there are other approaches for many of these things, and they overlap with what TypeChat does.
- 16:04
But the cool thing is that TypeChat is able to actually give you this level of validation for both JSON and programs. Um, and it's something that we're also experimenting with, with other languages too.
- 16:15
So for example, people at this conference have been saying, "Yeah, I, you know, TypeScript is very cool," and I agree with them because I work on TypeScript. Um, but how would I wor- make this work with Python?
- 16:27
And so we have been experimenting with this, and we've been getting fairly good results. I'm able to do something like the coffee shop with a very similar approach using types.
- 16:37
Um, I'm able to do something similar with the calculator app, just defining methods on a class with comments and all this other stuff that helps the model do a little bit better, and it works really well.
- 16:50
Um, we can even do more complex examples too. Like we have this CSV example. Um, maybe I want to be able to... Well, I'm not gonna get into... Oh, pipenv.
- 17:04
The demos, demo gods are gonna kill me here. [laughs]
- 17:10
That. Brutal. Okay. I can just create a program that does this now. I have this entire API that grabs columns and is able to perform certain operations, and then do joins that do filtering and joining and all this other stuff as well because it just sort of does this selection based on Booleans.
- 17:34
So read a CSV, find all the values that equal N/A, and then drop the rows. And so this becomes this sort of powerful approach, and this is just a prototype of the Python stuff that we've been working on as well.
- 17:45
Um, it's not primetime, and if you wanna talk to me about it, I'm definitely game.
- 17:52
So what I want from you all is to try TypeChat out. Reach out. What I'm here at this conference for is to learn about what you're all trying to build, trying to help bridge the gap as well between what we're all learning on the cutting edge and making that more accessible to everyday engineers who have been at
- 18:10
this more precise end of the spectrum, bringing the power of these language models that are so rich to the traditional apps. Thank you very much. Come see me at the Microsoft booth.
- 18:19
I'll be hanging out for a little bit. And thank you. [clapping] [upbeat music]