AI Engineer Summit 2023
The AI Pivot: With Chris White of Prefect & Bryan Bischof of Hex
Read the talk
The AI Pivot: Building Useful Features Around Unreliable Models
Prefect and Hex show how AI investment becomes product work: remove workflow friction, evaluate concrete tasks, preserve the core experience, and design for failure.
From a talk by Chris White, Bryan Bischof and Brittany Walker
Before you start: Basic familiarity with APIs, SQL, and data workflows will help; no machine learning training experience is required.
Where AI fits in an existing business
Where does AI fit when a startup already has customers demanding features, a product roadmap, and hiring and fundraising to manage? At the AI Engineer Summit 2023, Brittany Walker, a principal at early-stage investor CRV, puts that question to two people building from different starting points. Chris White is CTO of Prefect, which provides a workflow orchestration developer tool and remote orchestration service. Bryan Bischof leads AI at Hex, a data science notebook platform. Both have math PhDs: White studied non-convex optimization before moving through data science into developer tools; Bischof brings roughly a decade of data science and machine learning experience.
For Prefect, the connection starts with its users. Data engineers and data scientists already orchestrate data movement, and semantic search, retrieval, and summarization introduce more of that work—including possible changes to ETL around vector databases. Prefect is general purpose, so those use cases do not require an entirely new product category. White then reduces the production problem to a familiar dependency: an expensive, brittle, nondeterministic remote API. Much of the tooling needed to operate data APIs should transfer. The team also creates Marvin, a separate repository that gives it a reason to experiment publicly and keep learning.
Hex starts from the analyst’s work. Data science combines business knowledge, creativity, and sometimes difficult programming. Remembering matplotlib syntax consumes attention without necessarily improving the analysis. Removing that tedium creates room for the parts of the job that require judgment. Bischof also expects AI assistance to become table stakes for data platforms, making the investment both an opportunity to improve the workflow and a requirement for remaining useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the friction to remove
Prefect gives its two AI efforts different success criteria. Marvin is an open-source project the company maintains rather than sells; its return includes direct conversations with people experimenting with LLMs. White says the head of AI speaks with users multiple times a day. The core product has a narrower test: what happens when a workflow fails, and how quickly can a user understand and recover from it? Dashboard error summaries help users triage failures. Relevant measurements include time to find the right page and time to debug the workflow.
Hex’s charter is to make the product feel magical, but its example is concrete: a long SQL query containing several common table expressions, or CTEs, is awkward to work with. CTE Explode breaks the query into separate notebook cells and chains them together. Bischof describes it as straightforward to build and something he had wanted for eight years. The useful capability is the transformation of an existing working artifact into a form that is easier to inspect and edit.
For example, consider a query that totals paid orders by customer and keeps customers whose total reaches 100:
sql
WITH paid_orders AS (
SELECT customer_id, amount
FROM orders
WHERE status = 'paid'
),
customer_totals AS (
SELECT customer_id, SUM(amount) AS total
FROM paid_orders
GROUP BY customer_id
)
SELECT customer_id, total
FROM customer_totals
WHERE total >= 100;
An exploded arrangement would put the paid-order selection in a cell named paid_orders, the aggregation in a downstream cell named customer_totals, and the final filter in a cell that reads that result. These teaching names and values illustrate the feature Bischof describes: preserve the dependency chain while making its stages individually accessible. This is the kind of small, repeated inconvenience Hex uses to decide where AI assistance belongs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Small teams, distinct returns
Hex begins with one hacker building a promising prototype, then expands to a couple of people. Bischof reports keeping the team around five people despite a roadmap large enough for twenty engineers. That constraint is deliberate: it forces prioritization. Prefect hires Adam to own most AI work while leaving participation open to others; one interested engineer effectively transfers to that team to work on AI full-time.
White reports that Prefect’s small AI additions have already improved core-product time to value, without giving a numerical improvement. Regular conversations with companies beginning to productionize AI provide a separate return. Hex looks for a different signal: how often users reach for Magic. Repeated use across cells and projects would indicate that assistance is becoming part of the workflow. Supporting every part of the platform is therefore a design goal, not a claim that universal adoption has already happened.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the integration that matters
One of Bischof’s first questions on joining Hex in February is how the team evaluates its AI features. His ML experience makes objective framing and evaluation prerequisites for useful iteration. He recalls a much smaller evaluation-tool market then, compared with his estimate of more than twenty-five companies by the panel. Hex chooses to build its own evaluations because they need deep hooks into the platform and should run as close to production as possible, including using production where feasible.
Vector infrastructure gets a different answer. After six or seven years working with vector search, including FAISS and early Pinecone, Bischof does not want to build another vector database. Hex works with LanceDB, then invests heavily in a customized retrieval implementation for its retrieval-augmented generation, or RAG, pipeline.
| Component | Hex’s choice | Reason |
|---|---|---|
| Evaluations | Build internally | Deep production integration |
| Vector database | Use LanceDB | Avoid rebuilding complex infrastructure |
| Retrieval behavior | Customize heavily | Fit the application’s context needs |
Buying the storage layer leaves substantial application engineering to do. The boundary follows where the complexity creates value.
Prefect likewise has no interest in building a vector database. White says its experiments with Chroma and Lance are not yet in production. He describes a common progression: explore with a framework, extract the prompts that work, then call the API directly without the experimental framework in the middle. At that point, existing API tooling may already provide the required monitoring and observability. For the integrations White describes, Prefect uses its existing stack and makes no new tooling purchases.
There are still gaps worth filling. Bischof, a Weights & Biases alumnus, wants experimentation and iteration tightly connected to LLM observability. He is watching Rivet by Ironclad for its approach to experimentation; a strong connection to observability is something he would like to see, rather than an integration he says Hex has adopted. White’s priority is the machine-to-machine interface: natural-language inputs are useful, but backend processes need typed outputs they can consume. He sees function calling as a step toward that contract.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The prototype hides the long tail
For Prefect’s core AI additions, White reports no unexpected tooling or productionization obstacles. The harder problem is organizational enthusiasm. Engineers propose using AI for everything, including monitoring itself. The team has to bring the discussion back to specific sources of user friction and faster access to useful information. Scope control remains an ongoing task.
Hex encounters a related problem in estimates. Early AI experiments can work impressively on a few cases, making a feature look nearly finished. The distance from those successes to robust behavior is much longer than the prototype suggests. Bischof responds by turning enthusiasm into concrete evaluation cases:
- Start with the requested capability: what should Magic do?
- Ask for a specific prompt and the situation in which it will be used.
- Establish the expected behavior for that case.
- Ask whether a newly hired data scientist with little company context could do the same task.
That last question tests whether the request is feasible with the available context, rather than treating a broad wish as a specification.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A separate experiment, or part of the product?
A separate repository gives Prefect room to learn without committing its entire product to a new direction. Marvin still creates maintenance obligations, but even low adoption would have taught the team how to build interfaces to LLM APIs. Its actual traction brings new contacts working on AI and adjacent data problems.
The core product needs a sharper boundary. Because Prefect orchestrates arbitrary workflows, helping users write arbitrary code creates effectively unbounded scope. White sees little advantage in competing with OpenAI or GitHub at that task. Prefect explores putting AI directly into its SDK, then rejects the direction as too broad and insufficiently additive. Narrow improvements to the existing workflow offer a clearer reason for the feature to exist.
Hex draws its boundary around a single product experience: Magic augments Hex. During one sprint, the team creates Crystal Ball, an experience Bischof describes as capable, polished, and pleasant to use. But it pulls users away from the core Hex workflow. The CEO identifies that it is creating a separate ecosystem for Magic. Hex kills Crystal Ball despite its appeal, because the goal is to improve Hex for all its users rather than split the experience into two products.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Roll out to users who can expose the limits
Hex begins with a private beta and expands toward public beta as quickly as possible. Its early users are engaged, technically capable, and prepared to work with stochastic behavior inside already complicated data science workflows. The team then broadens participation across technical ability, so the product does not become useful only to the people best equipped to work around its limitations.
Marvin has a simpler release path: the team builds until it is proud of the repository, makes it public, and tweets about it. Prefect’s enterprise features require more care. Financial-services and healthcare customers are especially sensitive to data leaving their environment. White describes opt-in features with tooltips explaining that activation sends some data to a third-party provider. The historical error-summary announcement makes the account distinction precise: existing accounts had to opt in, while new accounts had the feature enabled by default. Consent and disclosure therefore need to be understood at the account level, not reduced to a blanket claim that every deployment was opt-in.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test the contract and the task
After launch, Bischof uses Magic throughout his working day—including to analyze Magic’s own product performance. That creates a feedback loop between direct experience and conventional product analytics: using the feature exposes friction, while analysis helps locate patterns of success and failure. The recursive quality is amusing, but the method remains ordinary data science applied to the product.
Prefect initially tests untyped string outputs with another LLM performing semantic comparisons. White acknowledges problems with that approach, while finding it useful. Moving toward typed outputs in Marvin makes tests easier to specify: an output has an expected type, and a known case can also have an expected value. Typed handshakes make components easier to compose and test, though structural validity does not establish semantic correctness. This is the panel’s historical Marvin design discussion; the repository now describes Marvin 3.0 built on Pydantic AI, rather than the implementation used in 2023.
Walker then asks about improvement over time, mentioning fine-tuning, RLHF, and RLAIF as possibilities—not techniques either company establishes it uses here. Bischof turns to the opposite problem: behavior can change underneath a deployed application. He recalls a paper associated with Matei Zaharia about changing model behavior. The likely reference is How is ChatGPT’s behavior changing over time?, first submitted in July 2023 and comparing March and June versions, rather than a paper published in June. Bischof reports that his evaluations began failing after a provider-side change even though he had not changed the endpoint he called. An unchanged endpoint need not imply an unchanged model version: OpenAI’s 2023 API announcement documented automatic upgrades of generic model names alongside dated versions. That explains the possibility without identifying Hex’s actual configuration.
The response is to define what performance means for the task. Bischof argues against substituting latent-distance distributions, KL divergence, or BLEU similarity for a clear account of whether an output does what the user needs. His preferred approach is to reduce a task to explicit yes-or-no checks wherever possible. Nuanced tasks may demand substantial work to find those checks; difficulty defining them is a reason to examine the task more closely. He frames this as established ML practice, recalling years of criticism of BLEU for these outputs and advocacy for composability. White connects that process back to orchestration: a changing, nondeterministic experimentation workflow still has structure that can be managed, even when the orchestrator does not choose the evaluation metric.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make agent workflows observable
When the panelists question each other, Bischof asks what workflow engineering can teach people building agents from DAGs and structured request-response chains. White starts with failure modes:
- Runaway processes: consider how a workflow can continue beyond its intended bounds.
- Unusual inputs and outputs: install observability early enough to catch odd behavior near its origin.
- Caching: use it where it helps, while recognizing that it introduces hazards of its own.
- Returned code: prioritize monitoring and debuggability when generated code is immediately evaluated or executed.
The operational layer becomes especially valuable as behavior grows harder to inspect inside individual components.
White then asks about hallucinations. A user may notice a strange result or see code fail, but the interface builder still has responsibility for reducing those failures. Bischof describes searching for advice and repeatedly finding recommendations to improve retrieval-augmented generation. That also matches Hex’s experience: Bischof attributes a reduction in Magic’s hallucinations to more carefully designed, application-specific retrieval, without reporting a reduction metric. He does not disclose the mechanism, saying it is absent even from his own book. The useful engineering boundary is clear: retrieval design matters, but this discussion does not provide an algorithm to reproduce or establish that hallucinations have been eliminated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The work behind the capability
White closes by challenging the reflex to put a chat interface everywhere: “chat is a product. AI is a tool.” His preferred investment is in machine-to-machine interfaces that let developers incorporate AI directly into their systems. That follows the same concern as typed outputs: make the capability usable inside a larger workflow.
Bischof’s closing warning is about the work itself. Exciting capabilities do not make the implementation process uniformly exciting. Much of it is tedious, and substantial data engineering lies between a promising experiment and a useful production feature. He sees that as a continuation of machine learning practice: the result can be rewarding, but getting there requires sustained attention to the unglamorous parts of the system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Python framework for structured outputs and AI workflows, with current installation instructions and examples.
Ironclad's introduction to its open-source visual environment for composing, testing and running AI graphs.
The official repository for LanceDB's embedded retrieval library.
Further reading
- Magic, for everyoneArticle
Bryan Bischof introduces Hex Magic's public beta, including CTE Explode, code assistance and human-review expectations.
Prefect's historical launch announcement explains failed-flow summaries and their account-level controls.
The original July 2023 study comparing March and June GPT-3.5 and GPT-4 behavior across four tasks.
OpenAI's 2023 announcement of function calling, model-alias upgrades and version-pinning options.
Updates since the talk
- Managing MagicArticle
Hex's November 2023 guide to improving SQL assistance through metadata, table retrieval and controls for string hallucinations.
Read the complete timestamped transcript
- 0:00
[upbeat music] Really excited to be moderating this panel between two of my favorite people working in AI.
- 0:21
Um, I'm Brittany. I'm a principal at CRV, which is an early-stage venture capital firm investing primarily in Seed and Series A startups. Chris, why don't you give us a little bit about yourself?
- 0:31
My name's Chris. I'm currently the CTO at Prefect. We're a workflow orchestration company. We build a workflow orchestration dev tool and sell an orchestration-- remote orchestration as a service.
- 0:42
Uh, little background. So I started kind of my journey into startup land and eventually AI and data. Um, got a PhD in math focused on non-convex optimization, which I'm sure a lot of people here are into.
- 0:55
Um, and then eventually, you know, data science and then into the kind of dev tool space, which is where I'm at now.
- 1:02
Awesome. And Bryan, fill us in on your side.
- 1:04
I'm Bryan. Uh, I lead AI at Hex. Hex is a data science notebook platform. Um, sort of like the best place to do data science workflows. Um, I was gonna say I started my journey by getting a math PhD, but he kinda already took that one.
- 1:18
Um, that's kinda awkward. Um, yeah. I've been doing data science and machine learning for about a decade. And, uh, yeah, currently find myself doing AI, as they call it these days.
- 1:28
Awesome. So both of you are at relatively early-stage startups. And as we all know, early-stage startups have a number of competing priorities, everything from hiring to fundraising to building products.
- 1:41
And one might say it would be a lot to kind of take a moment and just say, "What is this AI thing? What the fuck do we do with this?" [chuckles]
- 1:50
And so I'm wondering, how did you decide that AI was something that you really needed to invest in when you already had, you know, a established business, growing well, lots of users, lots of customers, presumably placing a lot of demands on your time?
- 2:03
So Chris, I would love to hear from you on how you guys thought about that choice.
- 2:07
Yeah. So there are a couple of different dimensions to it for us. So we are, you know, a workflow orchestration company, and our, our main user persona are data engineers and data scientists, but there's nothing inherent about our tool that requires you to have that type of use case.
- 2:21
And so, uh, one thing, one dimension for us is, right, we assumed that a big component of AI use cases were gonna be data-driven, right? Like semantic search or, uh, like retrieval, summarization, these sorts of things.
- 2:34
So just we wanted to make sure that, you know, we had a seat at the table to understand how people were productionizing these things, and like were there any new ETL considerations when you're, you know, moving data between maybe vector databases or something.
- 2:46
So that was one thing. Uh, another one that, uh, I think is interesting is when, when I look at AI going into production, I see basically a remote API that is expensive, brittle, and non-deterministic, and that's just the data API to me.
- 3:01
And so, right, if we can orchestrate these workflows that are building applications for data engineers, presumably a lot of that's gonna translate over. And so-- And I mean, last, like, you know, I'm sure the reason most people are here now is it's, you know, it's fun, and so we just wanted to learn in the open.
- 3:16
So we did end up just kind of creating a new repo f- uh, called Marvin, that I think Jason mentioned in his last talk, um, just to kinda keep up, you know, be incentivized to keep up.
- 3:28
And Bryan, you were literally brought on board to Hex to focus on this stuff. Would love to hear more about how that decision was made and how you've spent your time on it.
- 3:37
Yeah. I think a couple things. One is that data science is this unique interface between sort of like, like business acumen, creativity, and like pretty, like, difficult sometimes programming.
- 3:50
And it turns out that, like, the opportunity to unlock more creativity and more business acumen as part of that workflow is a really unique opportunity. I think a lot of data people, um, the favorite part of their job is not remembering matplotlib syntax.
- 4:05
And so the opportunity to sort of like take away that tedium is a really exciting place to be. Also, realistically, um, any data platform that isn't integrating AI is almost certainly going to be dooming themselves to the now and sort of it'll be table stakes pretty soon.
- 4:21
And so I think missing that opportunity would be pretty criminal.
- 4:25
Yeah. I totally agree with that. So you decided that you were gonna go ahead and do this. You were gonna go all in on AI. What criteria did you evaluate when you were determining how you were gonna build out these features or products?
- 4:39
Did you optimize for how quickly you could get to market, how hard it would be to build, ability to work within your existing resources? What criteria did you consider when you were saying, "Okay, this is how we're actually gonna take hold of this thing"?
- 4:53
So for us, I guess there's two different angles. There's the kind of just pure open source Marvin project. Not-- You know, it is a product, but not one that we sell, just one that we maintain.
- 5:02
Um, and then we do have some AI features built into our actual core product. And I think, uh, they have slightly different success criteria. So for Marvin, it's mainly just, um,
- 5:13
getting, uh, to see how people are experimenting with LLMs and just talking to users directly, right? It just kind of gives us that avenue and that audience, and so that's just been really useful and insightful for us.
- 5:24
So we just get on the phone. I mean, our head of AI gets-- you know, talks to users at least, you know, a couple times a day. Um, and then for our core product...
- 5:33
So one way that I, I love to think about dev tools and think about what we build is, uh, failure modes. So like I like to think of choosing tools for what happens when they fail.
- 5:43
Can I quickly recover from that failure and understand it? And so a lot of our features are geared towards that sort of kind of discoverability. And so for AI, it's kind of the same thing.
- 5:54
It's like quick error summaries, uh, shown on the dashboard for quick triage, and then measuring success there is, like, relatively straightforward, right? It's like how quickly are users kind of getting to the pages they want, and how quickly are they, uh, debugging their workflows, so.
- 6:09
Like very quantifiable.
- 6:11
Yeah. Would love to hear from you too.
- 6:13
Yeah. My team's charter is relatively simple. It's make Hex-- make using Hex feel magical. And so ultimately, we're constantly thinking about sort of what the user is trying to do in Hex during their data science workflow and making that as low friction as absolutely possible and giving them more enhancement opportunities.
- 6:31
So a simple example is I don't know how many times y'all have had a very long SQL query that's made up of a bunch of CTEs, and it's a giant pain in the ass to work with.
- 6:40
So we build an explode feature. It takes a single SQL query, breaks it into a bunch of different cells, and they're chained together in our platform. This is like such a trivial thing to build, but it's something that I've wanted for eight years. [laughs]
- 6:52
Like I've, I've done this so many times, so annoying. And so thinking like that makes it really easy to make trade-offs in terms of what is important and what we should focus on.
- 7:01
And so in terms of like how we think about, um, yeah, like where our positioning is, it's really just how do we make things feel magical and feel smooth and comfortable.
- 7:11
And how did you reallocate resources beyond, you know, they obviously hired you. That was a, a great step in the right direction. But what else did you do to actually get up and running in terms of operationalizing some of this stuff?
- 7:23
Yeah. I think we, we kept things pretty slim, and we continue to keep things pretty slim. Um, we started with one hacker. Um, he built out a very simple prototype that seemed like it showed promise, and then we started building out the team.
- 7:34
We scaled the team to a couple people, and we've always remained as slim as possible while building out new features. Um, these days, I have a roadmap long enough for twenty engineers, and we continue to stay around five, and that's not an accident.
- 7:47
Basically, like ruthless prioritization is definitely an advantage.
- 7:52
And Chris, you guys wound up hiring a guy as well, right?
- 7:54
Yeah. We hired a great guy. His name's Adam. Um, so he definitely owns most of our AI, but also rate like anyone at the company that wants to participate.
- 8:03
And so there was one engineer that got really into it and is a-- for, you know, all intents and purposes, has like effectively switched to Adam's team and is now doing AI full-time.
- 8:13
Yeah. So you guys are, are really dedicating a lot to solving this problem, including the hiring of, of two people, [laughs] one on your side and one on Bryan's. Um, so presumably you're gonna be looking for a return on that investment.
- 8:25
So how do you think about what a successful implementation of an AI-based feature or product looks like?
- 8:33
For us, I would say that already we've hit that success criterion. So now the question is like further investment or just kinda keep going with the way that we're doing it.
- 8:41
But, uh, so big thing was time to value in the core product. That we can just easily see has definitely happened with just the few sprinkles of AI that we put in.
- 8:51
So we'll just kind of keep pushing on that. Um, and then kind of like I said in the beginning, just getting involved in those conversation-- those really early conversations about companies looking to put AI in production, and we've been having those on the regular now.
- 9:06
So I would say like already feels like it was well worth the investment.
- 9:11
What about you guys? You obviously just had a, a big launch the other day too. Curious how you thought about success for that.
- 9:16
Yeah. Once again, it's sort of like how, how frequently do our users reach for this tool? Um, ultimately, Magic is a tool that we've given to our users to try to make them more efficient and have a better experience using Hex.
- 9:28
And so if they're constantly interacting with Magic, if they're using it in every cell and every project, then that's a good sign that we're succeeding. And so to make that possible, we really have to make sure that Magic has something to help with all parts of our platform.
- 9:42
We have a pretty complicated pro-- platform that can do a lot, and so finding applications of AI in every single aspect of that platform has been one of our sort of like, you know, north stars, and very intentionally so, to make sure that we're, you know, making our platform feel smooth at all times.
- 10:01
Awesome. Well, let's, let's move on to the next section, which we're gonna talk about what-- how you guys actually built some of these features and products. Since we're all here at the AI Engineer Summit, I assume we all have an interest in actually getting stuff done and putting it into prod.
- 10:16
So when you were making some of these initial determinations, Bryan, how did you guys determine what to build versus buy?
- 10:22
Yeah. So from day one, I think the quest-- one of the first questions I asked when I joined is what they were doing for evaluation. And you might say like, "Okay, yeah, we've heard a lot about evaluation today," but I would like to remind everyone here that that was February.
- 10:35
Um, and the reason that I was asking that question already in February is 'cause I've been working in machine learning for a long time, where evaluation sort of like gives you the opportunity to do a good job.
- 10:46
And if you've done a poor job of objective framing and done a poor job of evaluation, you don't have much hope. And so I think the first thing that we really looked into is evals.
- 10:55
And back then, there was not twenty-five companies starting evals. Um, there are now more than twenty-five. But ultimately, we made the call to build, um, and I'm very confident that that was the right call for a few reasons.
- 11:08
One, eval should be as closest to production as possible, is literally like using prod when possible. And so to do that, you have to have very deep hooks into your platform when you're moving at the speed that we try to move.
- 11:21
That's hard for a SaaS company to do. On the flip side, we chose to not build our own vector database. I've been, you know, doing semantic search with vectors for six, seven years now, and I've used [laughs] o-open source tools like Face and Pinecone back when it was more primitive.
- 11:40
Unfortunately, a lot of those tools are very complicated. And so having set up vector databases before, I didn't wanna go down that journey. So we ended up working with LanceDB and sort of built a very custom, uh, implementation of vector retrieval that really fits our use case.
- 11:57
That was highly nuanced and highly complicated, but it's what we needed to make our RAG pipeline really effective. So we spent a lot of effort on that. Um, so ultimately just sort of where is the complexity worth the, the squeeze?
- 12:13
And Chris, what about you guys? How did you do that?
- 12:16
So I have a couple of different kind of things that we decided on here, and some of which are still in, in the works. Um, vector databases, million percent agree with that.
- 12:26
Like, we would never build our own. Um, we haven't had as much need of one, I think, as Hex, but we've done a lot with both Chroma and with Lance, but neither in production yet.
- 12:37
Um, so none of those use cases are, are in prod. And so the way that I've... Y- the exposure that I've seen about people actually integrating AI into, you know, their workflows and things is there's a lot of experimentation that happens, and then you kind of wanna get out of that experimental framework maybe by just looking at
- 12:56
all of the prompts that you were using, and then just using those directly yourself with no framework in the middle. Um, and then once you're kind of in that mode, like I was saying before, like you're just...
- 13:06
At the end of the day, you're interacting with, with an API, and there's lots of tooling for that. And so I kind of see a lot of... The decisions at least that we had to confront on build versus buy is like, it's another just kind of tool in our stack.
- 13:20
Do we already have the sufficient dev tooling to support it, make sure it's observable, monitorable, and all this? And, and we did, so we didn't do any buying. It was all, all build.
- 13:30
Yeah. And as, and as you mentioned, I think, you know, you talked about many, many eval startups. I think we're all familiar with the, the broad landscape of vector databases as well.
- 13:40
Um, are there any pieces of your infrastructure stack that you wish people were building or you wish people were kind of tackling in a different way than, than what you've seen out there so far?
- 13:51
Either one of you.
- 13:54
Yeah. I mean, [chuckles] I think it would've been a hard sell to, to sell me on an eval platform. I think there was some opportunity to sell me on, like, an observability platform for LLMs.
- 14:07
Um, I've looked at a f- quite a few, and I will admit to being an alumni of Weights & Biases, so I have some bias. Um, but that being said, I think there is still a golden opportunity for a really fantastic, like, experimentation plus observability platform.
- 14:25
One thing that I'm watching quite carefully is Rivet by Ironclad. It's an open source library, and I think the way that they have approached the experimentation and iteration is really fantastic, and I'm really excited about that.
- 14:39
If I see something like that get laced really well into observability, that's something that I'd be excited about.
- 14:45
Anything to add on your side?
- 14:47
I think, like, small addition to what Bryan said, which is just more focus on kind of the machine to machine layer of the tooling. And so I think a lot, you know, right at the end of the day, the input is always kind of this natural language string, and that makes a lot of sense.
- 15:05
But the output, making it more of a guaranteed typed output, like with function calling and, and other things, I think is one step in the journey of making, of integrating AI actually into backend processes and machine to machine processes.
- 15:22
And so any focus in that area is where, you know, my interest gets piqued for sure.
- 15:28
Yeah. Totally. Okay. So you have, you have all your people, and you have all your tools, and then you're obviously completely good to go and fully in production. JK, we all know it doesn't work that way.
- 15:39
What challenges-
- 15:40
It's never working, right. [laughs]
- 15:42
What challenges did you run into along the way? Maybe ones that you, that you didn't expect or that were larger obstacles than you would've thought.
- 15:47
So I don't... Integrating AI into our core product, I would say from a tooling and developer perspective and, you know, productionizing perspective, none. Culturally, though, I would say we definitely hit, you know, some challenges, which is that when we first were like, "All right, let's start to incorporate some AI and, and do some ideation here," right?
- 16:10
A lot of engineers just started to throw everything at it. Like, "We should... It should do everything. It can monitor itself for, like, all of this stuff." And it was like, "All right, all right.
- 16:21
Everyone needs to kinda, like, backtrack." And so just that internal conversation of, like, you know, getting buy-in on, like, very specific focus areas, which, you know, at the end of the day, where, where we are focused is that just removal of user friction, whether it's through design or just through, like, quicker surfacing of information that AI just,
- 16:40
like, lets you do in a more guaranteed way. But yeah, uh, restraining the enthusiasm was the biggest challenge-
- 16:47
Yeah
- 16:47
... for sure, and it still exists to this day. [laughs]
- 16:50
Everyone wants to be-
- 16:51
It's a good challenge
- 16:51
... an AI engineer, right?
- 16:52
Yeah, exactly.
- 16:53
Yeah. What about you guys? Did you have similar or different issues?
- 16:56
That's interesting. It's a, like, similar flavor. Um, it's a little different instantiation, which is to say that, like, you know, I've never met an engineer that's good at estimating how long things take.
- 17:06
Um, and I would say that, like, that is somehow exacerbated with AI features because then you, your first few experiments show such great promise so quickly, but then the long tail feels even longer than most engineering corner case triage.
- 17:23
Just such a long journey between, "We got this to work for a few cases, and we think we can make it work," to, "It's bulletproof," is even more of a, a challenging journey.
- 17:34
And I-- Yeah, this, this, like, over-enthusiasm, I think, y- yeah, slightly different, uh, instantiation, but similar flavor.
- 17:41
Whenever you're on that journey, how, um, how are you testing and tracking along the way, if at all, which is totally, yeah-
- 17:48
Yeah, I mean-
- 17:49
... maybe just making that feel
- 17:50
... I, to be a broken record, a lot of, like, robust evals. [laughs] Like, trying really hard to codify things into evaluations, trying really hard to codify, like, if someone comes to us and says, "Wouldn't it be great if Magic could do X?"
- 18:04
We, we sort of pursue that conversation a little bit further and say, like, "Okay, what would you expect Magic to do with, with this prompt? What would you expect Magic to do in this case?"
- 18:13
And kind of get them to kind of, like, vet that out, and then sort of, um, using this, like, barometer of could a new data scientist at your company with very little context do that?
- 18:25
And sort of that, like, you know, cutting edge around what's feasible and what's possible.
- 18:31
Yeah, that's r- that makes a lot of sense. And, you know, one of the reasons I was excited to have the two of you up here together is because, you know, while Prefect has some elements of AI in the core product, as you mentioned, probably you're best known for the Marvin, uh, project that you guys have put
- 18:45
out there, which is kind of a, a standalone, uh, project. Which is a really interesting phenomenon that I'll say that I've kind of observed in this current wave of AI, which is, you know, companies that maybe weren't doing AI previously launching entirely separate brands, um, essentially alongside the core product.
- 19:02
So would love to understand more of what were your user experience considerations when you were building out, you know, Marvin as a separate product versus Prefect? What freedom did that allow you?
- 19:13
What restrictions did you still have?
- 19:15
Yeah, that's a good question. So a few different angles there. I think one kind of philosophical angle, um, is, you know, we try to do things that maximize our ability to learn without having to go full commitment.
- 19:32
And so I think starting a new open source repo, like, right, we definitely have some, uh, ties to it now. We have to maintain it. But past that, it's not all that high of a cost.
- 19:41
But, like, if it... You know, it's all upside, basically. If no one notices it, no big deal. We learned a little bit more about how to, you know, write APIs that, you know, interface with all of the different LLMs, for example, or something like that.
- 19:54
Um, or if it does take off, which, you know, it basically did for us, um, we got to meet all these new people who are working on interesting things like AI and data adjacent.
- 20:05
Um, but for, uh, the core product, this was maybe more, I guess, kind of interesting and, and Bryan, I'd be curious to hear about how much you had to, like, really focus some of your prompts to the use case that you cared about.
- 20:20
So Prefect is a general purpose orchestrator. And so the reason I, I say that again is our use case scope is, like, technically infinite. And so helping people write code to do completely arbitrary things is definitely not a value add we're gonna have over the engineers at OpenAI or at GitHub or something else.
- 20:40
So we knew that we couldn't invest in, like, that way of integrating AI. Um, and so then the next question was, like, okay, so then what are just the marginal ads?
- 20:51
And that's kind of where we landed, you know, where we are today. Um, but there was-- We did put energy initially, like, can we put this directly in, like, the SDK or something like that?
- 21:01
And just very quickly realized that it was just too large of scope, and at that point you might as well just have the user do it themselves. And, like, there's-
- 21:08
Yeah
- 21:09
... we're not adding anything to that workflow.
- 21:10
Yeah. Yeah, and on, and on the flip side, you know, Magic has been kind of a part of Hex, uh, basically it seems like since inception from the outside.
- 21:19
Obviously, we've all seen, again, a number of text to SQL players out there. We can make arguments about [chuckles] whether or not those should exist as standalone companies. But I'm curious, you know, how you guys had to think about UX considerations when you were building out Magic in the context of the existing Hex product.
- 21:35
Ultimately, I've been really fortunate to kind of, like, work with a great design team who sort of... They're just excellent. But the question about, like, how does Magic feel?
- 21:46
Magic is not its own product. I think that's one thing that's been important from early on. Magic is not a product. Magic is an augmentation of our product. So it is a collection of features that makes the product easier and more comfortable to use.
- 22:01
That is an easy sort of thing to keep in mind when deciding how to design, because it allows us to say, "Okay, like, we don't want this to distract from the core product experience."
- 22:12
I can tell a story. We had one sprint where we de- we designed something called Crystal Ball, and Crystal Ball was a really sick product. Um, it did exactly what we wanted to do, and it felt wonderful.
- 22:25
However, ultimately, it drew the user away from the core Hex experience. And very quickly, our CEO rightly was like, "I feel like this is kind of splitting Magic out into its own little ecosystem."
- 22:39
And that made it kind of clear that that might be the wrong direction to go. So even though Crystal Ball did feel really good and had a really incredible capability behind it, and frankly, the design on Crystal Ball was beautiful, the problem with that was it pulled us away from what we were really trying to do, which
- 22:56
was make Hex better for all of our users. Every Hex, like, consumer should be able to benefit from Magic features, and that was starting to split that. And so we literally killed Crystal Ball, uh, despite it being a really cool experience, uh, for that reason.
- 23:12
So genuinely, we've really stuck to the, like, it's one platform and Magic augments it.
- 23:18
Yeah. That makes a lot of sense. And obviously, you know, Hex already had a relatively sizable user base at the time you guys launched this. So I'm curious, how did you think about the rollout, like, just in terms of what users you gave it to and what timeline, what marketing to do?
- 23:33
All of those types of considerations.
- 23:35
Yeah. Generally, we start with a private beta, and then we, as quickly as possible, expand that to a sort of, like, public beta. Our goal is to find people that are, like, engaged with the product, and they are prepared for some of the limitations of AI tools.
- 23:50
Uh, stochasticity has come up many times, and ultimately, we're expecting the user to work with a stochastic thing. Al- als- also, they're working with something very complex, which is data science workflows.
- 24:02
So we're looking for people that are pretty technical in the early days. Then we wanna keep scaling and scaling to include the rest of the distribution in terms of technical capab- capabilities so that we can make sure that it's really serving all of our users.
- 24:16
And on the flip side, again, you had a little bit maybe more flexibility with the rollout just given, uh, it was a new repo. I'm curious if that was different, similar to what Bryan's talked about.
- 24:26
Well, so yeah. Well, the repo, no. It was... We hacked on it, you know, we had fun with it. We got it to a place where we felt proud of it, and then we clicked Make Public and then tweeted about it, and that was, like, the end of that.
- 24:38
Um, so that was just pure fun. Um, but for integrating AI into our core product, I mean, th- this isn't particularly deep, but it, you know, it's one of those things that I'm sure everyone here is thinking about and will continue to talk about, um, which is for us- A large part of our customer base are, like,
- 24:54
large enterprises in financial services and also healthcare. Um, and so, like, very, very security conscious. And so we definitely had to make sure that this was, like, a very opt-in type of feature.
- 25:06
But like, you know, we still wanna have little, uh, like tooltips, like, "Hey, if you click this... But also if you click this, we will send a couple of bits of data, you know, to a third-party provider."
- 25:16
So.
- 25:16
Yeah.
- 25:17
Yeah.
- 25:17
And post-rollout, just to go to kind of the, the last logical, um, part of the conversation here, how have you guys thought about continuing to kind of measure the outputs?
- 25:27
I mean, Bryan, you're the big evals guy up here, so I'm sure that'll be [laughs] the answer. But, uh, would love to hear more about how you think about that measurement and in terms of both the model itself, but also in terms of, you know, the model in the context of the product, which I think is also something
- 25:42
that people, you know, need to think about.
- 25:44
Yeah. Um, so I recently learned that there's a, a more friendly term than dogfooding, which is drinking your own champagne. And so I'll say I drink a lot of champagne.
- 25:55
Um, I use Magic every day, all through the day. Um, one of the fun things about trying to analyze product performance is that you normally do that via data science.
- 26:08
And so I have this fun thing where I'm using Magic to analyze Magic, and I put a lot of effort into trying to understand where it's succeeding and where it's failing, both through traditional product analytics guided by using the product itself.
- 26:21
And so there's a very Ouroboros feeling, but ultimately good old-fashioned data science.
- 26:27
Love to hear it, and appropriate with where you've come from. [laughs]
- 26:30
Yeah.
- 26:31
What about you guys?
- 26:32
Uh, for us it's, you know, I definitely don't have as much, uh, experience as Bryan on, on that side of it. But for a while, the one thing we were doing when it was pure just, like, prompt input, string output with no typing interface whatsoever, is then using that and then writing tests that again used an LLM
- 26:50
to do comparisons and semantic comparisons and like, right? There's obviously problems with that, but, like, it also kind of works. Um, but so then when we moved in kind of the typing world where, um, like Marvin is for, like type-- guaranteed typed outputs, essentially, it definitely becomes a lot easier to test in that world, which is, you
- 27:08
know, one reason that that's kind of the, the soapbox that I get on when I talk about LLM tooling, like bringing it into the back end is just, like having these typed handshakes because, you know, you can write prompts where you know what the output should be and it should have a certain type, and that's act- that's
- 27:22
a very easy thing to test most of the time.
- 27:24
Yeah. Yeah. Totally. And one of the things I think has been, you know, most fascinating about this wave of software, and Bryan, you alluded to this a little bit earlier with your comments around, you know, being stochastic essentially, is that it's not deterministic, right?
- 27:37
And also, I think that AI-based software doesn't have to be static either. It can be, you know, dynamic in a way that maybe traditional software isn't quite as much, and there's, you know, improvements that come along maybe on the UX side of things.
- 27:50
But also the model, we've heard a lot of people talk about techniques like fine-tuning, techniques like RLHF, RLAIF, all sorts of, you know, approaches to kind of continuing to improve the model itself, uh, in the context of the product over time.
- 28:04
So I'm curious about how you think about measuring that improvement, uh, as you continue to hopefully, you know, collect data and refine your understanding of the end user.
- 28:13
Totally. There was a paper that came out in like June-ish or something that was like kind of splashy. It was from the-- It was from, uh, Mattai from Spark, and it was like, oh, like the models are degrading over time even when they say they're not.
- 28:26
And like what I thought was interesting was for like the people that are doing this stuff in prod, we already knew that. Like my evals failed the first day they switched to the new endpoint.
- 28:36
I didn't even switch the endpoint over, and suddenly my evals were failing. So I think there is a certain amount of like when you're building this stuff, these things in a production environment, you're keeping a very close eye on the performance over time, and you're building evals in this very robust way.
- 28:51
And I've said evals enough time for this whole conversation already, but I think the thing that I keep coming back to is
- 28:59
what do you care about in terms of your performance? Boil your cases down to traditional methods of evaluation. We don't need latent distance distributions and KL divergence between those distributions.
- 29:13
We don't need that. Turns out like blue scores of similarity aren't very good for LLM outputs. This has been known for three, four years now. So take your task, understand what it means in a very clear, you know, human way, boil it down to binary yes or no's, and run your evals.
- 29:32
And to the people that say like, "My task is too complicated. I can't tell if it's right or wrong. I have to use something more latent," I would challenge you to try harder.
- 29:41
Um, the tasks that I'm evaluating are quite nuanced and quite complicated, and it hasn't always been easy for me to come up with binary evaluations, but you keep hunting and you evi- eventually find things.
- 29:53
You talked about type checking, and you talk about like type handshakes, and that's something that like a lot of people in ML have been preaching the gospel of composability for five years now. [laughs]
- 30:03
You know, these are not new ideas. They're just maybe new to some of the people that are thinking about evals today.
- 30:09
Yeah. Well, so moral of the story is try harder, essentially. That's what I take away- [laughs]
- 30:14
Yeah. Yeah. [laughs]
- 30:14
... from that. Uh, Chris, did you have anything to add there?
- 30:17
I think the only thing I'd add is I don't, I don't have much take on actually how someone should do it or what they could consider, but I think, you know, you just described a highly non-deterministic, very dynamic w- experimentation workflow and like, those are the sorts of things that just like our core product is meant for.
- 30:35
And so, um, like experimenting with those, like just knowing the structure of them is maybe the best way to say it, is what fascinates me more than the actual like details of what metrics you might be using.
- 30:46
Yeah. Well, you know, I think the other reason I was really excited to do this panel is because we have kind of maybe two sides of the same coin as it relates to being an AI engineer here, right?
- 30:56
One person coming from more of a traditional ML background, one person com- from more of a traditional engineering background, and both of you building these AI-based products. So I wanted to give you a second if you have any last questions to ask of each other.
- 31:09
Yeah. Um, so you work in this, like, data workflow space. And like, I've thought a lot about composability and, like, data workflows, and I've long been a fan of sort of like workflow centric ML.
- 31:19
And so what I, what I'd love to hear is sort of like when you think about building these agent pipelines, which are starting to get more into the, like, DAGs and the sort of like structured chains of response and, uh, request, what is the, like, one thing that, like, every AI engineer building agents should know from your
- 31:41
sphere that, that, that'll make it easier for them to build agents?
- 31:46
So... Oh, that's a really good question. I don't...
- 31:50
I think the main thing is something that I kind of alluded to earlier, which is think about failure modes. I think that is the biggest thing. So like runaway processes, um, capturing potential oddities in outputs or inputs as early as possible with some observability layer.
- 32:06
Um, and so the earlier you can get that wiring in, I think the better. Um, and then caching is like- [laughs] This is the only time I will ever say this.
- 32:17
It's definitely your friend in some of these situations-
- 32:20
Okay
- 32:20
... uh, but is also the root of all evil. So you gotta kinda, you know, balance that. Um, but yeah, I think just thinking about the observability and debugability layer, especially with some of the kind of black boxy and, like, people who are pushing it and actually having like immediate eval of the returned code or something like,
- 32:37
um, having that monitoring layer I think is just key.
- 32:41
Yeah. Chris, I know you've asked Bryan a bunch during this panel, but anything else you wanna...
- 32:44
Yeah. I mean, I'm just really curious, you know... I'm sure everybody asked you this, but the hallucination problem. Like, how... You know, o- obviously, us- your users can just confront it directly if it, it looks weird.
- 32:54
They can see that it looks weird or it errors out, but just how do you think about it as the person building that interface for your users?
- 33:00
Yeah. Um, someone recently asked me for like references on hallucination, and I was like, "What are some good references on hallucination?" And I Googled around, and I found that generally the advice that people are giving is to re- is to fix hallucination, r- basically rag harder, just, like, make a better retrieval augmented pipeline.
- 33:18
And when I said that and I looked at myself, I was like, "Honestly, that's like kinda how we solved it." Like, our reduction in hallucination for Magic, which is not an easy problem, was that we had to think a little bit more carefully about retrieval augmented generation.
- 33:33
And in particular, the retrieval is not something that you'll find in any book, um, even the book that I just published. [laughs] Like, even in there, I don't talk about this particular retrieval mechanism, but it's, it took us some additional thinking, but we got there.
- 33:47
Yeah. So again, moral of the story, try harder always.
- 33:50
Yeah. Just think, and just think carefully. [laughs] [laughs]
- 33:51
Yeah. All right, last thing just to wrap up. What is your hot take of the day for the closing out the AI engineer, uh, summit today?
- 33:59
I'd definitely stop building chat interfaces. Um, I think chat is a product. AI is a tool. And so finding ways to, once again, I know that I've said this before, but, like, improve on the machine-to-machine interfaces so that developers can actually benefit and use AI more directly as opposed to building chat everywhere.
- 34:19
Love that. Um, mine is a little bit mean-spirited, and so I apologize in advance. Um, I think, uh,
- 34:27
a lot of the work that's in front of you as you're building out AI capabilities is going to be incredibly boring, and I think you should be prepared for that.
- 34:36
The capability is really exciting. Well, the possibilities are amazing, and it's always been like this in ML. The journey feels very tedious. It's worth it in the end. It's so fun, but there's a lot of data engineering work in front of you, and I think people haven't yet appreciated how important that is.
- 34:54
Yeah. No, I think it's, it's very real and very fair take as all of us try to start hopefully moving into production with a bunch of this stuff. That's where the rubber meets the road, right?
- 35:03
Well, that's all for us, I think. Thank you so much, uh, the two of you, for coming up here with me. [upbeat music]