# Scaling AI in Education: A Khanmigo case study

Shawn Jansepar, Khan Academy

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/3E7VAZaTG9M-scaling-ai-in-education-khanmigo-case-study) · [Video](https://www.youtube.com/watch?v=3E7VAZaTG9M)

## Scaling AI in Education: A Khanmigo case study

Khan Academy’s classroom AI combines guided feedback, teacher visibility and shared infrastructure, while exposing the hard problems of tutoring accuracy, evaluation and affordable capacity.

Basic familiarity with language models and software testing is helpful; no education-technology background is required.

### Reaching the learners who need a tutor

[0:17](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=17s)

How can one-on-one tutoring reach students whose schools cannot afford to provide it? At Khan Academy, that question sits inside a larger mission: free, world-class education for anyone, anywhere. Shawn Jansepar, who introduces himself as leading AI and learning while also serving as a product leader, describes the scale already behind that mission. He reports 140 million registered users, seven billion learning minutes in 2021–2022, and 1.5 million very active learners using Khan Academy for two hours per month. At the time of the talk, he reports about 200,000 paid Khanmigo learners and teachers, approximately half through school districts and half paying independently.

Access matters as much as aggregate reach. Khan Academy prioritizes historically under-resourced learners through district partnerships, including schools with higher participation in free and reduced-price lunch programs. The opening chart illustrates the unfinished learning problem: Jansepar describes students as largely grade-level proficient in third grade, but an average of 1.1 grade levels behind by eighth grade. The proposed response is to make individual tutoring available at a scale that conventional staffing cannot support.

![Chart of average math grade-level growth from grades 2 through 8, with blue learning blocks, orange proficiency markers and annotations describing the widening gap.](https://ai.engineer/api/talk-essay-media/16126da6813afd8a57b05cd51ae7893bfa61bb1b059fdce07407bede14c3f031)

The unfinished learning gap widens across grade levels.

**Khanmigo is a tutor for learners and an assistant for teachers.** That role extends an existing direction rather than replacing it. Self-paced videos and practice already let learners get explanations on their own schedule; Jansepar recalls Sal Khan’s story that his cousins preferred the automated version of their cousin. LLMs add another way to move toward individualized tutoring. Borrowing a framing he attributes to Notion’s CEO, Jansepar treats the opportunity as an interface revolution as well as a technology revolution: the model becomes useful through an educational experience built around it.

### Catch the mistake without giving away the answer

[3:01](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=181s)

A classroom tutor needs more than a correct final answer. Khanmigo’s design combines Socratic questions, mathematical accuracy, awareness of the learner’s work, integration with course content, multilingual support and text-to-speech. The side-by-side demonstration makes the first two requirements concrete. The shown ChatGPT response moves toward answering the question, which can be useful for an older learner who already knows how to direct their learning. For classroom students, Khanmigo instead asks questions that require the learner to work toward the answer.

The more consequential difference appears when the student distributes negative four incorrectly. In the displayed exchange, ChatGPT praises the distribution as correct. Khanmigo identifies a distribution mistake, backs up and asks a guiding question. This single example exposes two separate failure modes: supplying an answer too readily, and reinforcing an incorrect intermediate step. **Accurate tutoring requires judging the student’s work, not merely solving the problem independently.**

![Side-by-side ChatGPT and Khan Academy conversations, with red arrows highlighting incorrect validation on the left and a distribution-error correction on the right.](https://ai.engineer/api/talk-essay-media/937305b3a97114b339c1d707e276cc801fb0b678df0ddcf75c03ac3fc568c02a)

ChatGPT accepts a distribution mistake; Khanmigo flags it and asks a guiding question.

Embedding the tutor inside an exercise gives it information a standalone chat window would otherwise need the student to supply. After an incorrect submission, the learner can ask for tutoring, and Khanmigo receives:

- **The question:** what the student was trying to solve.
- **The solution:** the step-by-step path through the problem.
- **The student’s entry:** the answer that needs diagnosis.

With that context, the tutor can analyze how the learner may have reached the wrong answer and offer a relevant hint. The timing is deliberate: this diagnostic intervention follows an incorrect submission; Khanmigo would not immediately dive into it before the student submits.

The interface also has to support the subject being taught. A math input widget makes complex equations practical to enter, and specialized graph rendering makes mathematical relationships easier to discuss. Essay writing needs a different interaction again. These are educational controls around the model, not details that a generic chat box automatically supplies.

### Move feedback inside the writing process

[5:06](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=306s)

The familiar essay workflow puts much of the feedback at the end: a student submits, a teacher marks the paper, and the student moves on without revisiting it. AI assistance can shorten that loop by making feedback available while revision is still possible. Jansepar imagines a future in which students might need to reach at least B-level quality before submitting because they have already iterated with an AI. That is a proposed future standard, not a requirement of the writing-coach preview.

The preview divides writing into stages, each with a different kind of help:

1. **Outline.** The student develops a thesis with feedback from the coach. The workflow does not let the student advance until the thesis is sufficiently strong.
2. **Draft.** The student writes the body of the essay and can ask for tips. If asked to write the next paragraph, the coach refuses.
3. **Revise.** The coach provides feedback on evidence, structure, style and tone, and the conclusion.

The boundary is consistent across the stages: the system helps the student improve the work while leaving the act of writing with the student.

Teacher visibility closes the loop. Teachers can inspect the conversation history to understand how an essay developed with AI assistance. Replay functionality is being built, and the system can flag a chunk of text that suddenly appears rather than emerging through the visible writing process. Jansepar illustrates this with a passage of roughly fifty words appearing at once. That gives the teacher process evidence to investigate, rather than just a polished final document. He also describes plans to bring similar support to Khan Academy’s coding platform.

### Give teachers a view into the learning

[7:02](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=422s)

Engagement features let students use energy points to customize Khanmigo, but classroom integration addresses a deeper access problem. A learner who finds Khan Academy through Google already has enough motivation to seek help. Reaching students who dislike learning or believe they are bad at math requires meeting them in the classroom, with a tight feedback loop between their activity and the teacher.

That connection turns concerns about AI into concrete product requirements. Inappropriate conversations trigger teacher notifications. A classroom snapshot uses Khan Academy activity data to summarize how a class is doing, then lets the teacher explore skills reports and graphs. From that overview, a teacher can ask for a progress report to Reese’s family using the learning data already available in the platform. The assistant becomes useful because it can move from a broad class question to a specific student communication without asking the teacher to reconstruct the underlying record.

Other teacher tools include lesson planning and exit tickets. Through [Microsoft’s sponsorship](https://www.microsoft.com/en-us/education/blog/2024/05/enhancing-the-future-of-education-with-khan-academy/), Jansepar says these tools are available free to US teachers, with expansion beyond the US still forthcoming in the talk. This offer concerns teacher tools, not free student tutoring.

The embedded classroom video shows how those product choices feel to learners. Miss Gregory’s class uses Khanmigo for stories and history, including learning about George Washington. A student explains that it helps them reach an answer rather than simply providing one. The clip then visits Ms. Barrera’s fifth-grade and Mr. Rodriguez’s sixth-grade classes, emphasizing personalized learning and adaptive feedback. One student compares the experience to a little person sitting beside them, or a teacher helping from behind. These are qualitative accounts of engagement and assistance, rather than measured learning gains.

### Find the product through working demos

[10:19](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=619s)

Khanmigo did not emerge from a complete specification of the perfect tutor. Jansepar briefly describes requirements, estimates and a Gantt chart, then retracts the story as a joke. The actual process centered on rapid prototyping and iteration. He recommends Ken Kocienda’s [Creative Selection](https://us.macmillan.com/books/9781250194473/creativeselection/) for its account of Apple’s demo-driven development: invest in promising demos and stop the ones that do not work. At Khan Academy, the productive pairing was strong builders with domain experts who understood the customer, ideally working together inside the team rather than handing experimentation to a separate R&D group.

The OpenAI relationship began, in Jansepar’s account, with a request for AP Biology questions to test a new model. An initial September preview produced little headway until December, when the teams shifted to joint prototyping and red teaming and narrowed their use cases. Once tutoring felt promising inside Khan Academy, a company-wide hackathon generated additional features. The strongest ideas were selected for the March launch alongside **GPT-4**; Khan Academy’s [launch retrospective](https://blog.khanacademy.org/khanmigo-top-ai-in-education-moments-of-2023-the-year-artificial-intelligence-dominated-education-news-kl/) dates that limited pilot to March 2023. Jansepar also reports that content-team members worked with OpenAI on reinforcement learning using 100 math-tutoring examples.

### Make iteration part of how the organization works

[12:18](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=738s)

Moving quickly required explicit decision rules, not just enthusiasm for prototypes. Khan Academy resisted perfectionism, made reversible two-way-door decisions quickly, and spent more time on consequential one-way-door questions such as trust and safety. Learning scientists and educators supplied judgment where testing every decision would have slowed progress. Demo-driven iteration ran against fixed deadlines to counter Parkinson’s law—the tendency for work to fill the available time.

The organization also expanded architecture decision records into **organization decision records**. For a strategic shift or major process change, the record identifies the problem, driver, approver and contributors, then circulates for asynchronous reading. Changes to the core values made the desired behavior durable: taking a stand now included trusting intuition, and delivering wow included resisting perfectionism. Those expectations could then shape hiring and performance reviews, rather than remaining advice given during a single project.

![Slide lists resisting perfectionism, trusting intuition, rapid iteration and timeboxed deadlines beside ODR #15 on changing how the organization works.](https://ai.engineer/api/talk-essay-media/7f0a8f0bce147e255cbd9a69c8a06e92dcc7bfda75087184ead2090073158a2a)

Decision principles alongside an organization decision record.

The development process changed with those values. The previous process looked agile, with loops and arrows suggesting iteration, but required substantial documentation for every feature. That rigor could make sense for something headed to all users; it was expensive for an uncertain prototype. The replacement linked engineering rigor to confidence and release scope:

| Feature confidence | Development approach | Release expectations |
| --- | --- | --- |
| Low | Scrappy prototype | Learn whether the idea works |
| Intermediate | A/B test or limited beta | Test with a subset of users |
| High | Broad release | Higher code quality, checklists and accessibility |

Teams could choose how to work within this framework. A common set of release expectations replaced a single heavyweight process for every stage of discovery.

### Put common AI capabilities behind a shared platform

[14:56](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=896s)

An AI platform team sits between infrastructure and product, owning the developer and prompt-engineer experience. One concrete mechanism is a Go component interface: when developers conform to it, tracing and [Langfuse](https://langfuse.com/users/khan-academy) become available through the shared infrastructure. The interface provides a common integration point instead of requiring each product team to build its own observability path.

The same team owns the AI router, hiding routing concerns from feature teams, and an extensible chat component with APIs for different product experiences. It also consults with teams that need AI expertise. A tentative Khanmigo-as-a-service effort extends the reuse boundary outside Khan Academy, exploring how third parties could embed the tutor instead of rebuilding one themselves.

### Separate mathematical diagnosis from tutoring

[15:54](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=954s)

Improving accuracy starts with giving the tutor somewhere to delegate mathematical work. Khanmigo can defer to a math agent that uses a basic calculator or Python for more advanced problems. The model does not have to handle every calculation through text generation alone.

The team also introduced a staged prompting process, which Jansepar calls chain-of-thought prompting. Rather than asking a model to help the student immediately, the workflow first asks how the student might have arrived at the submitted answer and what the correct solution path should have been. A separate model call receives that diagnostic package and decides how to tutor the student. Jansepar reports a substantial improvement from this separation, without giving a numerical result.

The handoff can be expressed in Go as two components with an explicit intermediate result. This illustrative interface keeps the student’s work attached to both calls and separates diagnosis from the learner-facing reply:

```go
package tutoring

import "context"

type Attempt struct {
    Question      string
    StudentAnswer string
    SolutionSteps []string
}

type Diagnosis struct {
    LikelyStudentPath string
    CorrectPath       string
}

type Diagnoser interface {
    Diagnose(context.Context, Attempt) (Diagnosis, error)
}

type Tutor interface {
    Respond(context.Context, Attempt, Diagnosis) (string, error)
}

func Guide(
    ctx context.Context,
    attempt Attempt,
    diagnoser Diagnoser,
    tutor Tutor,
) (string, error) {
    diagnosis, err := diagnoser.Diagnose(ctx, attempt)
    if err != nil {
        return "", err
    }
    return tutor.Respond(ctx, attempt, diagnosis)
}
```

The important boundary is between analyzing the mistake and choosing the next teaching move. The diagnostic result informs the tutor; it is not automatically the response shown to the student.

Additional context includes step-by-step solutions and material supplied through retrieval-augmented generation, or **RAG**. A tutoring-accuracy dashboard monitors quality as prompts change, models are upgraded and other system changes ship. Its purpose is both to catch math regressions and to see whether accuracy is improving over time.

The next evaluation target is accurate tutoring itself. At the time of the talk, Khan Academy is preparing a benchmark dataset and paper intended to test whether a tutor correctly evaluates student work, rather than only whether a model can solve math or reason. Jansepar also describes exploration of Wolfram integration for harder mathematics and fine-tuning. These are next steps in the talk, distinct from the calculator, Python, staged prompting and monitoring already described.

### Turn prompt experiments into testable components

[17:55](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1075s)

The initial prompt code grew through fast experimentation and became tangled. Test coverage was uneven: math tutoring had substantial coverage, while activities such as talking to a historical figure could have no test cases at all. To evaluate a model change for those activities, someone who knew the experience would try it manually. Debugging failures often meant searching logs for the relevant completion calls.

Refactoring into components with defined inputs and outputs made smaller pieces easier to identify and test. The team began building a minimum viable test suite across all activities, alongside tools that let humans run manual cases quickly. It was also moving toward model-graded evaluations where appropriate, while retaining human review for scenarios that still needed it. Langfuse made execution traces available for inspecting a failure and following what happened through the system.

![“Where we are now” slide lists clear component inputs and outputs, a minimum viable test suite across AI activities and tools for manual tests, beside a trace viewer screenshot.](https://ai.engineer/api/talk-essay-media/b27d9bf30cbf1fe55308bc98a8f859a02784f8e65261c8cb30599abc0ac726bc)

Component architecture, broader test coverage and tools for human-reviewed testing.

Automating the grader remains a separate reliability problem. Compared with the team’s own labels, Jansepar reports that AI graders catch math mistakes roughly 70% of the time. This is an internal estimate of error detection, not Khanmigo’s overall tutoring accuracy; the talk does not specify the evaluator, sample size or exact metric definition. More dependable model grading remains an open goal. Adding languages creates another pressure, because every new language can expand the testing and iteration workload. The team wants broader language support without letting that complexity grow unchecked.

### Make the wait understandable and the service affordable

[20:12](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1212s)

Classroom deployment imposes three simultaneous constraints: the service must be reliable enough for daily use, affordable enough for lower-income districts, and fast enough to avoid frustrating students and teachers. Khanmigo already uses different models for different production workflows and falls back to shared capacity when dedicated capacity reaches its limits.

Better tutoring can also introduce more waiting. When additional model chains slowed the experience, Jansepar required a UI change that showed users that work was happening before the feature could ship. A teammate later reported that their child did not perceive the wait as a problem because Khanmigo was doing math. The anecdote is about making latency intelligible, not reducing elapsed time: a visible explanation helped the student understand the pause. Microsoft’s donation of Azure LLM compute also eased the cost of providing Khanmigo free to teachers.

### Use capacity efficiently enough to expand access

[21:18](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1278s)

The deployment roadmap adds provider redundancy and more efficient models. Jansepar describes planned load balancing between OpenAI and Azure, evaluation of **GPT-4o** and smaller models such as **Phi-3**, and collaboration with Microsoft on math tutoring. He calls one model under development **Phi-3.14**, describing it as small to medium in size and optimized for tutoring mathematics. That name refers to the work in development described in the talk, not an established public release.

Capacity utilization is the other side of affordability. At launch, shared OpenAI capacity could not provide the throughput and speed Khanmigo needed, so the team moved to a dedicated instance. Dedicated capacity supplied headroom, but it had to be provisioned for peak demand and did little work at night. Dynamic scaling up and down is the desired next step: preserve enough capacity for classroom peaks while avoiding idle compute during quieter hours.

The closing objective is to use AI to enhance human potential and human intelligence. For an educational system, the engineering work ultimately serves that purpose: making useful help available to more people so they can do more of the learning themselves.

## Resources

- [Khan Academy tutoring accuracy dataset](https://github.com/Khan/tutoring-accuracy-dataset) — Official repository containing the tutoring-conversation benchmark and its accompanying research paper.
- [Evaluating student work in tutoring conversations](https://blog.khanacademy.org/introducing-a-new-dataset-to-further-the-field-of-ai-research/) — Khan Academy introduces 188 anonymized conversations for evaluating whether AI tutors correctly judge student work.
- [Creative Selection](https://us.macmillan.com/books/9781250194473/creativeselection/) — Ken Kocienda's account of Apple's iterative software design and development process.
- [Microsoft and Khan Academy's education partnership](https://www.microsoft.com/en-us/education/blog/2024/05/enhancing-the-future-of-education-with-khan-academy/) — The May 2024 announcement covers free teacher tools and research into Phi-3-based math tutoring.
- [Khan Academy's Go integration with Langfuse](https://langfuse.com/users/khan-academy) — A later customer story explains Khan Academy's custom Go client and use of traces across product teams.
- [Khanmigo's 2023 launch timeline](https://blog.khanacademy.org/khanmigo-top-ai-in-education-moments-of-2023-the-year-artificial-intelligence-dominated-education-news-kl/) — Khan Academy's retrospective describes early prompt development and the March 2023 pilot launch.

## Slides

### Presenting sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=5s) [Microsoft logo beneath the words presenting sponsor.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f227e03d5413a3f145480f3998b2dc3d19988d231e026e1f23d34ccc7102bb5.webp)

PRESENTING SPONSOR
Microsoft

### Innovation and platinum sponsors

[0:10](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=10s) [AWS is listed as innovation sponsor; MongoDB, Google Cloud and Neo4j are listed as platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/630a2c443ecc7597c52e9d1aeb861e20a19a5346dd1d271a75cb61d0c3f5b66c.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Scaling AI in the classroom: agenda

The talk moves from Khan Academy’s problems to product demos and its transformation into an AI-first organization.

[0:45](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=45s) [Agenda covering Khan Academy's problems, product demos and transformation into an AI-first organization.](https://assets.ai.engineer/sites/aiecode2025/sha256/c8ffc75f00ee513d92f25345a0f41fa58f16dd7aa35a8ffcdf88f903b846c1a3.webp)

Scaling AI in the classroom: A Khanmigo case study

• Problems we’re solving at Khan Academy
• How we’re solving those problems (product demo)
• How we transformed Khan Academy into an AI-first organization

2

Interface text:
AIE
aws

### Khan Academy's mission

Khan Academy, a nonprofit, frames its mission around free education, world class quality, and access for anyone, anywhere.

[0:55](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=55s) [Khan Academy's mission is to provide a free world-class education for anyone, anywhere.](https://assets.ai.engineer/sites/aiecode2025/sha256/94728571d200aee37a9ed4f2569d05c7a765ace4bc6fdee5d14d114d04e39ca9.webp)

OUR MISSION

Provide a free world class education for anyone, anywhere

Interface text:
AIE
Microsoft
smol ai

### Khan Academy and Khanmigo usage

Khan Academy reports 140 million registered users and, for 2021–2022, 7 billion learning minutes. Khanmigo’s limited first-year pilot had 200,000 paid learners and teachers, split evenly between individuals and district-funded classrooms.

[1:03](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=63s) [Four panels report 140 million registered users, 7 billion learning minutes, 1.5 million monthly very active learners and 200 thousand paid Khanmigo learners and teachers.](https://assets.ai.engineer/sites/aiecode2025/sha256/734afaf14f06020938745610955d4aa9ce096efc1fd5682072b4022455b5593e.webp)

140 Million
Registered Users
14.8M new users in 2021-2022 alone

7 Billion
Overall Learning Minutes
in 2021-2022

1.5 Million
Monthly Very Active Learners
using Khan Academy for 2+ hours a month in 190 countries and in 50+ languages

200 Thousand
Paid Khanmigo learners and teachers in year 1 of limited pilot
50% paying independently, 50% using Khanmigo in the classroom paid for by Districts

Interface text:
AIE
aws

### Khan Academy district learners

In SY21–22, Khan Academy district partnerships had higher shares than the national public school average across all three measures: 62% vs 52% for free or reduced price lunch, 20% vs 15% Black, and 36% vs 28% Hispanic/Latino.

[1:28](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=88s) [Bar chart comparing district partnerships with national averages: free or reduced price lunch 62% versus 52%, Black students 20% versus 15%, and Hispanic or Latino students 36% versus 28%.](https://assets.ai.engineer/sites/aiecode2025/sha256/71d51bea83db2734294a5a66a16ae261a95369d08289d33797731783916d84ce.webp)

KHAN ACADEMY DISTRICT LEARNERS
(SY21-22)

Free or Reduced Price Lunch: 62%, 52%
Black: 20%, 15%
Hispanic/Latino: 36%, 28%

Legend: Khan Academy District Partnerships; National Public School Average

Interface text: AIE

### Opportunities of LLMs in education

Khan Academy sees LLMs as a way to accelerate its longstanding goal of emulating tutors, building on self-paced videos and practice.

[2:15](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=135s) [Text describes emulating tutors as a long-standing goal, alongside images of Khan Academy practice and video lessons.](https://assets.ai.engineer/sites/aiecode2025/sha256/deba36018cf57a41e71e75cddafa94f711ed70fdc14502556ebca0f44d7e7540.webp)

Opportunities of LLMs in education

Emulating tutors has always been our north star (thus far we’ve been limited to videos and self-paced practice!)

“My cousins preferred the automated version of their cousin to their cousin!” - Sal Khan in 2011 TED Talk

LLMs are just a tool to help us accelerate that vision.

Fund. theorem of calculus
 dF/dx = d/dx ∫ₐˣ f(t) dt = f(x)

Interface text:
AIE
Simplifying radicals
Scientific notation
Trigonometry challenge
Probability
KHAN ACADEMY
Degrees to radians
Related videos: Radians and degrees
Back to Dashboard
Convert the angle 338° into radians. (Round to the nearest hundredth of a radian.)
To convert from degrees to radians, you multiply by π and then divide by 180°.
338° × π/180°
Answer
5.9 radians
Correct! Next Question...
Need help? (0 left)
Hide acceptable answer formats
an exact decimal, like 0.75
Stuck? Watch a video.
This does not set back your progress.

### AI for tutoring and teaching assistance

Khan Academy frames AI as a tool to accelerate its vision of tutoring and teaching assistance at scale.

[2:40](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=160s) [Section title asks how AI is being used for tutoring and teaching assistance at scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/504ca159b430b6570c5dc626f230ba59c662e2a8141af2ba915871368e8f0065.webp)

How we’re using AI to solve tutoring and teaching assistance at scale

15

Interface text:
AIE

### The interface revolution

Ivan Zhao describes AI as both an interface and a technology revolution. The speaker applies that idea to Khan Academy’s experience building educational technology for millions of learners.

[2:56](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=176s) [A quotation attributed to Notion CEO Ivan Zhao describes AI as both an interface and a technology revolution.](https://assets.ai.engineer/sites/aiecode2025/sha256/2356fcddcf8ce85c2b195cc223dabe637fc0888938817062e9dcbfb26fe43307.webp)

“This is as much an interface revolution as a technology revolution. And we’re good at interfaces.” - Ivan Zhao, Notion CEO on AI

16

Interface text:
AIE
aws

### Socratic Tutoring

ChatGPT accepts an incorrect distribution and solves from it. Khanmigo spots the mistake and asks a question that guides the learner back to the distribution step.

[3:17](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=197s) [Side-by-side chat screenshots compare a worked answer with Khanmigo's question about a distribution error.](https://assets.ai.engineer/sites/aiecode2025/sha256/3fdde3f3430109a371885e297c800f3ab7627b606ceaa2f9e2caba625cb16a02.webp)

Socratic Tutoring

ChatGPT:
I'm working on -4(2-x) = 8. I got to -8-4x=8, but I'm not sure what to do next

Great start! You correctly distributed the −4. Now, let's solve for x step by step.

You have:
−8 − 4x = 8

1. Add 8 to both sides to isolate the term with x:
−8 − 4x + 8 = 8 + 8
−4x = 16

2. Divide both sides by −4 to solve for x:
x = 16/−4
x = −4

So, x = −4.

Khanmigo:
[illegible] Academy exercise, please open the Khanmigo chat window on that page for more accurate and helpful support!

Give me 10 practice problems!

I'm working on -4(2-x) = 8. I got to -8-4x=8, but I'm not sure what to do next

It looks like there might have been a small mistake in the distribution step. Let's revisit that part. When you distribute −4 across 2 − x, what should each term be multiplied by −4?

Interface text:
AIE
ChatGPT 4o
Message ChatGPT
Khan Academy workspace chats aren't used to train our models. ChatGPT can make mistakes.
Khan Academy
Tutor me: Math and science
Khanmigo makes mistakes sometimes. Here's why
Leave feedback
Ask away...
Do NOT share any personal data.
This chat is logged and viewable by your teacher.

[3:29](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=209s) [ChatGPT accepts an incorrect distribution, while Khanmigo identifies the mistake and asks a guiding question.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1b3812d290fcdb9935823c9793c2837406385cc11dd8b33d872718709bb78a2.webp)

Socratic Tutoring

ChatGPT:
I'm working on -4(2-x) = 8. I got to -8-4x=8, but I'm not sure what to do next

Great start! You correctly distributed the −4. Now, let's solve for x step by step.

You have:
−8 − 4x = 8

1. Add 8 to both sides to isolate the term with x:
−8 − 4x + 8 = 8 + 8
−4x = 16

2. Divide both sides by −4 to solve for x:
x = 16/−4
x = −4

So, x = −4.

Khanmigo:
[illegible] Academy exercise, please open the Khanmigo chat window on that page for more accurate and helpful support!

Give me 10 practice problems!

I'm working on -4(2-x) = 8. I got to -8-4x=8, but I'm not sure what to do next

It looks like there might have been a small mistake in the distribution step. Let's revisit that part. When you distribute −4 across 2 − x, what should each term be multiplied by −4?

Interface text:
AIE
ChatGPT 4o
Message ChatGPT
Khan Academy workspace chats aren't used to train our models. ChatGPT can make mistakes.
Khan Academy
Tutor me: Math and science
Khanmigo makes mistakes sometimes. Here's why
Leave feedback
Ask away...
Do NOT share any personal data.
This chat is logged and viewable by your teacher.
Microsoft
smol ai

### Context-Aware Tutoring: Exercise

A systems-of-equations exercise shows x = 2 and y = 2 entered. The speaker says Khanmigo receives the question, its solution and the student's entry to guide tutoring.

[4:02](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=242s) [A Khan Academy system-of-equations exercise with 2 entered for both variables.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb938d9f65e53c53b4e59f368fd6961bb43cb1b7a66dfa065b2476952d16c87d.webp)

Context Aware

Systems of equations with substitution

You might need: Calculator

Solve the system of equations.
−5x + 4y = 3
x = 2y − 15

x = 2
y = 2

Show Calculator

Related content
Systems of equations with substitution: -3x-4y=-2 & y=2x-5

Interface text:
AIE
Courses
Search
Khan Academy
Donate
shawn khan
Google Classroom
Microsoft Teams
2 of 4
Skip
Check

### Context-Aware Tutoring: Feedback

After 2 is entered for both variables, the exercise flags the answer and offers tutoring. The speaker says Khanmigo receives the problem, solution steps, and entered answer to help analyze the mistake.

[4:13](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=253s) [The exercise displays incorrect-answer feedback and offers tutoring.](https://assets.ai.engineer/sites/aiecode2025/sha256/e94b3a8932abb604c03fb2481010824d5ea969236b6b63293e60993e6fea7e9e.webp)

Context Aware

Systems of equations with substitution

You might need: Calculator

Solve the system of equations.
−5x + 4y = 3
x = 2y − 15
x = 2
y = 2

Not quite!
I'm here to help!
Tutor Me

Related content
Systems of equations with substitution: [illegible]

Interface text:
Courses
Search
Khan Academy
Donate
shawn khan
Google Classroom
Microsoft Teams
Show Calculator
2 of 4
Skip
Try again
AIE
Microsoft
smol ai

### Context-Aware Tutoring: Request

After an incorrect (2, 2) submission, the demo sends Khanmigo the exercise context and asks for numbered steps and a guess at the mistake.

[4:18](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=258s) [Khanmigo receives a request to explain the solution and analyze the entered answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b9ddc94b88450e5b628a18329edb511fd09102caa9fbcef4816c0f5e595c951.webp)

Context Aware

Systems of equations with substitution

Solve the system of equations.
−5x + 4y = 3
x = 2y − 15
x = 2
y = 2

Help me solve this.

Break this question down into steps and show them to me in a numbered list.

Show me your best guess as to what I did wrong in the list and reference the step number.

I thought the answer was
2, 2

Interface text:
Courses
Search
Khan Academy
Donate
shawn khan
Khanmigo BETA
Khanmigo makes mistakes sometimes. Here's why
Google Classroom
Microsoft Teams
Help me teach this exercise.
Help me write a lesson plan!
You might need: Calculator
Leave feedback
Show Calculator
Sending ...
Ask away...
Related content
2 of 4
Skip
Check again
Do NOT share any personal data.
This chat is logged and viewable by your teacher.
AIE
Microsoft
smol ai

### AI UX for Learning

Khanmigo places a math input keyboard alongside an exercise and chat. The slide also identifies graph rendering and interfaces for essay writing as useful learning features.

[4:46](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=286s) [Khanmigo math input example beside a list of learning features: math input, graph rendering, and specialized essay-writing interfaces.](https://assets.ai.engineer/sites/aiecode2025/sha256/60adc89f2a3efdbcb7158b5c78b65da85352e24733aa465b348a21968a0673ad.webp)

AI UX for Learning

General chatbots are great, but they often don’t offer affordances needed to help learners, such as:
• Math Input
• Graph rendering
• Specialized UX for specific use-cases like Essay Writing.

Interface text:
Assign
Khanmigo BETA
Khanmigo makes mistakes sometimes. Here's why
Distance of complex numb...
You might need: Calculator
Find the distance d between z₁ = (2 + 11i) and z₂ = (6 + 6i).
Express your answer in [illegible] terms and simplify, if [illegible]
d =
Show Calculator
Related content
Distance & midpoint of complex numbers
I'm here to help you with this exercise. Here [illegible] you can ask me:
[illegible] learning this?
Leave feedback
z₁ = (2 + 11i)
Do NOT share any personal data.
This chat is logged and viewable by your teacher.
Check

## Full transcript

[0:00](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=0s) [upbeat music] All right, thanks so much for coming to my talk. Um, yeah, I'm gonna talk to you today about how we've been scaling AI in the classroom.

[0:17](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=17s) Um, so kind of a case study of Khanmigo. Um, my name's Shawn Jansepar. I'm the director of AI, um, and learning at Khan Academy, and [chuckles] I've also been doubling up as a product leader.

[0:26](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=26s) So thankfully, when we're trying to decide on features and roadmap, I don't have to negotiate with anyone but myself. There are other product managers, but, uh, it makes that process a lot easier.

[0:37](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=37s) So today I'm gonna talk about problems we're solving at Khan Academy, how we're solving those problems, walking through a few demos, uh, how we transformed Khan Academy into an AI-first organization, and some of the technical challenges solved, and what we're focused on next.

[0:53](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=53s) So for those of you who don't know, Khan Academy is a nonprofit with a mission to provide a free world-class education for anyone, anywhere. We have a hundred and forty million registered users, seven billion learning minutes just in twenty twenty-one to twenty twenty-two, one point five million very, uh, ler- very active learners who use Khan Academy for

[1:09](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=69s) two hours a month. And we, we most recently, we have about two hundred thousand paid Khanmigo learners and teachers, half of which are in school districts, uh, who are getting access in the classroom, half of which who are just paying on their own.

[1:22](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=82s) Um, and we're specifically committed to serving historically under-resourced learners in our education system. So we, through our partnerships with districts, tend to try to partner with schools who have a higher percentage of students who are in the free and reduced lunch category.

[1:36](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=96s) So we're really trying to, you know, reach the students who, um, normally don't get access to this type of technology and education.

[1:43](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=103s) And since the pandemic, outcomes have been worse than ever. So quickly, this is just a kinda graphic showing you that over time, in the third grade, students are largely grade-level proficient, but by the time they get to the eighth grade, the average student is one point one grade levels behind.

[1:59](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=119s) So we believe that AI can be used to successfully fill those unfinished learning gaps at s- uh, at scale by democratizing one-on-one tutoring for anyone, anywhere.

[2:09](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=129s) So that's where Khanmigo comes in. It's our tutor for learners, and it's our assistant for teachers.

[2:14](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=134s) And just even taking a step back for a second, emulating tutors has really actually been our North Star the entire time. But thus far, we've been limited by the technology.

[2:23](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=143s) We've been doing, you know, self-paced videos, self-paced practice. Sal famously said in his TED Talk that his cousins preferred the automated version of their cousin to their real cousin.

[2:32](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=152s) And LLMs for us are just kind of a tool that help us accelerate towards that vision. We're really not just trying to use AI for AI's sake.

[2:40](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=160s) So how are we using AI to solve tutoring and teaching assistance at scale? There was a really great quote from Notion's CEO where they mentioned that this is as much of an interface revolution as it is a technology revolution, and they are really good at interfaces.

[2:53](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=173s) We think it's the same for us, but we're really good about building educational technology at scale for millions of learners around the world.

[3:01](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=181s) So I'm not gonna go through this whole list, but just generally things we're doing. Building a Socratic tutor that doesn't give away answers, optimizes for the accuracy of math, is context-aware of your work, and is deeply integrated into our content, multilingual, text-to-speech, et cetera.

[3:14](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=194s) So I'm gonna go into some screenshots. So one example here is just comparing Khanmigo to ChatGPT, and you'll notice in this screenshot that here there's just kind of this diving into, "All right, you asked me a question, I'm gonna give you the answer," which can be great if you're an older learner and you know how to learn.

[3:32](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=212s) But as learners in the classroom, we don't want students just getting access to the answer. We want them to think about it deeply. We want a more of a Socratic method where you're asking leading questions to help guide you to the answer.

[3:43](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=223s) So you'll notice here, Khanmigo didn't away, give away the answer. It's asking leading questions. And then on the math accuracy front, you'll notice that ChatGPT said, "You correctly distributed negative four."

[3:55](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=235s) In fact, in this example, you did not correctly distribute negative four. It was incorrect. But Khanmigo's noticing that. Uh, it notices that there was a small mistake in the distribution, and then it kind of back, it backs up and tries to help you from there.

[4:10](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=250s) So we're also been doing a lot to kind of embed Khanmigo directly into our platform and into our content. So you can immediately just say, "Hey, tutor me, I got this wrong," and then Khanmigo's passed the context in of it has the full understanding of what this question is, what is, what is the step-by-step solution, what is

[4:28](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=268s) the, uh, thing that you entered, and then it'll anal-analyze your solution and help you with it, maybe give you helpful tips as to why you might have landed there.

[4:36](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=276s) Um, so that's kinda after you got the question wrong. Before you get the ques- before you submitted the question, we wouldn't just dive into this.

[4:44](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=284s) Um, things li- you know, general chatbots are great, but there's a lot of, uh, UX affordances that are always, that's very important for education. Things like having a great math input widget for doing complex equations, um, specialized graph rendering, et cetera.

[4:57](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=297s) And then also UX for, uh, use cases like essay writing, which I'm gonna dive into. So we've been thinking about how do we go beyond just that traditional chat interface.

[5:06](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=306s) Um, I'm sure you all have kinda had the experience when you were writing essays in grade school. You wrote that essay, your teacher marked it, there's some red ink on it.

[5:14](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=314s) You never probably reviewed that essay again. You moved on. What I think that education, or what I think LLMs can do in education that is gonna revolutionize the way we think about it is that it's gonna help us shorten feedback loops.

[5:27](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=327s) There's gonna be a situation maybe where in the future you won't be able to submit an essay unless it's at least a B because the teacher will know that you've been iteratively working on that essay with an AI.

[5:38](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=338s) And so I know there's a lot of concerns about cheating, so I'll talk about that in a second. But this is just kinda like an early preview into our writing coach.

[5:44](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=344s) So it kinda breaks down the essay writing process into these different categories. So in this example here, you're outlining an essay with a AI directly, and you might have questions about how to write that thesis, but it won't let you move on until you've written a pretty strong thesis, and it'll give you that feedback along the way.

[6:02](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=362s) And then there's drafting the actual meat of the essay. From there, you can kinda ask helpful, for helpful tips, but if you ask it to write the next paragraph for you, it'll refuse.

[6:12](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=372s) And then there's that last step around revision. So it'll give you feedback on axes of evidence, structure, style and tone, and conclusion. And then this is where things get really interesting with the teacher.

[6:23](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=383s) So the teacher now has full access to the history of what you built with the AI to kind of validate that you were the one who wrote that with the AI, and it wasn't just something that you copied and pasted from ChatGPT.

[6:35](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=395s) So you can kinda get a sense of that by looking at the conversation history. Um, we're building kind of these replay functionalities, and then it'll also say like, "Hey, these, you know, these fifty-four, or they're fifty-six words just show up, showed up out of nowhere."

[6:46](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=406s) So we really think that essay writing isn't actually gonna be hurt by generative AI, but we are really gonna revolutionize the way that works.

[6:56](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=416s) Similarly for code, we also have a coding platform, and we're planning on doing the same for code writing at some point.

[7:02](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=422s) We're also building a lot of fun engagement things. Uh, students love this kind of stuff. You know, you can reclaim energy points to customize Khanmigo. Um, but then there's integration into the classroom.

[7:11](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=431s) So, you know, one of the reasons why Khan Academy started focusing on classrooms instead of just focusing on independent learners who find us on Google is because kids who come to us on Google, it's great, but you have to care enough about learning to come to Khan Academy in the first place.

[7:25](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=445s) We really wanted to reach those kids who thought they were bad at math, who hated learning. And so it's important for Khan Academy to be integrated into the classroom and have that tight feedback loop between what the student is doing and so that the, uh, with the teacher so that the teacher can get insights into that.

[7:40](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=460s) So for instance, we have a moderation feature where if any conversation is kinda going off the rails, is inappropriate, an e- a teacher will be flagged and notified. So we kinda tried to do things when we were launching Khanmigo, uh, to turn some of the fears into actual features.

[7:56](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=476s) Um, this is an example of our classroom snapshot where a teacher can say, "Hey, how's my class doing?" And because it has access to all of that rich data about the usage of Khan- Khan Academy in the classroom, it'll give you that overview, and it'll also, um, give you the ability to kinda dive into skills reports and

[8:12](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=492s) dive deeper into our graphs. And then from there, as a teacher, you might be able to do things like, "Hey, I wanna write a progress report to Reese's family."

[8:20](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=500s) That's gonna save you a huge amount of time because all this data is directly available.

[8:26](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=506s) We built a ton of other teacher tools, things like lesson planning, uh, exit tickets, et cetera. I would've loved to demo them, but there's not enough time. Uh, huge shout-out to Microsoft.

[8:35](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=515s) They recently sponsored us, bringing these teacher tools to all US teachers in the States, and soon around the world for free. Um, so feel free to check it out.

[8:43](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=523s) Anyone can sign up. And oh, actually, I need to plug in the audio.

[8:52](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=532s) Instead of kinda talking about the results in the classroom, I thought I would just do a quick fun video showing how that's going.

[8:58](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=538s) Welcome to class. How much they love to use it. Let's, let's, let's have them show us.

[9:02](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=542s) Yes. Let's do this. Let's go. Hi, Khanmigo.

[9:07](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=547s) Miss Gregory's class was so much fun. Students in this class were opening up their imaginations, writing stories, learning about history, and they learned about America's first president.

[9:17](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=557s) What are you excited about George Washington?

[9:19](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=559s) Another lesson they learned is Khanmigo's here to help, but it's not gonna give you the answer.

[9:24](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=564s) No. It just, it just helps you out a bit. It didn't give you the answer, but it helped me out get the answer. It helped me write my stories, and it's so amazing.

[9:34](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=574s) Now we're in Ms. Barrera's fifth-grade class. Everybody here say hi to Khanmigo.

[9:41](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=581s) Hi, Khanmigo.

[9:41](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=581s) In Ms. Barrera's fifth-grade class and Mr. Rodriguez's sixth-grade class, one thing was for sure: Khanmigo was here to help with personalized learning and adaptive feedback.

[9:53](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=593s) I feel like maybe it's weird. It's like a little person sitting next to me, like telling you how to do it properly. It helps you figure out certain things that you wouldn't understand, but then he is like a teacher right behind you.

[10:07](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=607s) I absolutely love Khan Academy. We're moving into this space. AI is our future, and I can't wait to see how Khan Academy soars in the AI space in education.

[10:19](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=619s) All right. So [clears throat] I wanna dive a little bit into how we built Khanmigo and how did we transform Khan Academy into an AI-first organization. Um, you know, it didn't happen overnight.

[10:31](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=631s) So, you know, first we kinda wrote up some requirements about like, what would the perfect AI tutor and assistant, what would that be? We estimated the work. Then we wrote up this nice Gantt chart of the timeline.

[10:43](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=643s) Just kidding. We didn't do any of that. Uh, [laughs] the keys to our success were really rapid prototyping and iteration. Um, actually, there's a great book that I highly recommend.

[10:53](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=653s) It's called "Creative Selection." Um, it was about how the, uh, iPhone was created, um, really kinda driven through this process of iterative development, um, driving it through demos and, you know, feeding the demos that do really well and, you know, putting an end to the demos that don't re- do really well.

[11:08](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=668s) And I think what we really learned is that magic really comes from when you pair incredible builders with domain experts who have some unique insight. And ideally, really, these should be members of your team who deeply understand the customer you're building for as opposed to maybe just sending things off to an R&D team.

[11:24](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=684s) At least that's been in our experience. Uh, the history of our OpenAI partnership, so it actually all started when OpenAI needed some AP Bio questions to test their new model.

[11:33](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=693s) Uh, we got a first sneak peek in September. And then in December, actually between September and December, we didn't have much headway. It was a lot of like, "Well, what are you doing?

[11:42](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=702s) What are you doing?" And then we said, "You know what? Let's just do some rapid prototyping together and red team." Um, and then we narrowed down on a few use cases.

[11:49](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=709s) And then once we decided that we really liked the way that this tutor felt on our platform, we said, "Hey, let's try to launch something alongside of GPT-4, but let's do a company-wide hackathon to build a lot of new features, see what the team can come up with."

[12:01](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=721s) The best ideas from there won. We tried to ship those with GPT-4 in our launch in March. Um, and then we also participated in reinforcement learning, where a few members of our content team were embedded within, um, with OpenAI to kinda train it on a hundred, uh, math tutoring examples.

[12:18](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=738s) So I would say some keys to our success, one is that we resisted perfectionism, which I would call an issue [chuckles] of Khan Academy of the past. We didn't let perfect be the enemy of the good.

[12:28](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=748s) Uh, we made quick decisions for what we would call two-way door decisions, things you can reverse. But for one-way door decisions, things like trust and safety, we thought deeply about those problems.

[12:37](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=757s) Um, we trusted our intuition. We leveraged expertise if learning scientists, um, and educators, uh, kind of wanted to kind of work through that with us instead of us trying to A/B test every single decision.

[12:49](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=769s) And then we also iterated rapidly. I mentioned that with demo-driven development. And then we also kind of had some fixed timelines to avoid Parkinson's law, where work can fill all available space.

[13:00](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=780s) We also ended up driving forward what we called, at Khan Academy, we have this process called architecture decision records, and then more recently transitioned that to organization decision records.

[13:10](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=790s) So whenever we have, like, a big strategic shift at Khan Academy or a big process change at Khan Academy, we document what is the problem we're trying to solve, who is the driver, who is the approver, who's contributing.

[13:20](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=800s) And then once we've kind of written that out, all out, we share that with the team so that they can digest it at their own time and pace. But what we did was we changed some of our Khan Academy core values.

[13:28](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=808s) So take a stand, we added trusting your intuition, um, we added resisting perfectionism to deliver wow so that that could kind of trickle down in the way we do performance reviews, et cetera, with the team and how we hire.

[13:39](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=819s) And then we also did a big change to our product development process. So I'm sure this is probably a tale that many of you have experienced, but we had an old process that we would consider agile.

[13:49](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=829s) You'll notice that all these circles on the left, you know, they feel great. There's a lot of arrows moving in certain directions. They look like you can go backwards, but it ended up being, being a very heavyweight process, a lot of like heavy documentation you had to write for every single feature, et cetera.

[14:02](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=842s) Which makes sense when you're building something, I think, that you know you're just gonna ship to all users and it has to be high quality. But in our case, we were doing a lot of prototyping, um, and so we kind of created this new process around there's a different level of quality of your code that you should

[14:16](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=856s) write depending on your confidence in the feature. So if you're not very confident and you're not sure, that's just a prototype. Go build it, be scrappy. Uh, if it's a kind of you're somewhat confident, then maybe build it as an A/B test or to beta test it with a few subset of users.

[14:33](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=873s) And then if you feel really confident in it and you're gonna share it with all users, that's when the quality of the code needs to be higher. We have some checklists for that that people have to kind of go through.

[14:41](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=881s) There's a higher level of accessibility required when you get to that stage, et cetera. But then as long as you're following this framework, different teams are empowered to kind of build software in the way that makes sense for them, as opposed to having a one-size-fits-all for every team.

[14:56](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=896s) We also added a AI platform team as kind of a glue between our infrastructure team and product team, so there was obviously just a lot of AI platform team chat, so I think a lot of this is familiar.

[15:04](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=904s) But it's really this team that owns the AI developer and prompt engineer experience of Khanmigo. Um, so for example, we recently built this component interface in Go that when you as a developer conform to that component interface, it automatically think-- means that things like tracing and Langfuse are available as long as you conform.

[15:22](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=922s) Um, they own things like the AI router so that you as a product team don't have to worry about that. It just kind of is trans-transparently taken care of for you.

[15:29](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=929s) Uh, we have a chat component, um, that has to have APIs and be extensible for product teams. And it's also a team that will consult with other product teams t- when they're building features if they need some, some help and expertise, as well as what we're kind of tentatively calling Khanmigo-as-a-service, where we're trying to work with third-party

[15:46](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=946s) companies to embed Khanmigo into their products instead of them having to rebuild a brand-new tutor from scratch.

[15:54](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=954s) So what are some of the technical hurdles, uh, what were some of our technical hurdles that we overcame, and what are the challenges ahead?

[16:02](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=962s) So on the tutor accuracy side, things we've done, introduced a math agent. So we kind of give the AI the autonomy to defer to a math agent to work through problems, you know, get access to a basic calculator or, you know, Python might do some more advanced things if it's a more advanced math problem.

[16:19](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=979s) Um, we're doing something called Chain-of-Thought prompting. So it's not just going directly to the model and saying, you know, "Help the student." We're actually doing things like we have this process that will say, "Write out how a s- how you think the student might have arrived at that answer.

[16:34](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=994s) Write out how you think it should have arrived at that answer," and then it kind of feeds that package over to another call to the AI that suppli-supplies that and says, "Using this context, how would you tutor that student?"

[16:46](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1006s) We noticed that that helped, um, improve the performance dramatically.

[16:51](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1011s) Um, we've been providing it with additional context like steps, step-by-step solutions, additional context via RAG, and we've also been building this tutoring accuracy dashboard monitor, uh, to monitor quality over time.

[17:03](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1023s) So we can kind of get a sense of as we're shipping new changes to our prompts, as we're upgrading models, as we're, you know, making any change, we can make sure that, A, we don't have any regressions to the math quality, 'cause that's a really key part of this experience for learners, and we also can, you know,

[17:19](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1039s) ultimately just make sure we're trending in the right direction.

[17:24](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1044s) In terms of next steps, one thing that we're about to launch very, very soon, um, is a math accuracy benchmark data set. So right now, when you look at data sets that different model providers are grading against, it's often, you know, is it good at doing math?

[17:37](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1057s) Is it good at reasoning? But there's nothing that says, "How do I grade whether or not this is an accurate tutor?" So we're gonna be releasing that with a paper shortly.

[17:45](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1065s) Um, we're also looking into integration, uh, with tools like Wolfram for some of the more complex math, and then fine-tuning as well.

[17:55](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1075s) So prompt engineering. [chuckles] It's been a journey. So where we started was a lot of hacking together prompts really, really quickly. Lots of spaghetti code. I'm sure that's the same at a lot of your organizations.

[18:08](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1088s) Um, we had a kind of varied test coverage depending on the activity. So again, we had a lot of really good test coverage for the math portion of it.

[18:16](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1096s) How good does it do at math tutoring? But we had a lot of other activities that we were just like testing ideas and throwing things at the wall and see if they stuck.

[18:23](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1103s) So things like, you know, talk to a historical figure. We didn't really have any test cases for that. When we wanted to iterate on models, it was really just, you know, somebody on our team who knew it well enough that would just go in and test it.

[18:34](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1114s) And a lot of it was also just looking in logs, uh, for completion calls when we had to debug a particular scenario that was a failure case that we wanted to dive deeper into.

[18:44](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1124s) Where we're at now is we've refactored that code using the component architecture that I mentioned earlier, so you can kind of have clearly defined inputs and outputs, and it makes it a lot easier to test and know, like, what are sub pieces of the code that are worth testing.

[18:57](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1137s) Um, actively developing a kind of minimal viable, uh, test coverage suite across all of our activities, so we have a broad enough set of coverage.

[19:07](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1147s) And we have tools, uh, to run these manual test cases really quickly, but still with humans in the loop. Um, and then we're also trying to move in more towards in the direction of model-graded evals to quickly test new models.

[19:18](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1158s) But of course, some of that's not possible without humans in the loop, so it really depends on the scenario. And then also, I think I mentioned this already, all stack traces are now available in Langfuse.

[19:28](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1168s) If you want to go and see what happened, you can kind of go in, look through that, you know, stack trace and, um, debug from there.

[19:36](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1176s) In terms of next steps, this is not something we necessarily have an answer around, but we- we're looking for continued ways to get more reliable model-graded evals. Um, roughly, we find that when we label things ourselves, compare it to the, you know, what the model will grade if we're doing things like, was there any, you know, math

[19:53](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1193s) mistakes in this, uh, the AI is catching it roughly seventy percent of the time.

[19:59](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1199s) And we also wanna add a lot more languages, but, you know, how do we do that without bloom, uh, ballooning the test and iteration complexity? It's, there's probably not gonna be a way to do that, but we'd like to reduce that complexity.

[20:12](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1212s) And then in terms of, this is the last one, uh, in terms of scale, performance, and cost, we need Khanmigo to be reliable enough for teachers and students to use it daily, cheap enough for lower income districts to afford it, and fast enough such that it doesn't lead to user frustration.

[20:26](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1226s) So things we've done so far is leveraging multiple models in production for different workflows, uh, fall back to shared capacity when we're reaching limits on our dedicated capacity, and continuing to do things like perceived performance improvements.

[20:39](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1239s) Like if you notice in this, uh, bottom screenshot, we added this UI when we actually added a few chains to the experience that made it slower. And I said to the team, "We can't ship this unless we add a perceived perform-performance improvements change so that users know that something's happening here."

[20:54](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1254s) And anecdotally, somebody on our team said, "Hey, after we shipped that, my s- my kid was using it," and I said, "Do you think it feels slow?" And that person's, that person's kid said, "No, I don't think so.

[21:03](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1263s) Khanmigo's doing math." So that was a win.

[21:08](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1268s) And then another one is just we have gotten a sizable donation from Microsoft, um, of Azure LLM Compute to bring Khanmigo, uh, free to all teachers, so that's obviously helping as well.

[21:18](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1278s) In terms of next steps, uh, we're planning on doing things like load balancing between OpenAI and Azure for redundancy. Um, we wanna evaluate more efficient models. We're in the middle of evaluating models like GPT-4o as well as small language models like Phi-3 and actively kind of working with Microsoft on Phi-3.

[21:34](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1294s) They're building what they're calling Phi-3.14, which is a small to medium size, um, language model that's optimized for math tutoring.

[21:44](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1304s) And we also wanna get to a point where we can dynamically scale up and down depending on our needs. Uh, when we launched Khanmigo, we needed more throughput than OpenAI could offer on the shared capacity, and we wanted faster performance, so we ended up moving towards a dedicated instance.

[21:59](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1319s) But you know, that dedicated instance is great. We need to provision it for the peaks, but it's not doing a whole lot at night. So ideally, we're, we're making more efficient use of our compute.

[22:11](https://www.youtube.com/watch?v=3E7VAZaTG9M&t=1331s) That's pretty much it. I just challenge us all to leverage artificial intelligence to help us enhance human potential and human intelligence. Thank you. [upbeat music]