Building a Chess Coach

Talk transcript

1. 0:00

   [upbeat music] Afternoon, everyone.

2. 0:16

   So our next talk will be something a little bit different. We're gonna dive into the world of chess. Quick show of hands. Who has heard of Magnus Carlsen?

3. 0:27

   Okay, fantastic. No introduction needed, but widely considered the best chess player in the world. He also founded a company called Take Take Take. Uh, this is, uh, where myself, Anant, and my colleague Asbjørn, uh, currently work at, and we're gonna talk to you today about how we built our AI chess coach that now you can use and

4. 0:47

   is in production. So first up, quick agenda. We'll quickly discuss a bit more about Take Take Take, what it is we actually built, what it is we actually launched.

5. 0:58

   Uh, Asbjørn will then go into a quick history of chess and AI, a lot of links there. We'll briefly touch on why LLMs are actually bad at chess and how we managed to solve this problem.

6. 1:08

   We're then gonna deep dive into actually understanding our game review and sort of closing the loop with our autonomous agent, and you'll get a demo. And then finally, some latency versus quality trade-offs, as this is a consumer-focused AI application.

7. 1:22

   And then lastly, some learnings. So first up, what is Take Take Take? In its simplest form today, it's currently an iOS and Android application. You can go on and play your friends, and you can post about your games.

8. 1:36

   What's relevant for our particular talk is that after you play a game, you get presented with our game review, and this is powered by our, uh, AI pipeline. So for example, just showing you how it works, in this particular position, it's leading to a checkmate.

9. 1:53

   The last move that white has played has moved the knight from this yellow square over here on F3, captured the pawn on E5. It is a brilliant move, so automatically gets the brilliant sort of notation, and the commentary below is actually generated by our system, and we're using, uh, an LLM and the pipeline we'll get into in

10. 2:11

    a second. But what's quite interesting about it is we're able to give you the nuance of why it is a tactic, what detectors from a positional and tactical sense have fired, what are the threats you're trying to do, and actually explain sort of the why behind the move.

11. 2:27

    So that's the system we're gonna be talking about today.

12. 2:31

    Finally, on the last of the step of our, of our application, we've started revealing, uh, insights about your play, and this could be things like how accurate you played in a particular game phase, maybe your current rating or your current depth in a particular opening.

13. 2:46

    And these insights form the next layer of, uh, analysis that we present to the coach, who then gives them back to you as opportunities for learning and improving. We hope by using this, you'll be able to improve and become better at the game.

14. 3:00

    All right. So first, a brief history of chess and AI, since they've been intertwined for so long, just to give you a little bit of cr- uh, backstory. 1949, Claude Shannon, the OG Claude, wrote the, the paper, Programming a Computer to Play Chess, and here he envisioned that, or he proposed that there, there are two types of,

15. 3:19

    of, uh, chess engines, Type A and Type B. Uh, Type A were these brute force engines that searched through all possible, uh, possible moves and figure out the best move, while Type B were those who we know from, uh, from 2017 and, and onward that can selectively le- uh, pick out the, the best moves.

16. 3:40

    Uh, back then, he assumed that we would need Type B computers, uh, to, to play chess because, uh, computers were so weak back then. You couldn't search through the whole, whole tree of, of moves.

17. 3:51

    But computers quickly became better, and uh, people just started scaling these Type A computers. Uh, they got better and better until they, in 1997, Deep Blue versus Kasparov, uh, the first time, uh, a chess engine beat the best chess player at the time.

18. 4:09

    Uh, so people didn't really bother about these Type P co- uh, B computers for a while, these, uh, intuitive, uh, engines, until DeepMind, shout out to DeepMind, uh, released first AlphaGo, because Go is a much more complex game than chess.

19. 4:26

    So you, you can't solve this with these Type A computers. You would need this intuitive approach, neural network approach that actually selectively, uh, figure out which lines to, to calculate.

20. 4:35

    Uh, but after that, they released AlphaZero, who could play not only Go, but also chess and, and Shogi. Uh, and some y- some years later, uh, LLMs came, and people started playing chess against the LLMs, and quickly turned out that they can't really play chess.

21. 4:52

    Uh, sometimes they, they make some right moves and they, they can, to an extent, uh, play, play a nice opening, but they quickly start to hallucinate.

22. 5:03

    So let's see if we can show that. Yeah. There's a video of-

23. 5:09

    Grok went for the poisoned pawn line with queen B6 early on and lost pretty badly, not necessarily be- because of the opening, but because it does not really know how to play chess.

24. 5:22

    There, that was Magnus Carlsen commentating a LLM chess tournament from our office in, in Oslo. Um, that was a tournament organized by Kaggle, uh, l- when they launched their game arena, which was a benchmark for benchmarking LLMs, uh, when playing different types of games.

25. 5:42

    One of them was chess, and now they've started to, to add, uh, more games. Also added Werewolf recently, where you can watch LLMs try to deceive each other in, in social deduction games, which is-- I can recommend [chuckles] watching But yeah, uh, we see that LLMs, uh, often, uh, hallucinate, uh, because obviously they train on language.

26. 6:03

    They're not-- They can't calculate. Uh, they can't re-- Like, high reasoning models can, to an extent, calculate through the reasoning steps where they can actually play out moves, but they quickly, uh, f- uh, fall apart.

27. 6:16

    Um, but there's nothing inherently wrong about the architecture of the, like, the transformer architectures to play chess. DeepMind has trained, uh, a transformer to, instead of predicting the next token, they predict the evaluation based on a chess position where they've trained it on millions of, uh, chess positions to, uh, Stockfish evaluations pair.

28. 6:37

    And that has actually led the transformer to, to play at a, uh, grandmaster level strength. But these aren't trained on language, so these can't explain chess. So how do we, uh, bridge the gap between these old chess computers that can understand and play really good chess, uh, between the LLMs that can explain chess?

29. 7:00

    So we're gonna go through our pipeline of how our game review, uh, explain chess in our app. When you play a game, the first thing we do is we run Stockfish through the whole game.

30. 7:11

    Stockfish is the leading chess engine now. Um, that's like a classical chess engine that, that, uh, calculates the best move. So it's what Stockfish says is considered to be the solution in the, in the chess position.

31. 7:26

    We then extract a lot of, uh, context in the position 'cause we want to explain not only the best move, we want to explain the threats, the plans, um, the tactics that could, could arise in the position, what you should have played.

32. 7:41

    A lot of these, uh, nuances that is useful when if you wanna learn how to become better at chess. Um, so we have a lot of detectors that tries to figure out all, all of this, the forks, pins, skewers, uh, positional structural themes.

33. 7:58

    Uh, doubled pawns, for example, is a disadvantage, so we need to be aware of all of those, uh, kind of things. And there's also a new novel chess engine called Maia, uh, which is behind a research project by the University of Toronto, uh, where instead of building a chess engine that is trained to play the best, they've

34. 8:20

    trained a chess engine. Uh, it's a neural network that predicts the moves that humans would play in certain positions. So given a rating, for example, an online rating of fifteen hundred, it outputs the probability distribution over all the moves in the position.

35. 8:35

    And by doing this, we, we could actually say that a move is, a move is really-- it's, it's the best move. We know that because of Stockfish, but we also know it's really hard to find that move because the probability of playing it at certain levels are, uh, are so low.

36. 8:50

    And all of this information, we feed that to the, uh, LLM, and that-- the LLMs-- For now, the LLM's job is only to translate the info-information, uh, into English, because we really don't want it to try to figure out too much on its own, because it quickly lead to hallucination.

37. 9:07

    It still does, but we want everything to be grounded in the information that we, uh, give it. And that could result into a comment like this. If you play chess, uh, or know, know about chess, this is a game that I played.

38. 9:19

    My opponent played F5 here, uh, which is a bad move. So by using Stockfish, you could see that you get, like, a bad move, uh, indicator. Um, but that's not so useful to just know it's a bad move.

39. 9:30

    So we are running our detectors to figure out that, okay, F5 is threatening to trap my queen. You can also see it draws a, a line with bishop G5.

40. 9:39

    But you can also say, while it threatens to, to, to trap the queen, I can just capture the pawn in the middle because that's defense this square, so that my queen can get out of the situation.

41. 9:53

    Mm. So that's how we, uh, get to that, uh, situation. Now I'm gonna explain a bit on how we improve our, uh, our game review using agents. We have a-- We have closed the loop from user feedback to the product request, essentially, with humans in the, in the loop.

42. 10:15

    But, uh, what happens when users downloads a commentary in your app, because you can download it if you think it's, it's bad, it posts it to Slack, but it also sends it to a Claude Code channel.

43. 10:27

    Channel is a new feature in the research preview that is essentially an MCP server that can in-inject events into a running Claude Code session, so kind of like OpenClau, if you use that.

44. 10:38

    So you have this continuously running channel, and you can inject events to it. So-- And then, uh, Claude Code starts working or on the, on the commentary. It gets all the information.

45. 10:50

    It runs a commentary triage skill that we created that outlines its process, how it should go about to investigate what's going-- what's wrong in a position. Uh, it has some scripts to actually run the generation, so it can modify, for example, the prompt.

46. 11:07

    It could change some of the detectors, create some new detectors, and then it can generate the, uh, commentary again, given this new information, and verify its own work. And then it will also ask questions back to Slack so that I could be on the bus and I could get a message from Claude, who is working on this

47. 11:23

    problem, where it will ask me if this seems right, and I can guide it. Uh, and if it, if it looks right, I'll just tell it to submit the PR, and I open GitHub on my mobile [chuckles] uh, walk- walks through and, and merge it.

48. 11:36

    Uh, I'm gonna show how this works by, uh... So we have a running Claude Code, uh, Cl- channel here. Here is the chat, the, the Slack channel, where the commentary appears.

49. 11:51

    This is just me having tested a bunch of time. I'm gonna open up the app on my phone, go to, uh, comment,

50. 12:01

    and report it as, as bad

51. 12:06

    Now we see it posts a, uh, a comment. We can see the position, the commentary that was generated is there. Now, I haven't really looked at the commentary, so it could be it's, it's, it's probably, it's probably good.

52. 12:20

    Uh, but we can also see that it injects it to the Claude channel, who invokes the commentary triage skill and starts working. Now, this is now running on high effort, so this could take a while.

53. 12:31

    So I'm thinking we should just go, uh, to the next slide, and then we could get back to it to see if it is- [coughs] ... something is happening.

54. 12:39

    Uh, fantastic. So we'll come back to that, uh, in a few seconds. So as we built this for, you know, end users, we had to really kind of consider this trade-off between latency versus quality.

55. 12:49

    So typically, when you finish a chess game, you wanna get the analysis and the results pretty quick. You wanna cycle through the moves kind of one by one. So we really couldn't show you, like, a coach's thinking screen, you know, indefinitely while reasoning tokens are kind of running in the background as an example.

56. 13:03

    So we, we had to get this done, which felt almost instant. In AI world, that's a few seconds at, at best. So we- we're aiming for sub three seconds to generate our coach, uh, sort of feedback.

57. 13:15

    How do we do this? We use Gemini 3 Flash. Time to first token is typically being about a second. End-to-end latency on average is about three seconds, which kind of meets our criteria.

58. 13:25

    We have experimented with other reasoning, uh, models, and we'll get into that on the, on the next slide. The analysis is not incorrect, so the quality is, is definitely good, but the, the challenge is it's unpredictable as to how long it's gonna take to finish.

59. 13:38

    So we have a new set of features kind of planned for a more, you know, chat with your coach type experience, where we can kind of expect the user to be more patient and, and wait for a response rather than in this sort of phase where it needs to be more instantaneous.

60. 13:52

    The, the last thing about quality is, uh, Asbjørn and I are both are good chess players, so we ultimately kind of have the final say is when we look at a position to actually use how we would calculate and how we would play and compare it to the LLM's response.

61. 14:05

    This allows us to actually evaluate whether it's doing the right thing or not. So if we talk about evals in more, in more detail, like I said, Gemini Flash is kind of our, our benchmark, but we have multiple chess scenarios.

62. 14:17

    Currently, we have 16 different scenarios that we created. These are around themes like tactical patterns, blunders, and sort of limiting hallucination. So as an example, you know, there might be a knight fork on the particular chess position, and we're trying to assert that can the LLM actually understand and mention this when we run it through with our

63. 14:36

    sort of context engine. And how do we do this? We extract scenarios from real games. We use LLM as a judge, very powerful sort of technique to, to test.

64. 14:44

    We then run the model, uh, in OpenRadar. OpenRadar's come in handy because new models are being released, you know, so fast, so frequently. We just wanna be able to quickly swap in and swap out maybe a new version of, say, Gemini.

65. 14:57

    We wanna check, you know, check out the latest GPT-5 model or one of the Claude models. So we'll then compare and, and see sort of the quality, ultimately relying on our own skill to detect whether this is good or, or bad.

66. 15:10

    And as a sort of final point on this, we, we ran our three models, Gemini, Claude, and, and GPT-5, and, you know, typically, uh, Gemini Flash is about 75%.

67. 15:19

    It still doesn't pass all the, the scenarios we've set it, so we're always kind of seeing if a new model will actually exceed some of the, the tricky cases we've set up.

68. 15:27

    Um, Claude on more thinking gets us to about just under 60%, but the latency is much longer. GPT-5 Mini given us a smaller sort of model, lower latency, or the slower latency, but lower accuracy as well.

69. 15:39

    So we kind of continuously run through these to, to update.

70. 15:43

    Last thing on sort of our, our learnings and how this can sort of apply to, to your world sitting in front of us. Number one, you know, really important to separate that sort of data pipeline, uh, from the language generation.

71. 15:53

    Uh, LLMs can do a lot of different things, but if you need, you know, high latency or quick latency, it's a good sort of technique. Really try to close the loop with autonomous agents.

72. 16:01

    Kind of the flow that Asbjørn showed is now very common and very powerful and really allows you to iterate quickly. Uh, always try to build a very, a clear sort of context extraction model.

73. 16:11

    This, uh, unfortunately in the beginning is a very slow, sort of painful process. It's a large, you know, ultimately JSON file. It just keeps sort of starting big, and you start to prune step by step, and you see how quality improves over time.

74. 16:22

    Automated evals really do help, and I, I hope in your domains you also have a set of, you know, SMEs that you can help rely on to evaluate the output.

75. 16:30

    And sometimes that's not necessarily the person actually building it. Could be someone else who is a domain expert. So remember to sort of partner if needed.

76. 16:38

    The last thing, just on the, on a fun sort of note before we go back to the, the output of the, uh, the sort of coding agent, uh, we do have some chess sets on the, the third floor at the entrance you may have seen.

77. 16:49

    We're gonna host, uh, a chess simul, uh, today in the afternoon around 3:45 PM. A chess simul, for those who are unfamiliar, is when, uh, one person, in this case me or Asbjørn, uh, plays multiple people at the same time.

78. 17:01

    So we have four chess sets. We will play four people at the same time. Uh, we'll have a slightly more time for us 'cause we have to walk around to play multiple boards.

79. 17:09

    If you happen to play and you happen to win, you will get one of the wooden chess boards at the end of the event. They're very, uh, nice, high quality chess sets.

80. 17:16

    Uh, if no one wins, we will still determine who the two best players are, and we will still give you, uh, a set.

81. 17:22

    And if everyone wins, we need to buy more boards.

82. 17:24

    More sets.

83. 17:24

    So let's- [laughs]

84. 17:25

    Hopefully not everyone wins. Uh, there's a QR code if you wanna sign up or you just stop by at 3:45, uh, you're welcome to, to do that. And then, uh, yeah.

85. 17:33

    Yeah.

86. 17:34

    Close it off.

87. 17:34

    Let's go back to see if, uh, what has been happening here. Oh, we see it's still thinking. It has, it has actually added a comment. "Looking into this now.

88. 17:42

    Investigating the position. Quick question. What specifically feels wrong about the commentary?" Yeah, that's-- It got me there. It's, "There's nothing wrong." You're, you're absolutely right. [laughs] Nothing wrong.

89. 17:54

    So yeah, it's [laughs] now it's gonna close this off 'cause, yeah, it, it worked well. [laughs]

90. 18:01

    Fantastic. Well, thank you so much and happy to take any questions. [clapping] [upbeat music]