Z.ai GLM-4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

Talk transcript

1. 0:01

   Hello, everyone. I'm Zhang Yuxuan from Z.ai, and I'm very happy to here to talk about our latest model series, uh, GLM 4.6 series. And let's jump right in.

2. 0:18

   First, uh, I will introduce the GLM series model. GLM 4.6 is not our first open source model. Since 2022, starting from the very first GLM 130B, we have been quite serious about open source our work.

3. 0:37

   Over the years, we have released a whole family of models such as ChatGLM 6B for language model and the CogVLM for vision understanding, CogView for image generation, uh, the CogVideo for video generation, and very, uh, many more across the different domains.

4. 0:55

   And on this slide, you can see a map of our open source models so far, uh, includes the different color, such as white is for the language model, the serie-- uh, GLM series, and the pink for the multi-modal understanding, such as the CogVLM and now it's called GLM-V.

5. 1:13

   And the green one is for image generation, and the yellow is for video generation.

6. 1:20

   Uh, 2025 is our open source year of... In this year, we added even more models, including the GLM 4, Geo414 dense model, including like 9B and 32B, and the GLM 4.5, GLM 4.6 and MoE series model, which is actually our first MoE models family.

7. 1:44

   So up to now, we have released over 65 model in total, in the closed platform like Hugging Face, ModelScope, and others. We have already passed one hundred million downloads.

8. 1:59

   If you search for the GLM or CogVideo on GitHub, you will find 1,500 community projects built on top of them, and it's much a community-driven ecosystem now.

9. 2:13

   Let's move to, uh, GLM 4.6. Uh, I will introduce it now.

10. 2:20

    So GLM 4.6 is our latest flagship model. On many public benchmark, especially in math and coding, GLM 4.6 shows a clear gain over GLM 4.5. It also outperforms open source model released in the same period, like DeepSeek V3.2, and even beats commercial models such as the Claude Sonnet 4 on several

11. 2:45

    benchmarks. Mm, of course, if we compare to like Claude Code 4.5, there's still a noticeable, uh, noticeable gap. So we are not claiming we've stopped everything, but we're getting closer and closer.

12. 2:58

    So, uh, but what makes us especially happy is here is, um, LM Arena, uh, this benchmark.

13. 3:07

    Uh, it is, uh, which is closer to real user preference and on LM Arena, GLM 4.6 is tied for number one together with G-GPT 5 and the Claude, Claude Sonnet 4.5, and it's the only open source model here.

14. 3:22

    And so I'm very happy, appreciate, and also thanks all developer who try to-- who try our model and voice their voice. So let's move to the CC-Bench.

15. 3:36

    So beside the usual benchmark, we also build our own dataset called the CC-Bench here. We want to test agent-style coding in realistic workflow, not just isolate legal problem. So we built a agent coding test platform based on the Claude Code.

16. 3:54

    And on top of that, we create CC-Bench V1.1. So compared with version one, version one, uh, the new version added 22 hard coding task, and we systematically evaluate Claude Code Sonnet 4, uh, Claude Sonnet 4 and GLM 4.5, like Kimi K2 and the DeepSeek V3.1 Terminus.

17. 4:18

    In total, CC-Bench have 74 tasks. It's covering the front-end development and internal tool development in the data analytics, and also algorithm implementation. So for every model, we record the full agent trajectory, the planning step, the code calls, and code adds, and execution.

18. 4:39

    We fully open source this benchmark, so you can check all the link, uh, below in the Hugging Face. And GLM 4.6 made a clear jump over GLM 4.5, and over perf- uh, over performance is close to Claude Sonnet 4 with about 68.6% win rate, while being significant better than other open source baseline.

19. 5:01

    So, uh, where does the performance come from? A lot, uh, is let's talk about GLM 4.6 training. And in this pre-- uh, we will start from the data when training design.

20. 5:15

    First part is the general pre-training. So we start with about 15 billion, uh, 15 trillion tokens of the general proposal data, includes web page, books, uh, encyclopedia, and then multi, uh, multi, uh, multilingual content, and so on.

21. 5:33

    So this stage is all about building a strong all-rounder base model. The context length here is, uh, 4,000 tokens, and the next step is coding the reasoning continual pre-training.

22. 5:46

    So on top of that base with about seven trillion token of extra code in the reasoning data. So it's part of this co-- uh, part of this con- comes from the high quality open source reports, and another part is math, science, and complex program with full state-by-state reasoning.

23. 6:05

    Then we come to the mid-training. So we move to repo-level codes, uh, include the multiple files, issues, and pull requests, and the difference from the same project. And all of this packed into the one long context.

24. 6:20

    Then the goal is to teach the model the following a closed file and understand the change, and to also understand the pull request patch change, and read the real project structure end to end.

25. 6:35

    So at this stage, we extend the content to thirty-two thousand, and, and model can basically see the key file of a medium-sized repo on the one shot.

26. 6:47

    Then as a synthetic reasoning data, we added about five hundred billion token of synthetic reasoning data. So it covers the math, science, and algorithm with explicit thinking trace. So it means the lay of the groundwork of future agent behavior, like breaking down the task, reflect, uh, reflecting on the mistake, and doing long-chain reasoning.

27. 7:10

    Uh, the next step is the long context and agent data. Uh, finally, we use about one hundred billion token of a long context and agent data here. The sequence length is now pushed further to one hundred eighty, uh, twenty, uh, one hundred and twenty-eight thousand.

28. 7:26

    For GLM-4.6 is two hundred thousand. So the model can handle full documents, the whole data, uh, code base, and very long chats. At the same time, we feed lots of agent trajectory.

29. 7:38

    So include like multi-step tool calls, the search, and the code execution, uh, like extract. So, uh, in this slide, improve the model long content capability and the agent capability.

30. 7:51

    Also, in this slide, we introduce SLIME. Um, it's our reinforcement learning framework based on SGLang in-inference stack. Uh, in practice, uh, we design an in-house RL training framework here, and we also open source it.

31. 8:09

    Uh, we found that the different RL tasks need very different system design.

32. 8:14

    Mm. For short reasoning tasks, like the math or the code completion, so the best setup is, uh, Claude 8, uh, serial architecture. So we train an inference set in the same GPU, so after one batch update weight, so the next batch immediately then perform the latest policy.

33. 8:33

    This squeeze the most of GPU memory and compute. Um, but for agent tasks, and for example, the real software engineering, uh,

34. 8:43

    usually have m-many steps. Like for example, open the browser and hit, uh, backend API and for the external response, um, extra. So if we force every worker to stay in the same pace, the whole system will get dragged down by the slowest few tasks in the GPU speed, uh, either.

35. 9:03

    For in SLIME, we decide a hybrid architecture to support both, uh, sequential and asynchronous model. If you look at the diagram, the blue part is Megatron batch training engine with read/write data buffer and opposite weights.

36. 9:21

    And the green part is high throughout SGLang inference cluster. So with the routing function of dispatch request, and in the middle, the data buffer act like the shared memory systems.

37. 9:32

    So one side connect training and the other side different agent environments. For regular reinforcement learning tasks, we keep training and inf, uh, inference on the same GPU pool using with the synchronous mode, and the dynamic sampling, instant update, and the maximum throughput.

38. 9:49

    Once we switch to complete agent task, we move to a decoupled and asynchronous mode. So the rollout side talks directly to real environments and just keep regenerate trajectory and write them into the buffer.

39. 10:02

    And then the training side consume the data in own space, update the me-- model, and, uh, periodically push new weight back to the rollout worker.

40. 10:14

    So the nice thing is, even if some tasks super slow, they don't block the whole training pipeline. So on top of that, we have done a branch of efficiently optimization.

41. 10:25

    Mm. Like the main chain still run bflo systems that, uh, stability, but after each policy update, we do blockwise FP8 quantization on the latest weights, and send the FP8 version to the rollout worker.

42. 10:42

    So the most expensive part, the data generation and running FP8 with much higher throughput while training these still keep BF, BF16 precision. So in practice, we will get the benefit for accuracy and speed in this framework.

43. 11:00

    Now let's zoom in regional RL, and in this slide with some plots. So the first one is about the two-stage curriculum we use. We don't train on a fixed data set from start to finish.

44. 11:12

    Instead, we use a two-stage difficulty curriculum. In stage one, we use median difficulty problem. In each batch, some answer are right and some are wrong. So the rewards have various, and the gradient are meaningful.

45. 11:26

    When the model gets stronger, we switch to extremely hard problem in stage two. But with five hundred and twelve samples, we can still occasionally get a correct solution. So you can see on the plots, the blue curve is our method.

46. 11:41

    After switching to the hard problem, the curve is keep going up. However, use the, uh, median difficulty the way, uh, is not on the red curve. The next picture is about a single, uh, stage reinforcement learning at sixty-four thousand tokens.

47. 11:58

    Mm. Some previous works suggest multi-stage reasoning, uh, reinforcement learning. For is that, uh, uh, for example, is theme there is 32, then 48, and finally six-- uh, 64. But we found that for model that has already been trained with 64,000 token, uh, SFT, those shorter RL chains actually make it forget its long content

48. 12:23

    ability. So average output less, and the final, uh, 64ks token stage can't fully recover the loss. So the red curve here is our approach, which starts directly with 40, uh, 64,000, uh, token and train in one single stage to the end.

49. 12:44

    It's clearly outperformed in the blue middle, uh, multi-stage curve. Uh,

50. 12:51

    the, the picture below is for the code RL. So on the left bottom plot, we compare two ways of computing the loss for code RL. So the blue one is classic sequence mean loss.

51. 13:04

    Each sequence has one loss value, and the red one is our token-based mean loss, which average over token instead of sequence. The token-based version converge faster and more sta-steadily, and it reduce the tendency to generate very short template answer just to gain the reward.

52. 13:24

    In the right, you can see the data. Uh, we do get a science re, uh, reinforcement learning on GPQA dataset, and the messages are most opposite of more data is better.

53. 13:38

    The red curve, red curve is trained only the small set of expert-verified, but high-quality multiple true expression, and the blue curve use mixed quality data. So this result that a small but clean data set gives much better performance.

54. 13:58

    So for scientific reasoning, data quality really matters more than raw size.

55. 14:06

    After talking about GLM 4.6 language model, we move to the multi model.

56. 14:14

    GLM 4.5 version supports the both image and the video understanding. It is our latest visual understanding model, and go-- and on grounding and the image understanding benchmark, it shows strong performance and clear advantages over other open source model released around the same time.

57. 14:35

    So architecturally, we have the three main parts here.

58. 14:41

    The one is a vision transformer encoder backbone, and then, then it's like with MLP projector, and the final is GLM 4.5 base model as decoder. So we try hard to keep the visual input as original as possible.

59. 14:58

    So the model can see the image native resolution and aspect ratio instead of focusing everything into a fixed square. So this matches a lot of UI screenshots and also long vertical image and the PowerPoint slides.

60. 15:12

    So for the video, we in-- also insert a time index token after each time. Basically tell the model this is the first scene, and this is the second scene, and that help it understand the temporal order and event, which is crucial for action understanding and step-by-step re, uh, producer.

61. 15:32

    Uh, we also use a method, uh, as we researched before Co-- uh, in CogAgent. Now, the GUI agent capability is also supported on GLM 4.5V. So it can, like, uh, it can also help you to control a computer and also, like, a website to control, uh...

62. 15:50

    It can use the mouse or the keyboard touching and to communicate with, uh, uh, browser, also computer or mobile environments.

63. 16:02

    So how to use GLM 4.6 or GLM 4.5V model?

64. 16:07

    The first one is using open source weights. Uh, as we know, this, both these two model is open source, so you can use the SGLang or vLLM or other framework to inference it.

65. 16:20

    Uh, along with the weights, uh, on the release day, we already had SGLang and the vLLM integrate ready, and we also work with very sup-- popular open source frameworks like Llama Factory or MS Swift.

66. 16:33

    So thank you to this community. There you have-- You can choose, uh, any framework you want and to try our model. But

67. 16:42

    GLM, uh, GLM 4.6 model is a large model with, like, more than 305, uh, 355 billion parameter. So if you don't have that H100 or, like, other, uh, GPU, there's an easier way to, uh, use our model.

68. 17:02

    So in this slide, we show the deploy, uh, commands of using SGLang or the vLLM here.

69. 17:10

    The next slide, uh, we can use the GLM on the Z.ai, uh, Z.ai. This is a website, and you can try GLM, uh, directly, and you can use it writing code, you can use it to generate PowerPoint, and then also on.

70. 17:28

    And in this, uh, demo is, uh, using

71. 17:33

    one command to write the Google searching in our [REDACTED:url], uh, demo. So you can just, uh, communicate with it.

72. 17:44

    And also, GLM is famous in coding capability. So it also provides the GLM coding plan, which connects GLM with tools and our plugins like Claude Code or other coding develop tool, uh, develop tools, and to provide a very strong coding assistant experience.

73. 18:06

    We also have a short demo video that, uh, show how to replace, uh, a old model in a Claude Code lifestyle with GLM 4.6 here. And you can, uh, see the...

74. 18:18

    You can watch this on YouTube. Then is the, uh, our community activity.

75. 18:29

    Beyond today talk, we are regularly host events both online and offline. So whenever we release a new model, we usually run the several community session afterwards. Uh, as the first one, there is AMA in the Reddit, and we also have some, uh, we also have some, uh, offline, um, on-site, uh, techno- uh, technology sharing, so you can

76. 18:52

    join us. The final, uh, slide is some important links you may to know, uh, is about website, uh, as I would, as I mentioned before, to try GLM model at Z.ai and also our API on here.

77. 19:11

    Then we also provide GLM 4.6 technical bo-- uh, technical blog, uh, GLM 4.5 tech reports. Uh, you can check it, and if you want to join our community, here is the Discord link.

78. 19:25

    And also the GitHub link is below with the open source model, including the, uh, README to how to u- deploy on the open source method. That's all of today.

79. 19:36

    Thank you very much.