Task Fidelity Scaling Laws — Kobie Crawford, Snorkel AI

Talk transcript

1. 0:00

   [on-hold music] My name's Kobie Crawford.

2. 0:15

   I'm a developer advocate at Snorkel. Uh, we are the frontier, uh, AI data lab and, uh, what that means is [chuckles] that we produce datasets for, uh, foundation models to hill climb on.

3. 0:28

   Uh, so our research team is, uh, highly integrated with the work that we do in terms of our production work and, uh, we put a lot of emphasis on how we integrate research in that this company's origins actually begin from a Stanford, uh, University, uh, AI research lab and, uh, the, uh, work that they were doing there

4. 0:49

   actually was part of one of the, the, the CEO's Ph-- C-- uh, PhD thesis and then that became a, a, a library that was used open source for a while and then we've grown into, uh, focusing on delivering things, um, uh, with, uh, uh, datasets for our, our customers.

5. 1:07

   Uh, one of the things that's been a consistent through line for Snorkel since, uh, we-- the-- since they got started in, uh, you know, as a company in 2019, is that the, the core thesis has been that the quality of data is critical and, uh, that you-- the data that you're looking at you want to make sure

6. 1:24

   is, is top quality, uh, and in all those cases. So we look at how that applies to the datasets that we provide as well as, as things move into the agentic space, uh, how that applies to, uh, agentic tasks and what we wanted to show in this context is how data quality impacts, uh, things in the context

7. 1:42

   of task quality and, and then the task, the task quality and data quality are [sneezing] largely the same thing.

8. 1:52

   So we're gonna talk about the particular research objective here was looking at again, uh, how task quality affects the training outcomes that you get when you're trying to improve models and, uh, then from there we're going to talk about the techniques and, and the path that we chose to, to, to ve-verify that these behaviors were actually happening

9. 2:10

   for us. Um, so does the task quality actually matter is, uh, you can s- you can tell from where we-- from our thesis about it that we expec-actually expect that of course it does and, uh, the way that we're gonna break down talking about that is just make sure we understand wh-where we're approaching this.

10. 2:27

    We're talking specifically in the context of, uh, agentic, uh, Terminal Bench style tasks. Uh, so we're working with, uh, a flow that is gonna be a containerized environment and then a task, uh, definition within that.

11. 2:39

    Um, we wanna show that when you're looking at how, uh,

12. 2:45

    you look at the tasks themselves and how they're built that, that what, what we do in the agentic context, uh, is still also governed by the same data quality premise.

13. 2:54

    So that applies in terms of talking about task quality as well as data quality and so if your architecture changes, if the harness that you're using changes, these kind of things, all of those things are also obviously impactful things, but underlying all of that is still that data quality is at the center of it.

14. 3:11

    So here what we're doing actually puts some specific kinds of rigor and delev-delivering empirical evidence to validate that this is true. That we want to just not just sort of say we accept this as a, as like a thing that we like to say is true, we actually want to verify it, that, that, that's the case.

15. 3:30

    So in the definition of task quality we're talking about basically four core things. Um, if you've worked with these kinds of environments, the Harbor framework, uh, OpenEnv, uh, when you've built tasks for agentic purposes what we're talking about in the context of evaluation, benchmarking, RL, is that we are creating an environment in which that is gonna run

16. 3:50

    and we have it containerized for reproducibility and isolation and that also is allows it to parallelizing for rollouts of these kind of practical elements of how that works. Uh, inside of that you have, uh, uh, looking at what's in the logic of the task you want to talk about, um, that the task is achievable, that it is

17. 4:07

    non-trivial, uh, that it is functionally correct, that the logic actually plays as expected and then the environment itself is reliable and so that r- that environment reliability is a, is a key as well.

18. 4:20

    Those four criteria, uh, for us as we work on this, as the Snorkel team has built in our research, uh, research harnesses, uh, we've built a setup where we verify all four of those criteria and, and our tests to verify those criteria are the ter- the, the, the, the tests that we use.

19. 4:38

    If a task passes all of those tests then we consider it an accepted task, is accepted and then becomes something we can use for our training and research and purposes and then if it's not accepted then it would be, uh, put in the rejected bucket.

20. 4:51

    And we use those two buckets as a then basis for talking about how we're gonna compare what is a high quality task, the accepted ones, to what is a low quality task.

21. 5:02

    So let's look at those comparisons for just a moment and make sure that when we actually take a look at it and level set, does the acceptance criteria that we use tend to correlate with actual performance behaviors that we think we wanna see then and how that, how they differ.

22. 5:17

    Uh, and so the way that we did that was we used, uh, Sonnet 4.5 so this was obviously some months ago, um, and Codex, uh, which had like, uh, GPT-5.2, 5.1 and sometimes 4.0, uh, included in terms of the tests that were run, but we, um, used those two, uh, for, uh, running these, uh, tests and we

23. 5:36

    compared how those tasks, uh, were completed. Uh, and, and in the completions we found that, uh, our accepted tasks averaged twice as many tool calls, uh, demonstrating more difficulty, more steps needed and, uh, and more engagement with, uh, external tools.

24. 5:55

    A lower pass rate so higher difficulty, uh, intrinsically and then, uh, also more output tokens needed so there was more reasoning that was done by the models to actually do that.

25. 6:06

    Um- The failure modes, however. So if we-- let's go back to the pass rate for a second. It's possible in the context of a, a, a lower pass rate that you could have failure modes that actually don't show meaningful signals.

26. 6:19

    So we actually wanted to dig into the failure modes a little bit as well. And so, the next step here is to just say, like, okay, what does it mean to be talking about those, the failures?

27. 6:28

    Uh, to do that, we broke down the failures into categories, uh, to identify the kinds of failures that represent something's meaningful, like this model is not performing, uh, the task completely because the model's un- not, not achieving a logical conclusion that it needs to, uh, versus a failure that's more, say, of a kind of a degenerate case

28. 6:50

    where you have a problem that is an environmental problem, something that, like, literally makes it so, like, no model would be able to solve that problem and not complete that task at that-- in this particular flow.

29. 7:00

    So we have a breakdown of those, and then given those, again, we wanted to compare the accepted versus rejected tasks and see where the failures are occurring in those tasks and see what we get from that.

30. 7:14

    Um, so here's a summary, a breakdown of, of, of each of these things across, uh, the percentage of failures that we saw in each of these different categories. And you can see, I like to highlight in particular, uh, the logic error and the incomplete task, uh, uh, bars and just observe that, uh, you can see that the,

31. 7:33

    uh, in, in each case, you can see a reversal of, like, which one had a higher percentage appear in those there. And then this breakdown, um, even with this, uh, with a, with a comparative, uh, analysis based on percentages of failures, you can see that-- you can see the difference in terms of, like, where you see the,

32. 7:49

    the, the, the un-- overrepresentation or underrepresentation of these kinds of failures across the types of failures that we, uh, categorized and again, between where the rejected tasks versus where the accepted tasks, uh, uh, had errors.

33. 8:04

    Um, the general tendency that we are taking away from this is that the accepted tasks are producing cleaner failures. Uh, these are failures due to the task itself being more difficult, truly more difficult, that, that the steps that it needs to, to accomplish are, are more difficult and that that means that this is a test that would

34. 8:22

    be actually very useful for the model to be able to hill climb on, provide some data samples that could help it actually be able to be improved in terms of those performance beha- patterns versus something where it's just, like, a failure that's not super meaningful in terms of, like, it's just a, a tactical thing that's happening inside

35. 8:38

    of the context that's not working. Um, so

36. 8:44

    with that in mind, we then take it as a, you know, we've accepted that we've actually put together enough analysis that gives us a pretty strong sense that the accepted tasks are also higher quality tasks in the main, and the same thing again that given that we have a differentiation between the higher quality tasks and the lower

37. 9:03

    quality tasks, now we wanna actually see, can we see an impact on model performance when we use it? So following forward from that, we just actually run, uh, a training run, RL training run with the same model,

38. 9:16

    the same compute budget, the same number of tasks in each case, and, uh, and then look at the, the difference there. So that-that's where we've level set that we have a set of tes- tasks that we consider high quality, a level set of tasks that we consider lower quality, and that that usage is gonna help us say

39. 9:32

    something about that. So we trained it twice, and we wanted to see what we got. And the performance uplift is, uh, actually very meaningful. Uh, we're talking about, uh, about a one percent, uh, improvement with using the low quality tasks.

40. 9:51

    So after the RL training was done, the low quality tasks only improved the base model by about a one percent, uh, improvement, but the improvement was about a six percent improvement with the higher quality tasks.

41. 10:04

    So that uplift of, uh, uh, the five x uplift difference, uh, uh, based on, uh, on, on, on just quality is, uh, really striking from our point of view.

42. 10:14

    We think that that, you know, cements the, the, the intuition that, like, it really is important for the data quality to be high. Uh, the way that Snorkel generates, uh, data sets,

43. 10:25

    uh, and, and the way that we put together RL environments, we're using, uh, human expertise all, all, uh, and, and having experts in the loop for generating the data.

44. 10:34

    And we have a strong feeling that the expert in the loop is an important element of delivering data quality. And between those, that ultimately gives you, uh, you know-- Uh, we can talk more about how our platform, uh, how we use our platform to help, uh, make that work by our experts be something that we scale and

45. 10:52

    we can deliver of quality at, at, at scale. But the, but the key is that we have-- we want to put the emphasis on making sure that quality is the first thing that people think about, about what you need your data to, uh, to, what, what you need from your data to make sure that you're getting good

46. 11:06

    results. So that's the, that's the end of what we got out of that. Um, hopefully, uh, if you have, uh, additional questions, uh, we have just a couple of additional minutes.

47. 11:16

    But, um, here's a, a couple of quick links to the research page. This is summarizing how our, summarizing how our research team works and what we do and where we put our emphasis.

48. 11:25

    And then, and then the leaderboard, uh, is pointing to a couple of benchmarks that Snorkel, uh, built and curates, uh, uh, ourselves that are similar to things that you've seen.

49. 11:34

    Uh, we have one called agentic coding, which is focused specifically on these Terminal Bench style tasks. Um, and, uh, we're doing the same kinds of, of, of evaluation, but we want-- we put-- apply a certain kind of rigor to how we're going about it that we wanted to make sure that people could see the difference.

50. 11:48

    Thanks very much, and then please submit a question. [clapping]

51. 11:54

    Can I take any follow-up questions? I'll start here.

52. 11:56

    Uh, I'm just kinda wondering if you would look at, um, another kind of ABC, I guess, of like all the tasks rejected, accepted to see if it's like, i-i-is there-

53. 12:13

    Actually, the model can get over it as long as all the tasks are in there

54. 12:17

    Um-

55. 12:17

    Analysis

56. 12:18

    ... a good question. Um, the interesting thing about, uh, so

57. 12:25

    I don't have a specific quick direct answer about, like, from that analysis that we did in this particular case. Um, but, um, one of the things that we saw in surveying and, and working with, for example, working with the Terminal Bench team, um, and looking at the tasks that were, uh, in Terminal Bench 1 versus what we

58. 12:43

    did for Terminal, Terminal Bench 2, and in some other contexts around, uh, some of the sweep bench, um, some of the variants of the sweep bench, uh, we've been some, doing some analysis internally to, uh, compare the various public benchmarks that are out there and looking at what, what we see across those things.

59. 13:00

    And when we look at that, um, you certainly see that the, uh, the, the, the failure rates and, like, which, you know, where the, where the model, models have been improving over time, um, whether they're getting to saturation faster or the, the benchmarks themselves are getting to saturation faster.

60. 13:16

    Uh, sometimes we ended up seeing sort of some noise about, uh, that because you end up with a certain number of tasks that, that never get completed. So then we start to see that, like, these tasks will never be completed literally just because they actually can't be, right?

61. 13:28

    Yeah. And because, because of that, we ended up, like, finding that that was actually sort of a source of noise in the, in the process of actually evaluating whether the model improvement was actually happening.

62. 13:37

    So in that way, I can't say it's, like, sort of like whether or not the models are actually improving ended up being something that ended up being more sort of masked by, like, the quality of task issue as opposed to being something where we could tell, did the models actually improve or not, uh, despite their, their presence,

63. 13:50

    um, it ended up being some more source of noise. Um, I saw a couple other questions. Right here.

64. 13:55

    I have... Do, do you, do you have any sense of how the input is making this task, uh, you know, there's a correlation sometimes, and input can be a little bit, uh, less prescriptive, could make it harder, or it could be very prescriptive, could make it easier.

65. 14:14

    I don't know, but I'm just wondering if you encounter something like that.

66. 14:18

    Yes, absolutely. In fact, uh, the way that we've been looking at it, uh, a lot of times what makes a task, uh, sort of a, a, or one of the rejected task is it being underspecified in terms of like, you know, when you've, when the task is defined in a way that the desired testable outcome is not,

67. 14:37

    uh, clearly ex-specified in the task definition up front, but then on the back end, the tests themselves expect certain things to pass that were never actually requested. Those kinds of mismatches are so the, sort of some of the places where you can see where the task becomes, uh, or at least appears to be harder because the tests

68. 14:54

    don't match what the, the requested, uh, task set up. Um, also sometimes there are like implicit dependencies in the, in the testing that, uh, that the task doesn't specify, uh, be there and then in the, and then without knowing that the dependency, uh, is required in the first place and what hasn't, and hasn't been fed into the

69. 15:11

    context of the model, then the model doesn't even have the, the right context to be able to, to approach those dependencies.

70. 15:17

    Just a follow-up or-

71. 15:19

    Yeah

72. 15:19

    ... like then maybe cla- classifying them as a fail could be an issue because not every task needs to be complete. You know, there's iteration, there's a journey. Typically, when we solve problems, it's never a one-shot kind of thing in most of the problem in the world.

73. 15:39

    So yeah, uh, differentiating that with the underspecified could maybe go, I, I don't know, but-

74. 15:47

    Certainly. Certainly. Uh, 100% agreed that, that those kinds of things about like what we ultimately want the models to do tends to go like that. In the context of building benchmarking tasks, we work to make it so that we have something that's verifiable on the back end, and that, that, that in principle, if we do it right,

75. 16:03

    that the skills that are being learned are still going to be more applicable in the context of unverifiable results or things where there's going to be an iteration that needs to occur following the, the step that you're working on currently.

76. 16:15

    Yeah. Yeah. So.

77. 16:16

    Thank you.

78. 16:16

    Sure. Sure. Yes.

79. 16:18

    Um, in terms of like future challenges, next steps, like are you working on tasks that are not as straightforward, verifiable, and then maybe more towards the very long horizon?

80. 16:30

    We certainly are looking at all of those, uh, and different projects o- of, of ours are, were in those different spaces, especially once you get outside of the places where verification isn't easy and coding and, and math make it straightforward and then things that are more fuzzy are, are different.

81. 16:46

    We're, uh, we have an open benchmark grants program that we're working with, where we're partnering with folks who are developing, uh, benchmarks and evaluations in, in some more of the less verifiable areas and, uh, there's a lot of very interesting stuff that we're doing that has to do with,

82. 17:02

    uh, one of them that, uh, there's a, a, a, a, an interesting organization, uh, that's working on, um, trying to remember the name of it, but it was about sort of like, uh,

83. 17:13

    really looking at things that, that involve like sort of emotional like level things and stuff like that. It's very, lots of very human-centric, uh, thinking. And so there, um, the, even, even the notion of like, you know, what is correct or not is something where we wanna have like actually sort of multiple possible outcomes and then score

84. 17:28

    them differently but, you know, but have them all sort of fit on the spectrum somewhere. And so there's, there's a lot that we're trying to do in the, in the, the varied things in a lot of different dimensions.

85. 17:37

    Um, but yeah, so it's a very interesting space.

86. 17:38

    No, no, no, no. [laughing]

87. 17:40

    One last question in the back.

88. 17:43

    Yeah. Um, yeah, I was, I was just putting an extension of, um, this conversation.

89. 17:48

    Mm.

90. 17:48

    Like, uh, uh, how does your system, uh, handle this, um, inter-annotator, annotator, um, I guess, agreement or disagreement-

91. 17:57

    Ah

92. 17:58

    ... uh, I guess in the, in the multi-step, uh, task that you were saying?

93. 18:02

    Yeah.

94. 18:03

    I guess right now it's maybe more verifiable like you say because it's, it's like, um, the answer is a bit discretionary, like, uh, like, um, for, like for coding and math.

95. 18:13

    But in that world, like you said, there's many possible-

96. 18:20

    I'll speak to inter-annotator dis- uh, inter-inter-annotator agreement, uh, sort of centrally, um, and then y- the complexity you're, you're, you're talking about 100% is, uh, is something that like the longer the horizon, the multiple steps involved, and the different dimensions, um, are, are an issue.

97. 18:33

    This is the last question, yeah? Or are we out done on time? Um, y- do I still have one minute left? You have one minute. Okay, brilliant. Um, so the way that our platform works, we actually do a number of things to bring together human annotators, as well as using LLM judges.

98. 18:49

    Um, and that's partly to help us, uh, replicate and scale what our human annotators are, are delivering, but also these, these kinds of agreement. Um, we, we feel like the way that we are doing things with rubrics these days, and providing sort of like a longer list of, of, of...

99. 19:06

    And data points and, and, and qu- criteria that need to be met, that as we sort of l- build out a set of rubrics that then can be used both by LLM judges and people, that we're actually sort of like looking at, uh, high level, uh, s- qualitative things, as well as individual sort of like, uh, more

100. 19:25

     quantitative, uh, comparisons. Um, and so through a long, through a longer list of things built on a rubric, and then using the human annotators or the experts to help us give us the, the information, the ground truth information that we can inform LLM judges to, to look at.

101. 19:40

     We're actually looking to make sure, for example, that we actually, uh, test and get inter-annotator, inter-annotator, inter-inter-annotator agreement very high between both individual humans, as well as between the LLM judges and humans, and then use all of those, uh, comparisons to, to, to do quality assessment.

102. 19:57

     So it's part of our assessment process. Uh, and so we... And we use that so for, for each of these kinds of, of, of tests. So in the context here, where it's explicitly verifiable, uh, and tests will pass or tests will fail, um, it's still obviously, uh, an easier domain than others.

103. 20:11

     But we still keep using that sort of guiding principle apla- a- across all these domains.

104. 20:18

     All right. Well, thank you very much. I really appreciate your time. It's great to have you. [audience applauding] [upbeat electronic music]