Semantic Blindness: 500,000 Sensors Confused an LLM - Raahul Singh & Vanč Levstik, Phaidra

Talk transcript

1. 0:00

   Welcome everyone. My name is Raahul Singh. I'm a staff AI research engineer at Phaidra.

2. 0:05

   And I'm Vanč Levstik. I'm a senior engineering manager, also at Phaidra.

3. 0:09

   And today, we want to talk about the time when we gave an LLM five hundred thousand sensor names, and it got confused. We call this problem semantic blindness. At Phaidra, we build AI agents for AI factories.

4. 0:22

   This includes agents which allow our customers to talk about their data centers and explain to themselves and understand how the data centers are working and what problems they are facing on a day-to-day basis.

5. 0:34

   User queries can be anything from what chiller is running hot, to analyze the distribution of temperatures across my data halls, to is any of my GPUs facing any problems?

6. 0:47

   Now, from this variety of, uh, queries, you can see that these include talking about specific equipments, is chiller six all right, to talking about groups of equipments, GPUs in data hall one.

7. 1:01

   The industry has not really figured out a common naming pattern yet, and every si-single customer can have their own things. From simple names like, uh, racks with GPUs, with data halls, which give you an exact idea of where different things are, to something that is more difficult to comprehend, like CH3 something, something six.

8. 1:21

   We've seen all kinds of names, uh, in the industry going forward. Now, when you're building a demo system, uh, this works because you're working with at a small scale.

9. 1:30

   Uh, s-a simple LLM can, uh, look at all the names of your equipment and figure out what the, uh, user is talking about. But this problem really becomes intractable as you go to for scale.

10. 1:42

    For example, at one gigawatt scale factories, uh, you will see four hundred thousand plus GPUs, and to support those GPUs, you have power meters, you have chillers, you have other equipments.

11. 1:53

    LLM context windows are finite, and you will very quickly saturate, uh, them, and it just becomes a problem. Like we say, a product is something that works for all scenarios and does not fail silently.

12. 2:03

    A demo just has to work for one.

13. 2:07

    In addition to having the LLM figure out these names, uh, we could also have, uh, embedded them in a RAG database, uh, a vector embedding approach. But the problem is oftentimes the names are so similar that semantic search just fails.

14. 2:21

    Uh, there's very little difference between a vector of, sorry, a string name of twenty characters long, which is differs by, let's say, one character, chiller six instead of chiller seven or, uh, CPU something, uh, versus, well, something else.

15. 2:35

    It's very small, so you get a lot of, uh, problems with getting accurate, uh, recall. Also, LLM suffer from what we call a frequency, uh, penalty. If you keep on outputting very similar names over and over again, or very similar tokens, uh, more accurately over and over again, there are internal penalties in the LLM which just shut

16. 2:55

    off their output. So if the user says gi-- list all the names of, uh, let's say GPUs in aisle seven, and th-there are, let's say, a hundred. Just by listing th-through them, the LLMs, uh, guardrails would see, uh, would think that, well, it's spiraling, and it would just shut the system up.

17. 3:13

    So we can't really have these two approaches. RAG would not work. Just naive LLMs would not work. As we move into production systems, we need something that can scale as the system scales.

18. 3:24

    Now, there are naive solutions which we shall discuss going forward. A naive approach to solve this problem would be just to, well, divide and conquer. Take your different eq, uh, equipments, branch them in different shards and pass them through LLMs.

19. 3:38

    Uh, parallel calls should work, right? Well, that's what we thought. The problem is you get horrible recall and hallucinations. You will see LLMs invent phantom equipment that do not exist and also silently drop things that do exist.

20. 3:53

    Now, this creates a problem for mission, uh, critical systems where the users need to know exactly what is happening with their systems. Any problems there will quickly erode user trust.

21. 4:03

    At the same time, you will miss on specific problems which can cascade into bigger and bigger problems going forward. Anyhow, something that we realize here is that as the size of these, uh, physical infrastructure grows, LLMs cannot, uh, the LLM-based solutions cannot grow with the size of individual components or instances or nodes.

22. 4:25

    They-- we have to find something that grows sublinearly with increasing equipment count, and this is what we figured out. So we should not grow with instances, we should grow with tree depth.

23. 4:36

    Now, what do we mean by tree depth here? We realized that, uh, there's a hierarchical structure in which an AI factory is arranged. You will have data centers, then each data centers will have different data halls.

24. 4:45

    You will have dif-- Each of them will have different aisles, and you will have different rows and racks, and then GPUs. Similarly, a chiller plant will have, uh, rooms where chillers are arranged.

25. 4:55

    Then you will have pumps. There will be a separate cooling tower unit. It's kind of like a tree. The depth of the tree grows very slowly. The width grows extremely fast.

26. 5:05

    In other words, you will have a hierarchy that only adds new equipments very rarely, but it adds a lot of them when it does. You will have a lot of GPUs, but without GPUs, you won't have a lot of things.

27. 5:18

    Now, we realized that this could be used to solve our problems, and th-there are four insights that really come into the picture for this. One is a linearizer. What I mean by the linearizer is the LLM has to figure out where each of these things are arranged and how to map from a vague user query to specific

28. 5:39

    equipment or groups of equipments. Here, uh, a summarized representation of our systems graph can be really useful. For example, uh, if your, uh, a one gigawatt scale, uh, factory can have over a million nodes and each node represents a unique, uh, equipment.

29. 5:57

    But because you want to go from the root to the leaf, all you have to do is describe all the parts, and that's a very f-small finite list. For example, here, uh, you can see that to get to a GPU, you can just-- you just need four different, uh, layers to get to it.

30. 6:13

    To get to a chiller, similarly, uh, four layers. To get to switches, similarly, four layers.

31. 6:19

    With this, a sixty-four GPU, uh, layer, uh, s-sorry, a s-sixty-four GPU, uh, system and a four hundred and sixty thousand GPU system produce roughly the same size of summaries.

32. 6:32

    This consolidated, uh, context gives your LLM, uh, all it needs to know to figure out how the distribute-- h-how a plant is arranged and how different equipment are distributed in that plant.

33. 6:44

    The second insight that we had was LLMs are good for planning but not good for searching. This is, uh, what we realized when we saw very poor recall with our sharded solutions.

34. 6:55

    So instead of making the LLM sift through all the different fuzzy names that we can get from our users, we ask the LLM to f- uh, give us exactly how to look for them.

35. 7:06

    So for example, here the query says, "Give me all the GPUs that are running hot in data hall eleven." The LLM does not need to go through all these names of all our GPUs to figure out which one of those are in data hall eleven and then figure out how they are, uh, running hot.

36. 7:20

    All it needs to do is structured outputs, which tells us that, well, we need to collect, uh, GPUs. The scope under which we need to collect is a subtree, which is just data hall eleven.

37. 7:32

    And the filter that we need to apply to finally figure out what exactly we need is GPUs that are running hot. Now, this can be different things depending on the context, and you can implement different filters.

38. 7:42

    All we need to know is how do we create the set that we want to know, uh, get to. The third thing that we realized is that once you have a structured output to figure out what exactly you need, uh,

39. 7:55

    building the backend for it is relatively simple and straightforward. Now, all we need to do is create different, uh, subsets of, uh, our equipment or pre-indexed trees, as we like to call them, based on, uh, location and based on how they interact with other, uh, equipment around to get an idea of what you need to look at.

40. 8:14

    So for example, from the previous, uh, slide, we saw that we wanted to look at GPUs in data hall eleven. We could just get all the GPUs in data hall eleven in one pre-indexed subtree or other data halls, for example, or other racks, for example.

41. 8:27

    And then to get to the final result, all we need to do is, uh, run a query to find all GPUs that are running hot and take their intersection.

42. 8:35

    Set operations ensure that we have perfect recall and accuracy irrespective of, uh, how we want to filter, uh, the queries and what, uh, you know, fuzziness the user may have in their query.

43. 8:48

    Next slide, please. Yes. And finally, uh, what if you have something that is very, very vague? In which case, uh, you have to get the LLMs to, uh, get to searching.

44. 9:01

    But instead of searching directly with names, we figured out that getting the LLMs to, uh, give us patterns to look for. This could be, uh, patterns in the data, patterns in names, can be much more useful to figure out, uh, what the user is, uh, talking about instead of just passing the entire name list, uh, to the

45. 9:18

    LLM. The LLM never has to see a very large number of tokens. All it needs to do is, uh, see some patterns in the naming convention, figure out what exactly the user is talking about, create a pattern, and then we can execute it, uh, on the backend on ourselves.

46. 9:35

    This makes sure that the LLMs has a constant or relatively constant, uh, cost of operation. Whereas if we had to read through everything, we would have been scaling linearly with, uh, increasing equipment count, which itself grows exponentially as the size of the system increases.

47. 9:54

    Finally, to, uh, process a user query end to end, we go from the user's, uh, query, which could be anything from, uh, one equipment or a group of equipment to a planner LLM, which figures out user's intent and gives us a search pattern.

48. 10:10

    Uh, based on that search plan, we have a deterministic resolver which indexes, does set operations, figures out exactly what we need to do, and creates the final set that, uh, maps to the user's query, and this is what we call the result set.

49. 10:25

    All of this is a two or three-step process instead of, uh, a multi-step agentic loop which can keep on running over and over again. And this keeps our total cost also relatively flat and constant.

50. 10:40

    Cool. Yeah, um, if Rahul's job is to design the architecture, my job is production readiness. So [clears throat] we need to make sure that whatever we designed, the elegant solutions we came up actually hold up under real load, real customer data, and all the messy edge cases you only ever see in production.

51. 10:59

    So before any of this went near a customer, we put it through some extensive tests and evals. We measured the new system head to head against the old ones with same LLM model, same data, and we did three runs per case just to make sure.

52. 11:13

    Our goal was simple, prove that Rahul's solutions are ready for production. And they definitely are. If you look at some stats here, the old approach degraded pretty fast with scale.

53. 11:24

    So we got eighty percent correctness at sixty-four GPUs, and that dropped to about thirty percent when the GPU grew to four hundred-- six-- four hundred and sixty thousand. On the other hand, the-- our new approach has maintained correctness with hundred percent accuracy across all of those, um, tests that we give it.

54. 11:42

    And this is not just synthetic test, uh, this is also our real data. So we, we had sixty-six cases on six real production systems, and they product-- they produced zero failures as well.

55. 11:56

    And not just correctness, it's also dramatically lighter. So when we talk about one gigawatt scale, uh, data center, the old approach burned one hundred and sixteen million tokens for just a single evaluation pass while still having a lot of errors.

56. 12:11

    If you look at the new one, it's three hundred and ninety thousand tokens, which comes to around three hundred less tokens, fewer tokens. But the part that matters the most, the cost is flat.

57. 12:22

    As, as Rahul was talking about before, the, the system grows In size. But we, the cost of the query was 9,000 tokens a query where the system was 64 GPUs or 460,000.

58. 12:35

    So instead of getting the cost grow exponentially, w- when the customers grow their systems, it stays the same.

59. 12:43

    That's the impact. Now, there's something about, uh, what we learned, and the part I want you to take back to your own work. It starts with a lens from Karpathy.

60. 12:53

    Most of you have probably seen his presentation or his framing about this before. He talks about different kinds of software. Software 1.0 is a deterministic code you write. It's predictable, exact, but a lot less flexible.

61. 13:06

    On the other hand, we have Software 3.0, which is basically a behavior you prompt out of an LLM. In our case, that might be, "Show me the GPUs in data hall 11 that are running hot."

62. 13:17

    There, this is very flexible, smart, but it's a bit fuzzy.

63. 13:22

    Now, his observation was that in legacy software, the 3.0 is steadily eating 1.0. More of what used to be deterministic code becomes a prompt.

64. 13:32

    Just hold that picture, because the lesson for new-built AI-native sy- systems runs a little bit the other way.

65. 13:41

    But the real skill is knowing which work is the LLM not best for. And you always want to keep the thing- the L- LLM should be used for the things that it does well.

66. 13:52

    It's great at parsing on a biggest request, judging where to look for data and what to look for, handling phrasing we've never seen from a new user that has a different query, and at the end, also synthesizing and writing the final human-readable answer.

67. 14:08

    But, but everything you, you can data model, you should move into code. This is the key one, especially for large systems. If your data has structure, call it a hierarchy, graph, or a schema, a language model scanning it token by token is definitely the wrong tool.

68. 14:24

    Both retrieval, exact set logic, counting, DDAP across near identical names, which happens a lot in the data center land. Anything that, that must be 100% reproducible, it should be a deterministic code.

69. 14:38

    The simple heuristic that usually works, if you can write down the structure or, or the rules, it's a 1.0 job, and pure LLM is weakest exactly when the system is large and well-structured, which is precisely where we operate and our customers.

70. 14:55

    So in a sense, we ran Karpathy's trend backwards. We started almost pure 3.0. We threw everything in a context window because that is the fastest way to find out what's even worth building.

71. 15:06

    And as we said, it worked pretty well on a simple demo to start. Then, as it met real scale, we moved the parts that can be treated as known engineering problems into 1.0, and we still kept the hard judgment in the LLM.

72. 15:21

    So that's the inversion. Legacy software drifts from 1.0 towards 3.0, and new AI-native software starts at 3.0 and matures towards 1.0 for the use cases that earn it, of course.

73. 15:34

    We are not here to replace model judgment. We wanna feed it with the 1.0 tools as we call it. So every 1.0 function you add is more reliable ground for the LLM to stand on.

74. 15:48

    So to finish up, let the LLM keep the hard decisions. Everything it shouldn't be guessing on, add, add it in as code, and hand that structure back to the model to work with.

75. 16:00

    So demo with Software 3.0, productionize by adding in Software 3.0... 1.0.

76. 16:08

    Thanks for watching, everyone. Uh, any follow-up questions at all, you can reach us by email, Raahul, [REDACTED:email_address], [REDACTED:email_address], or you can drop them in the video comments below.

77. 16:20

    Thank you for watching.

78. 16:22

    Thanks.