User Signal Die at the Retrieval Boundary

Talk transcript

1. 0:01

   Hey, everyone. I'm Suyim. I'm the CEO and co-founder of Starlight Search, and today my talk is, user signals die at retrieval boundary. So we'll look into, uh, what are agents, essentially why agent fails, what is, uh, the cause of fails in retrieval particularly, and how to make actually signals cross the retrieval boundary, and how to make your

2. 0:25

   agent basically outcome fair. So, uh, let's get started. So what is an agent? An agent is an LLM that has agency to reason, invoke tools, uh, interact with the real world, retrieve, uh, the memory to complete the task.

3. 0:43

   One major loop here is missing is, uh, learning. It should also learn from what worked and what it didn't work. Suppose, uh, if I have to, uh,

4. 0:54

   explain what is agent, I can explain with ReAct agent. So if I have to explain, uh, what agent is, I'll explain it with ReAct agent. So basically, user prompt the agent, uh, uh, execute it in a loop, uh, call tool, retrieval, search, and then pause when the task is complete.

5. 1:14

   This is very basic ReAct architecture. One thing that is missing is how to make agent learn from the outcome.

6. 1:24

   So agent keeps failing at the same task. Gartner reports that eighty-five percent of AI projects fail in production, and so does McKinsey's twenty-twenty-five report. The problem came out to be most of the time is that retrieval is static.

7. 1:40

   Seventy-three percent of, uh, RAG pipeline fails because of retrieval, not generation, and context stuffing. So a recent, uh, uh, post from Ram Sriharsha, the ex-CTO of Pinecone said, uh, "We have been optimizing for the wrong thing.

8. 1:59

   You are paying a lot for your agent's memory. It is probably broken, and we have been optimizing for the wrong things. We made wrong answers appear faster and cheaper that we forgot to make retrieval learn."

9. 2:12

   So why this is-- does, does, does it matter?

10. 2:16

    Again, uh, the third problem is agents are not outcome-informed. So there's a missing layer between evals and action. Your observability has all the traces, uh, all the stack that capture-- eh-uh, observability is the stack that capture every tool call, every LLM completion, every exceptions.

11. 2:37

    Your eval suite judges whether the final output was correct or wrong, basically pass or fail. But these evals are not reflected in agent, uh, context, skills, MD files, or agent action in any ways.

12. 2:57

    Right? So the agent doesn't have any access to why yesterday's runs, uh, runs passed or failed. Eval signal dies in the dashboard. This is a missing layer, a system that consume traces, absorb eval, and convert both into retrieval guidance for future runs.

13. 3:17

    So there's a manual improvement tax. An engineer actually has to sit and see if the eval and observability perform well, rewrite the prompt, redeploy it, upgrade-- either upgrade to expensive model, restructure tool or hardness, or fine-tune the custom models.

14. 3:36

    Why are current memories failing? Why-- Memories was designed to actually address this, but, uh, uh, it's not.

15. 3:47

    So let's see, uh, eh, what we have in-- as a current system, uh, and current memory is that they basically store user preferences, profile, conversational history, or long-lived personal-- personalization.

16. 4:05

    So chat experience is not self-improving learning systems for production. If you see, uh, the already existing approach in the, uh, market, there's, uh, uh, LangChain, uh, there's Memzero, which does extracted fact, preferences.

17. 4:21

    A user's retrieval signal is, uh, embedding similarity. Does it learn from outcome? No. Uh, so we have come up with something called utility score, which is a similarity we-weighted by how useful it is for the agent to execute the task.

18. 4:41

    It has actually the history of past traces and past outcomes.

19. 4:47

    So we came up with Agent AgX, and that is agents with runtime experience. It's a runtime, uh, layer that let production agent improve from experiences without pre-training, fine-tuning, or manual prompt engineering.

20. 5:01

    Ha-- It's a bit different from compile time like DSPy because, uh, you bake in all the lessons in the prompt. Here, it's actually improving, uh, while it is executing the task.

21. 5:14

    So let's, uh, again introduce this, uh, utility score. So you do not retrieve by keyword, you retrieve by semantic similarity to the current task, weighted by whether those memories have historically helped or hurt the execution or the outcome.

22. 5:32

    The eval outcome becomes a first-class signal in the retrieval re-ranking and not just for feedback.

23. 5:41

    One of the key things is it treat memory as reasoning, not as facts, statics fact with no context and no history, but reasoning. Like suppose, uh, if, uh, if there's a-- if there is a customer support bot looking for refund, it will not only say, "Hey, user prefers," uh, uh, uh-

24. 6:06

    Dark theme or user prefers, uh, be-- to be called by a shorter name. It's actually reason about a query, like, uh, if some-someone asks for refund, you should check the settlement, uh, before refunding it so that the, the, the customer doesn't get paid, uh, refund twice.

25. 6:26

    So ReAct based on usefulness. Context is updated based on tasks. So this is a very big thing, uh, because most of the agents fail with context stuffing, and this has been brought up in the past and learned from history and reasoning, right?

26. 6:42

    Talking about benchmarks. So, uh, we have benchmarked, uh, our memory system Reflect with-- on τ-bench, which essentially measure if agents have followed a policy well or not. So we have seen, uh, the, uh, uh, the performance improve from sixty-six to seventy-six percent without baking in, uh, skills.

27. 7:05

    And with skills, uh, Reflect performs at eighty. So, uh, once there are, uh, enough memory, like ten, uh, memories, uh, what we do is we bake in the, uh, reasoning and the understanding into skills so that your o-agent always remains updated.

28. 7:24

    What happens most of the time, we have seen, suppose you have a product SQL agent, and there's a system-- there's a column in system prompt. Even though that column is no useful anymore, it remains as the system prompt.

29. 7:36

    So there's no system right now that can update that, "Hey, there's no column, uh, right now called this, so maybe probably you shouldn't entertain it in the future." And this is possible with skills that, uh, because agents always uses, uh, calls that skill, uh, updated skill all the time.

30. 7:58

    And the similar behavior has been s- uh, seen in GPT-5.4. Uh, we have also benchmarked it on agentic tasks, which essentially test a model's ability to reason, plan, and use tools over extended multi-step workflows rather than measuring a static Q&A.

31. 8:16

    So you can see here, uh, with the, suppose the HumanLast exam, uh, with RAG, you get forty-seven point five. If it is starting from the baseline thirty-five point seven, uh, with the, uh, the memory system, it gets to fifty-eight point two, but with the Reflected Memory System, uh, it gets to sixty-one point three percent.

32. 8:38

    So this k- uh, this kind of trend is shown in another, uh, agentic benchmarks, uh, as well, like, uh, BigCodeBench, LifeLongTV, et cetera. So of course, there are limitations to this, uh, uh, to this approach.

33. 8:56

    First of all, there's a cold start. So in the beginning, it's pure semantic search until, uh, enough reviews have been accumulated. Um, there's a utility l-drift. Maybe sometimes, uh, similar memories could come up.

34. 9:12

    There are a lot of problems that could come at scale with this experience, but we have combat most of them. Uh, there's a u-review quality, so noisy labels can make the utility noisy as well.

35. 9:25

    And there is a hyperparameter called lambda that is associated with credit and re-ranking. So we have bu-built Reflect in such a way that most of these problems and most of these, uh, limits are now reduced, uh, except for cold start, which we cannot do much about it.

36. 9:44

    Uh, let's now get into the demo. So let's check this demo.

37. 9:57

    Um, so it's a-- basically a product SQL demo. Uh, I'll give-- ask it to search some product in a SQL database, and let's see if it is able to find it out.

38. 10:25

    So I gave it, "Find me a gaming mouse."

39. 10:30

    Zero memories retrieved. I couldn't find a gaming mouse. Uh, okay. So maybe let's just go and see what's happening in the dashboard.

40. 10:43

    Okay, it came out that... So coming on another benchmark that with the agent bench, which ess-essentially measures if agent has actually done the right reasoning, planning, and, uh, have followed.

41. 11:09

    Which essentially test a model's ability to reason, plan, and use tools over extended multi-step workflows rather than measuring a static Q&A. So you can see here, uh, with the, suppose the HumanLast exam, uh, with RAG, you get forty-seven point five.

42. 11:28

    If it is starting from the baseline thirty-five point seven, uh, with the, uh, the memory system, it gets to fifty-eight point two, but with the Reflected Memory System, uh, it gets to sixty-one point three percent.

43. 11:41

    So this k- uh, this kind of trend is shown in another, uh, agentic benchmarks, uh, as well, like, uh, BigCodeBench, LifeLongTV, et cetera.

44. 11:56

    Wait, let's check. Uh, it couldn't find-- I couldn't find any gaming mouse in the product catalog. So suppose I want to mark it fail and s-tell it

45. 12:07

    Give any relevant. Because there is, uh, there is a wireless, uh, mouse in the database, and I have submitted the failure. Uh, this was the input, this was the response.

46. 12:25

    I couldn't find any, and this was a trajectory and tool call it took. So you can see the product is empty

47. 12:33

    with this product or with this query. Let's, uh, let's check again what happens now.

48. 12:48

    Let's see now, now what happens. ID, uh...

49. 13:05

    So it searched a wireless mouse. Uh, let's see what happens in the dashboard now.

50. 13:16

    Finally, I gave it mouse was the input. The response was, "I found a product related to mouse," as we asked it to find any kind of relevant. Let-- But the most important thing is how the tool call evolved.

51. 13:30

    You can see that previously, uh, the tool calls, uh, or the trajectory, uh, used to look just with one search, that is search products, uh, it called, uh, and the product was empty.

52. 13:45

    Now, the trajectory has changed in production, and, uh, it found something in the product, and it- it's still calling search product tool call, but it is getting some answer that we fed in as a feedback.

53. 14:02

    So, uh, that's the demo, and the most important thing that is happening over here is it's forming memories, which is retrieved based on the utility score. That is the score which basically keeps improving-- uh, keeps re-ranking itself on the basis of how useful that memory was.

54. 14:26

    So, um... You can see from these traces, past traces which memory. Uh, so this memory scores keep changing, and after a certain while, after suppose a five reviews need, you can take in these findings or these, uh, new

55. 14:50

    updates in a skill without, uh, so without changing any system prompt, you can actually update certain things that agent, uh, uh, draws, uh, a lot from.

56. 15:05

    So, uh, uh, you can update the skill, which is very cool. So, uh, I hope everything was...

57. 15:15

    I hope you get to try, uh, this new, uh, feature, this new, uh, runtime experience that we have built.

58. 15:26

    If you want more details, you can visit our website, and, uh, you can also contact me if at [REDACTED:email_address] or visit me. Thank you.