The Unbearable Lightness of Agent Optimization

Talk transcript

1. 0:03

   Right. Hello, everyone. Uh, today I will present Meta-Adaptive Context Engineering, or Meta-ACE for short, which is a new framework designed to optimize AI agents beyond single dimension approaches. We will explore how orchestrating multiple adaptation strategies can overcome the limitations of existing context engineering methods.

2. 0:25

   Now, a little introduction about myself. Uh, so I'm Alberto Romero. I'm the co-founder and CEO at Jointly. And for context, at Jointly, we build domain-specialized agents for regulated industries where policy adherence constraints are particularly strict.

3. 0:41

   Most of our research work is in the area of self-optimizing agent architectures, uh, using systematic approaches.

4. 0:49

   Now, about myself, I have spent, uh, twenty plus years at the intersection of AI and data. Uh, some of my recent experience includes being the CTO and co-founder of HumanAI, uh, think ML-based risk prediction for mobility, which was acquired by Aoni in twenty twenty-three.

5. 1:07

   And in my previous role, I headed up Citibank's GenAI engineering team.

6. 1:13

   Now, here's our agenda for today. Um, we'll begin with the motivation and problems that current systems face. Then we'll review the ACE framework and its limitations.

7. 1:26

   Um, after surveying recent research, uh, insights, we'll introduce the Meta-ACE approach. We'll discuss its architecture and strategy toolbox, show some results, um, and finish with future directions and challenges.

8. 1:43

   Now, the Agentic Context Engineering framework, or ACE for short, uh, for which you've got the paper link, uh, on the slide there. So it's, it's a very popular framework, um, and the paper, um, came out a few months ago.

9. 1:58

   Um, basically organizes adaptation into three roles. First of all, there's a generator that produces reasoning paths. Then there's a reflector that extracts lessons. And finally, there is a curator that synthesizes these lessons into incremental updates.

10. 2:14

    ACE, uh, uses incremental delta updates and a grow and refine mechanism to prevent context collapse and maintain relevance.

11. 2:23

    Now, most importantly, it can improve without label data by learning directly from execution feedback.

12. 2:32

    Now, so ACE has been, um, quite successful and has achieved substantial gains across some of the most popular HM benchmarks like AppWorld or FiNER, uh, of almost an eleven percent compared to previous state-of-the-art approaches such as GEPA or DC.

13. 2:50

    Um, and it's also achieved an eight point six percent, um, uh, gain on financial reasoning tasks.

14. 2:58

    Um, there are four fundamental limitations, um, for ACE that I'm going to reflect on and, um, just discuss on the next slide. Um, and those form the basis for, um, for Meta-ACE, basically.

15. 3:15

    Now, as I was saying, um, despite its strengths, ACE has got four critical failure modes. First, it is highly dependent on the reflector. Um, so when reflection fails, the context becomes noisy and even harmful.

16. 3:32

    Uh, secondly, there's feedback brittleness, which means that when ground truth signals are weak or absent, ACE may reinforce incorrect behaviors.

17. 3:43

    Third, the, the task complexity blindness, um, which leads to treat simple and complex tasks the same, which can be a waste of resource, uh, and also a means of opportunities, um, for optimization.

18. 3:59

    And then finally, um, ACE optimizes only the context dimension, so ignores compute, memory, and parameter updates.

19. 4:08

    Now, the twenty-four and twenty-five research landscape offers, um, four key insights, in my views. First of all, uh, verification mech-mechanisms, uh, like self-evaluation, multimodal consensus, and execution checks are really important for robustness of any solution.

20. 4:28

    Secondly, uh, adaptive compute allocation shows that small models can outperform much larger ones by selectively increasing inference steps.

21. 4:39

    The third one is that structured memory architectures, uh, outperform linear context, context accumulation by organizing facts as graphs or multi-granular memories.

22. 4:51

    And then finally, test-time training bridges inference and learning, uh, and enables, uh, temporary parameter updates to yield large accuracy gains. So these advances suggest that we need a hybrid multi-dimensional system.

23. 5:08

    Now, Meta-ACE, um, addresses ACE's limitation by adding a meta-controller that learns to orchestrate multiple adaptation strategies based on a task's complexity, uncertainty, verifiability, and also resource constraints.

24. 5:25

    So instead of applying the same procedure to every problem, Meta-ACE profiles each task and allocates the right combination of strategies across context, compute, verification, memory, and parameter dimensions.

25. 5:40

    Um, so this adaptive, uh, learned coordination is what enables it to outperform single dimension methods.

26. 5:50

    Now, the, the Meta-ACE framework consists of four layers, so getting into the architecture. Um, the first layer is the task profiling one, which assesses complexity, uncertainty, verifiability, and resource budgets.

27. 6:06

    Then there is a lightweight meta-controller that selects and allocates adaptation strategies accordingly.

28. 6:13

    The next layer down is a strategy execution one, and that carries out the reflection, adaptive compute, hierarchical verification, structure memory retrieval, and selective, uh, test-time training. And then finally, uh, there's a feedback aggregation layer that collects the outcomes and updates the meta-controller's policy through meta-learning.

29. 6:36

    So this layer design allows the system to learn from its experience and, uh, continuously refine its decision-making.

30. 6:47

    In terms of the task profiling, um, there are four key dimensions that are being assessed. The first one is, uh, semantic co-complexity. So this is basically an embedding-based similarity to, uh, known task distributions that gets produced.

31. 7:05

    Uh, second one is uncertainty quantification. Uh, think of it as a relative softmax, uh, scoring pr- that predicts model confidence.

32. 7:15

    The third one is verifiability assessment, so whether we can execute and validate the output.

33. 7:22

    And then the fourth one is resource availability. So we take into consideration the context window, the compute budget, and even other constraints such as time.

34. 7:33

    So the output of this layer, of the task profiling layer, is a thirty-two dimensional task embedding, which is what feeds, uh, as input into the meta-controller.

35. 7:45

    Now, in terms of the strategy toolbox, um, Meta-ACE draws from six strategies. First one is minimal context, which uses concise prompts for simple tasks.

36. 7:58

    Uh, then we use ACE reflection, uh, which retains the generator-reflector-curator loop for incremental knowledge accumulation, um, as established by, uh, standard ACE.

37. 8:11

    Then we also use adaptive compute, which scales the number of reasoning steps or samples based on the task difficulty.

38. 8:19

    We also use, uh, hierarchical verification that combines self-evaluation and multimodal consensus and execution checks.

39. 8:28

    Uh, adaptive memory, uh, that retrieves relevant information from structured multi-granular memories. And then finally, we use selective test-time training, which applies temporary parameter updates such as lower adapters for high-stakes tasks.

40. 8:44

    So the meta-controller learns to combine these tools effectively over time.

41. 8:51

    Now, the, um, reward formula, um, upon which the, the learning strategy is selected accounts for the following components. Um, the first one is the correctness of an action or prediction, which is accuracy.

42. 9:07

    Then we also have the penalty associated, um, with resources used or negative outcomes, so one minus cost. And then is the trustworthiness of the models, which is self-expressed certainty.

43. 9:21

    So the confidence calibration basically, uh, with weighted importance determined by the hyperparameters alpha, beta, and gamma.

44. 9:31

    In terms of the, um, uh, meta-learning loop, um, we have four sources of feedback collection. Uh, first of all is task outcomes, the success, failure, or correctness, um, of the task.

45. 9:46

    Then we've got the strategy performance. So what is the individual contribution of each strategy to the overall performance of the task?

46. 9:56

    Then we also have efficiency metrics such as the compute, latency, memory. And then finally, we've got confidence calibration. So were predictions accurate?

47. 10:10

    Um, so moving on to, um, how we go on about, uh, solving the, um, uh, the limitations from ACE. The first one was the weak reflector problem. So ACE's issue is that there is, uh, a fifty to sixty percent performance drop when reflector quality degrades.

48. 10:30

    Um, with Meta-ACE, we introduce, um, uh, three things, basically. So first of all is, uh, quality gates. Um, so it's a learned classifier that blocks harmful deltas. And secondly, there's a multi-signal reflector, uh, or reflection, which basically, um, is an ensemble of specialist models, uh, when there is a level of uncertainty.

49. 10:56

    Uh, and then the third one is adaptive strategy allocation. So the meta-controller learns when reflection fails, and then it routes to verification or test-time compute instead.

50. 11:09

    Um, so we, we can expect to maintain an eighty percent plus performance even when the, uh, reflector degrades around, uh, thirty percent.

51. 11:21

    Now, the, the second, um, limitation we had was, um, the feedback quality brittleness. So what we observe with ACE is that there can be significant degradation without reliable ground truth signals.

52. 11:36

    Uh, with Meta-ACE, we introduce a hierarchical verification cascade, um, where we can expect a fifty to sixty percent reduction in errors from poor feedback, and that's through three tiers.

53. 11:49

    The first tier is self-verification, which is just fast filter. We just accept if the confidence level is over a certain value. Second tier is a multimodal consensus, so we leverage a diverse range of models such as GPT-4, Claude, and DeepSeek, and we do confidence-weighted voting.

54. 12:11

    And then the tier three is execution-based verification, uh, where we leverage code sandbox, APA, API validation, and schema compliance.

55. 12:23

    Um, the, the third, um, limitation we had was, uh, task complexity mismatch. Um, so in a sense, the fact that ACE, uh, uses uniform processing, um, also for simple tasks, which can be a, a waste of resource.

56. 12:42

    So Meta-ACE adapts, uh, strategy allocation dynamically rather than using the same heavy pipeline for everything. The alphas are allocation weights for the six optimization strategies, and they represent how much computational budget is assigned to each strategy for a given task.

57. 13:01

    So simple tasks, um, require minimal processing, can save, uh, nine... around a ninety percent, uh, compute compared to standard ACE. Moderate tasks, um, is more of a balanced approach, um, that include ACE plus verification.

58. 13:21

    And then complex tasks, um, basically heavy test time compute, multiple attempts, and memory retrieval.

59. 13:31

    Um, so just to conclude with some results, um, and, and these are initial results. Uh, we have observed, um, around an eight to eleven percent, uh, improvement on agent benchmarks.

60. 13:46

    Um, we have also observed a six to eight points improvement on, on some domain-specific tasks. Um, also a thirty to forty percent reduction in compute costs, um, through the allocation of, um, adaptive strategies.

61. 14:04

    Um, and overall, there's, um, there's more robustness, more consistency. Um, and, you know, we can generalize better. We can use the framework across, uh, a diverse, uh, range of, of domains.

62. 14:19

    So the conclusion is that, um, Meta-ACE can, can orchestrate a context, compute, and verification, and memory and parameter adaptation, um, and produce a robust, uh, self-improvement, um, framework for agents.

63. 14:35

    Um, future work will implement, uh, and evaluate the full system across, uh, a, a more diverse range of domains, and we'll continue exploring meta-learning, um, and this would involve also incorporating, um, additional strategies as well.

64. 14:56

    Now, I also wanted to touch on, um, additional applications of Meta-ACE that I think are, are quite relevant. Um, so first one is, um, for multimodal AI systems. So for example, deciding when to use vision versus, uh, language processing, again, can be, um, a, uh, like a, a meta-adaptive, uh, strategy decisioning.

65. 15:22

    Um, also when you have, uh, compound AI systems that, um, require different models for different stages, um, and the complexity is, um, you know, is substantial, uh, we can actually, um, uh, in a, in a meta-adaptive manner, uh, select the most effective, uh, strategies to, to

66. 15:46

    resolve a task end-to-end. Um, also, um, for human collaboration. Um, so in other words, to determine when to have a human in the loop. And also for continual learning systems, um, where we are balancing exploration versus exploitation.

67. 16:07

    Um, so the, the core takeaway is that optimization requires a meta layer of intelligence, um, and that has to be trained, um, and, you know, um, it requires, um, a lot of trial and error before it can actually, um, perform at the right level.

68. 16:27

    In terms of the future direction and challenges, um, there are still several challenges that remain. So the meta-controller's training, um, may be unstable, um, due to sparse rewards, and th-this can be mitigated through curriculum learning.

69. 16:42

    Uh, also robust advantage estimation and, um, regularization of entropy.

70. 16:49

    Also, the computational overhead from profiling on multiple, uh, strategies, um, needs to be reduced with efficient models. Um, we can leverage things like lazy execution, batching, and caching.

71. 17:04

    Um, also, uh, the ver-verification, uh, cascades can be brittle if all models, um, make the same mistake. So we need diverse models, um, with confidence weighting and human oversight, um, as well as active learning.

72. 17:22

    Uh, meta-learning loops require substantial data. Uh, synthetic task gen, uh, task generation or policy learning, uh, transfer from related domains and simple, efficient algorithms, uh, can also help as well.

73. 17:37

    And finally, uh, addressing these chan- these challenges, um, is going to be key to scaling, uh, Meta-ACE and applying it across, um, a wide range of domains.

74. 17:48

    So that was all from me. Thank you very much for listening. Um, and yeah, uh, appreciate you being there. Thank you.