Anthropic's CCA Exam as a Field-Guide for Agentic Engineering

Talk transcript

1. 0:00

   [outro jingle] Okay, getting rolling and, uh, welcome aboard.

2. 0:15

   Just had a little technical issues, but, uh, resolved them. So my name is Frank Coyle. Uh, I am a computer science guy. I've been teaching computer science for over thirty years, and I'm now teaching at Berkeley.

3. 0:28

   And one of the problems that, uh, all my students, past and present, are having is AI, because computer science is no longer the magic pathway to a job. So I've been trying to figure out ways to, uh, help them come up with schemes to help them get ready for this world of agentic AI.

4. 0:49

   And one of the things that sort of, uh, dropped into my, uh, plate was the-- something called the Claude Certified Architect Exam, which I will be talking about today, and it has, um, a number of aspects to it.

5. 1:03

   And I think if you're interested in a career in agentic AI to certainly take a look at least what the exam is about, because I feel that, uh, Anthropic knows how people are using their system and what the issues are gonna be.

6. 1:19

   So before we jump into that, I wanna give a little bit of my, uh, my philosophy.

7. 1:27

   Wup, wup, wup, wup. May have to do this manually. Getting stuck.

8. 1:36

   So this is a quote from, uh, a [REDACTED:gender] named Sister Corita Kent: "Nothing is a mistake. There's no win and no fail. There's only make." Bottom line here is experiment, experiment, experiment.

9. 1:52

   Not only should you read, but you should do. You should make stuff. Now, what happens when you make stuff? A lot of times things don't work.

10. 2:03

    Thomas Edison said, "I have not failed. I've only found ten thousand ways that don't work." And

11. 2:13

    w-what I want to emphasize here is that what this shows us are something that in the design patterns movement which came around in the early nineteen nineties with object-oriented programming, we had patterns for objects.

12. 2:27

    We now have patterns for agents, but there's also anti-patterns, and I think anti-patterns are a key to understanding what you should not do, because understanding what you should not do is the key to leading you to what you should do.

13. 2:44

    So a little bit about the Claude Certified Exam, released in, uh, March, so it's brand new. It is, uh, it is based on scenarios. It is timed. It is proctored.

14. 2:57

    It is available to companies in the Claude ecosystem, the Anthropic ecosystem, but individuals can pay ninety-nine dollars and take the exam once every, uh, once every six months. And it's not just multiple choice questions.

15. 3:15

    It is multiple choice, but they're so-- they are based on, uh, realistic constraints and realistic scenarios.

16. 3:24

    The five domains. There are five domains that are covered, and they give you the percentages of each. So agentic architecture, twenty-seven percent. Claude Code, how to configure the Claude Code system and workflow, twenty percent.

17. 3:39

    How to... Doing prompt engineering, structuring your output using JSON all over the place. Tool design. Model Context Protocol integration. These are topics that you should understand and know whether you're gonna take the exam or not.

18. 3:56

    This is gonna help you get ready for whatever the agentic world is gonna throw at you. And then there's gonna be context management and reliability. So these are the areas of, of the kind of questions you're gonna run into.

19. 4:13

    Then there are-- and they, they provide you with six production scenarios, and your-- the exam will randomly choose four, and all the questions will be centered around the four that they choose.

20. 4:29

    And what I'm gonna do is walk you through, um, the production scenarios and give you some anti-patterns to be aware of, because there's a number of ways you can solve the problem, but one of the big things is what not to do, and that often can be the key to getting these questions right.

21. 4:47

    So number one, customer support resolution agent. So we have agentic loops, control, something called stop_reason, which is, uh, what Claude Code has. Every time something happens, there's a stop_reason, and you need to take a look at that because that can give you a lot of information about what's going on.

22. 5:05

    Uh, scenario two, code generation. Three, multi-agent research system, which we'll look at. How do you, how do you distribute your agents? Hub and spoke? Who's the orchestrator? How much information should they know?

23. 5:20

    All these are im-important factors. Um, scenario four, developer productivity with code. So how do you do subtask isolation? Keep your tasks in their little universes. And this harkens back to what we learned in computer science from doing multi-threaded programming.

24. 5:39

    When you have multiple threads operating and sharing memory, then you get into issues with synchronization. You gotta put locks in. Keep the little threads independent. Keep your agents independent.

25. 5:53

    Um, and then some Claude Code for continuous integration, and then we'll look at some patterns for structured data extraction. Okay, that's kinda where we're gonna go.

26. 6:06

    Now, here's something that I, I, I like to point out. Everybody's talking about loops, right? Every-- The loop is the new thing. Um, uh, Boris Cherny says he doesn't write code, but his job is to write loops.

27. 6:22

    And Peter Steinberger, master of OpenClaw, says, "I don't, I don't, uh, I don't code anymore. I just design loops that prompt your agents." So loops are the new big thing, right?

28. 6:34

    Well, no, they're not. Okay? [chuckles] Um, back in the day,

29. 6:41

    uh, early days of computing, we had programming languages were exploding. We had Fortran, we had COBOL, and there were big fights. "My program, my programming language is better than yours.

30. 6:53

    It can do more." "No, it can't. We can do this." Beame and Jacopini, nineteen sixty-six, proved that if you want a language to be Turing complete, which means can compute anything that computers are possibly able to compute, then you need only three things:

31. 7:13

    the ability to s- to, to write statements sequentially, okay? To have if-then conditionals, and the third piece is the loop. If you add the loop, you have Turing computability.

32. 7:28

    And now we are seeing this being resurrected in the agentic world with the focus on loops. 'Cause up to now, we've had sort of sequences. You have prompts. You have maybe if-then, but now we have a loop.

33. 7:42

    And now this is what's giving us the power. This is where the agentic stuff is getting very exciting. Okay.

34. 7:51

    Start with, uh, with scenario one, customer support resolution. So here we have

35. 7:59

    a loop operating and the... I'm gonna jump to the anti-pattern. What you don't want is just to let the agent go and do something and get the response back and

36. 8:11

    use it. Okay? What you wanna do is you want to loop with something called the stop_reason. So I'm gonna show you a little code here.

37. 8:19

    So here we have while loop. It's a while True. It's a loop. We're looping right here. Okay? So the first little block is where we call, uh, we call the model, okay?

38. 8:30

    And we pass it the messages. The messages are essentially the sequence of prompts that n- exist in the context window. Okay. And we are asking the... W- And we have a, we have a prompt, and we have, we have the context, and we have a tool.

39. 8:48

    And we're asking the LLM to do something with this tool and help us out. The problem is the LLM can't do anything. It is just a probabilistic next word predictor.

40. 9:01

    It can't execute tools. So what it does though is it can figure out.

41. 9:08

    If you point it to a tool, it can figure out how to set things up so that you or your code can execute it. So it's important to understand that the LLM is not executing these tools.

42. 9:20

    It can't do anything except talk back to you, very intelligently sometimes, but all it can do is talk back to you. So when it finishes this task and has a result, which is basically, "Here is...

43. 9:37

    I've s-- I know what you want. I know what the tool can do. Here's how," uh, it sets up the parameters that can then be, uh, then, then used to actually execute the tool.

44. 9:48

    So this second block, you see, why did the LLM come back to us? That's our stop reason. tool_use. Oh, okay. We've stopped because the, the LLM, it wants to use the tool.

45. 10:03

    So let's just run the tool. So that's what this second block is. run_tool. The response is what the LLM said, and it's basically the parameters that it has extracted from the data that you provided it.

46. 10:17

    Okay? Then it executes that. Then it goes back. Then, then it continues. Continues means the LLM sees it, says, "Oh, successful run. So okay." Come back down.

47. 10:31

    We're not running a tool anymore. We're end-- ended our loop. Bingo. Now, then we take the answer, and this is an opportunity for you to have a human-in-the-loop potentially.

48. 10:43

    You check the confidence. If it looks good, you keep it. If you don't, then you escalate to a human. So, now there's another reason why you w- need to make sure you check your stop reason.

49. 10:55

    One of the stop reasons may be you have run out of tokens, and this response is based on partial when the LLM had to stop. And it's gonna give you a response, but if you have run out of tokens, then you need to take action.

50. 11:12

    Okay. Um, next scenario. Uh, code generation with Claude. So Claude Code has this co- has this concept of the CLAUDE.md file, a markdown file where you put all the things you want it to know.

51. 11:28

    What Anthropic recommends is you have three levels of Claude. One that you have at the top level of your project, the other that you have in- inside your sort of, uh, the, the project folder, and then within directories you can also specify.

52. 11:48

    So the idea is to have a hierarchical set of rules that, that could then control how the system is gonna respond.

53. 11:59

    Okay. Moving right along. Uh, we have a multi-agent research system. So here we're gonna have, uh, the, the problem is-

54. 12:10

    How do I, how do I get my agents to, to go off and do stuff and bring the answers back in a reasonable way? The anti-pattern,

55. 12:18

    you have one agent and you load it up with tools, all right? So I like to think about you- you know, you hire somebody to come to your house, you hire a carpenter to come to the house, and the guy shows up with, uh, uh, uh, plumbing tools, carpenter tools, electrical tools.

56. 12:34

    He says, "I can do anything." Well, maybe you don't want this guy. Maybe you want a, a professional carpenter. So that's the kind of idea. And this kind of b- back-- takes us back to some of the, the functional programming, uh, ideas, that functions should be do one thing, and if you can get your agents to do

57. 12:53

    one thing, you... with maybe one or two tools available to it, then that's gonna be a win, and that's gonna help you with this exam. So specialize, don't overload.

58. 13:07

    The other part of this is don't let your agent's context s- spill over into the main context, because context means tokens, tokens mean money, and the more context you have, the more confused the LLM is gonna be in giving you an answer.

59. 13:27

    So even though, oh, a million token context window, I can put everything in there. No, no, don't put everything in there. Limit what's gonna go in there because then you're gonna get a much more accurate system.

60. 13:42

    So here's a, here's an example of a, uh, specialized sub-agents. You're giving it--

61. 13:50

    So this would be the critic. So let's say you've run some stuff. Now you wanna get an agent to look at what's happened. What you wanna do is just give it what it needs to solve that critic problem.

62. 14:03

    I'm only giving it here the, um, we're passing it the claim and the evidence. So this is-- your claim is sort of how we're gonna solve the problem. Here's, here's the evidence, but you're not giving it the, the thought processes that went in to creating this claim.

63. 14:24

    Why? When you, uh, uh, when you get a bunch of agents together collaborating and talking to each other, there's a tendency to have groupthink, and all the agents seem to kind of devolve into one idea.

64. 14:41

    I mean, it's, it's like, you know, you're in a group, you know, you're at a party, and everybody wants pizza except you. But then people talk you into-- You, you know, you don't wanna be, uh, you don't wanna spoil the party, so you go along.

65. 14:55

    And it seems that agents kind of work in the same way. So you're gonna return-- Y- basically, you're gonna give each agent only a slice. I didn't think about the pizza analogy, but yes, [laughs] every, every agent gets its own slice and, and it, uh, it should come through.

66. 15:12

    Okay. Fourth scenario, developer productivity. So the anti-pattern,

67. 15:25

    let every subtask dump its full output into the primary thread, crowding out the context. Again, this is what we're ju-- I was just talking about. This is bad. Let the context grow unbounded.

68. 15:37

    Bad, right? For the reasons we just talked about. You wanna isolate your subtask output, and you want to compact long sessions. I'm gonna take a second to talk about that.

69. 15:50

    So here's, here's a, uh, an example of a pattern. Uh, you want to have your agent, uh, look at the logs and create a summary of where the problems are in the log.

70. 16:06

    So here's your task, scan all the logs for error. Context, fork. So you're forking the agent into a s-- like a separate thread where whatever the agent does, and thinks, and adds tokens to, does not come back and pollute the main, uh, the main context.

71. 16:28

    Now, you see here what happens, then you take this summation, and then you add that summation without all the other stuff into the overriding context. Now, this last little block is kind of interesting, I think, because you can check your token count, and you can determine how big the token count is.

72. 16:54

    And if you can set some limit, uh, you know, so if, if you have more than a hundred and fifty thousand tokens, then what you wanna do is you can run a compact.

73. 17:03

    So Anthropic and Claude have these compaction algorithms that take this giant context and, and compact it in some way, shape, or form. Not quite sure how the implementation is of that, but there is compaction.

74. 17:18

    Now, a little side effect, a little side channel. I've been walking around-- When you walk outside, you see, see these guys handing out these books, okay? A- anybody see these guys handing out these-- But take them.

75. 17:30

    This is, this is actually a pretty good little book. In fact, I was looking at it last night, and one of the things it had in it was-- This is by this guy, Sam, Sam Bhagwat.

76. 17:40

    I have no connection. I don't even know Sam. But it-- there's a-- on line page thirty-two, it says, uh, "His company provides custom logic for compression of context." So he's got an a-- And you can write your own.

77. 17:55

    He's got a, he's got-- You can extend his base class and have your own compression of your data, whatever you think is important. So I think that's kind of an interesting spin on this whole thing.

78. 18:07

    Okay. Claude Code for, uh, uh, continuous integration.

79. 18:16

    Uh, anti-pattern. Always have interactive modes in a pipeline. Well, no, no, no, 'cause interactive modes mean,

80. 18:24

    uh, uh, Claude will stop and ask you, "You wanna do this? You wanna do that? Can I have permission for that?" So there are ways to set it up so that it'll just run straight through, okay?

81. 18:35

    The other, uh, the other tip that I'll give you here

82. 18:41

    is there's something called the, uh, the batch. So you can take your prompts, you can take your work, and you can put them in a batch, and for fifty percent fewer token cost, you'll get the result they promise in at, at least twenty-four hours.

83. 19:00

    So if you're gonna go take a nap, you're gonna go on vacation, you're gonna go out, take a, a day off, run your stuff in batch mode, and you're gonna have, uh, uh, less to pay.

84. 19:13

    Where am I here? All right. I've only got a few, few minutes left, few seconds left, but I wanna conclude with this. Remember, nothing is a mistake. There's no win, there's no fail, there's no exam, only make.

85. 19:29

    You do it, and you make it, and you're gonna succeed. If you wanna reach out to me, reach out to me, uh, [REDACTED:email_address]. Look at my websites. I got a website, codesupremeai.

86. 19:40

    I'm a big jazz fan, and I named this website after John Coltrane, "Love Supreme," if you know that song. Great. Anyway, that's my story, and I'm sticking to it, and I'm about to zero time. [audience clapping]

87. 19:50

    Okay. Thank you. [outro music]