Burn Your Flags: Interactive CLIs for Agents — Mark Lummus & Navinkumar Patil, PayPal

Hi, everyone. Welcome in. My name is Mark Lummus. I'm Head of Developer Tools at PayPal. With me, I have Navinkumar. He's our tech lead for the developer tools team. And today, we're gonna talk to you a little bit about CLI. The title of our presentation is meant to be a little provocative, and you'll understand why at the end of the presentation.

So today, we're gonna talk a little bit about a specific kind of CLI, interactive CLI, and we're going to set up an exercise. And we're gonna give you, for those of you who wish to participate, this is a workshop after all, we'll give you a CLI that you can make work better in an agentic workflow. So that's the purpose of this

workshop today. So we're gonna give you plenty of time. We'll talk for a few minutes, try to keep the talking to a minimum, and then we'll get you started.

Now, it's kind of interesting. I think someone said, "What's old is new." When I started my career, long time ago, I have gray hair, everything was a CLI. Everything was a command line interface. Any of you remember working with MS-DOS before we had Microsoft Windows? There you go. My first job out of college, I was designing embedded systems, and so you would connect to a computer

over a serial port, and you would pull up a VT100 compatible terminal, and you would start typing, and that was your CLI. That was the, the, the way you interfaced with the computer. So with today, CLIs have k-kind of had a resurgence, I guess, you know, with the-- maybe it was Claude Code that really made them much more popular. Now we have Codex. Um, and so a lot of people are, are adding CLI, uh,

tools so that it's easier to access the capabilities that they want to expose to an agent. Now, with interactive CLI, this is why we're here today, with what we wanna talk about. Um, agents shell out to bash constantly, and that's good. But if they run an interactive CLI, that may be bad because not all harnesses handle

interaction very well. And if they don't handle it very well, they'll fail quietly. You won't know. No errors, no nothing.

But why do we care? Interactive CLIs are great because the next question can emerge based on what-- the way you answer a previous question. Flags multiply if you try to create flags that predict w- all the possible flows. The choices are contextual, and they can also be responsive to what's happening in an environment. So if your CLI

is calling an API and getting a response, and then the question comes up based on what the API returns, that's really, really challenging to handle with flags. And the guardrails can live in the interactive flow.

So here's an example. Over here on the, on the left, we see a, a, a sample CLI. It's interactive. The CLI's opened an interactive prompt, an arrow key prompt. The human sees it. The human can use the arrow keys, but the agent doesn't see it. The agent sees a redraw loop, not a question. And the harness

loses the UI state. So here's the failure mode. It's silent and invisible.

So the common fix, we'll just add flags. And flags are fine for simple use cases. You have, uh, you know what the, you know what the flows are upfront. It's a single pass. All the options are independent. Great, great use case for a, a, a, a simple flag.

But when the answers change based on the context, or they're changing based on reactions to real-time events, that's when it becomes a problem. You really can't use flags for those situations.

Now, an agent is not like an automation script. An automation script knows nothing. All of the decisions are pre-baked upfront. The agent can reason, it can react, it can ask, it can retry, it can back up, and most importantly, an agent can walk a decision tree if you provide, provide one. You can't do

that with flags.

So we already have a model today for what's happening here. It's called accessibility And so if you map what we do for humans to what we need to do for agents, then the solutions become apparent. It's the same discipline with a new reader.

So here's the, here's the challenge: how might we keep the guardrails and the guidance and stop pretending that agents and humans have the same abilities? Now, I'm not gonna answer this question. That's the whole purpose of this workshop.

So here's the challenge. We've got a CLI, we call it Bris- Barista nine thousand. It's gonna take coffee orders,

and it's gonna fail when you try to run it from an agent because it's interactive.

And if you choose to participate in the challenge, the winner will win a Mac Mini. Walk, walk away with it today.

It's in that bag right there.

It's real.

That was a challenge in and of itself, getting a Mac Mini. I'll be straight up. Okay. Now, simple rules. We're going to give you a CLI.

You make changes to that CLI, run it through an agent, and it's gonna send some information up to our server, and then it's going to update a leaderboard. Now, we'll put, put the leaderboard up in a moment. We don't have two inputs, so we're gonna change the slide deck out for the leaderboard so you can see it. Now, here's Naveen. He's gonna talk you through this.

Thank you, Mark.

All right?

Yeah.

Cool. Yeah, uh, I think it's an exciting assignment for everybody, and this topic is very near and dear to me, but, uh, keeping things aside, Mark, are we getting any coffee after we build this?

I hope so. If we're successful.

Okay. Cool. Mm. So yeah. Let me, uh... Before we start into, um, the assignment, I wanna take to a journey, right? Uh, when I started my engineering career, I started to play with IDEs. I started to use Eclipse because I was primarily using Java, and as time went by, moved into IntelliJ, uh, JetBrains. It was amazing. Looking back

five or six years behind, uh, things started to evolve. There were new IDEs in the market, especially Microsoft VS Code, which gave birth to a lot of, uh, other IDEs as well.

So if you look right now, Microsoft had its own VS Code editor, but later on, new AI IDEs started to pop out, like Cursor, Windsurf, to name a few. In all these journeys, CLIs were constant, right? At some point, I thought, "Okay, you know, CLIs are having a slow death of their own. Maybe they are not gonna come back." But, you know, uh, I, I'd, I'd

actually like to say this Hangover movie dialogue that, um, you know, all, all the four friends are in the car and they say, "Hey, we are back." Imagine CLIs saying that to each other, and in the world of AI, they are actually back, right? Whoever bought it, yeah, I don't know. But the whole story is now the CLIs have changed, and who's responsible for that? Agents, right? So this workshop is

primarily focusing on agents, your CLIs, and this is gonna be a hands-on exercise. There are rules to it. There are no rules to it. So feel free to hack as much as possible and you know the prize is here. Um, so yeah. We have... Oh, maybe, uh, Vinod, if I can get the laptop. So we have couple of things. So we have... Yeah. Um,

so we have the repo, which is PayPal Cafe. It has the CLI that you can actually do- download. That's the bare bone CLI. Now, it's not gonna be that easy, uh, for a Mac Mini, yeah. Uh, so you will get some initial commands with it, uh, play around with it, register yourself, uh, provide your username, whatever you like, um, your nicknames, anything, and try to brew a coffee out of it, okay?

That's, that's the goal. And in this whole exercise, we want you to also be innovative in your approaches, and going forward, it's gonna get interesting with agents using the CLI, so think it from that perspective. Let me change this. I just change my laptops here.

Now, during the, during the session, we're, we're not starting you out with any hints. Once we feel like enough people have gotten the, the repo installed, can run the, the CLI as is, then maybe every fifteen or twenty minutes we'll share a good idea. Now, I'm not gonna call it a best practice. I'm just gonna call it a good idea.

Because I feel like we're early enough in the journey here where we don't know what the best practices are, and we're kind of feeling them out. Now, they're good ideas. I'm also hoping that at the end, we'll hear some great ideas from you in the audience so we can share. So just keep that in mind. We're, we're all learning. This is a journey. We'll come back to that towards the end.

It's not working.

Give us a couple of minutes. I think we'll get the technical issues resolved.

Seems like people are actually using it. So on the right side, you will actually see Get CLI, and, um, if you see paypalcafe/rules, that's where you will have all the rules outlined to it. Feel free to start working on the CLI till we figure it out.

Okay.

Do we have the URL?

Use something else.

Oh, is it that one? Yeah.

We can do that one.

No. I think...

Yeah.

Hey, there you go.

Nice.

Sorry, I just stopped it.

Awesome.

Thank you.

Thank you so much. Hey.

Cool. Mm. So you can see this is... Oh, we do see some signings. Okay, so on the right side you will see Get CLI. You can download that and I hope you guys can see it. It's at the bottom right and there are some rules here. You can click on these rules and it will pop out some rules.

So we'll probably take around about twenty-five to thirty minutes, um, and in between we'll interrupt with more slides and make it interesting.

Where's the laptop for the other slides? Vinod's got it? Okay. That's fine. No, no, no, you, you...

And we're here to help. If you're having trouble, please raise your hand. We'll try to assist you as best we can.

I'll tell a little bit, I'll tell a little bit about the models.

Don't use high models.

We-

Do you, do you want me to, you want me to say something?

You know, we had a, a little bit of a debate internally. There are some great models out there.

The interesting thing about this challenge, if you use the highest thinking mode on the bleeding edge LLMs, you can probably brute way, brute force your way through this. But I feel like in the current climate, teams are starting to think about their token budget. And so you don't necessarily wanna waste your high thinking tokens on running

a CLI tool. So the real challenge is can you make Haiku or Sonnet or something at that level run your CLI? We know you can do it with the highest. Can you do it with something that's somewhere in the middle that's faster and cheaper? That's the, that's the interesting challenge if you, if you wanna take it on.

Sounds fun. Yeah.

Awesome.

Since we are on the coffee topic, uh, Mark, were you aware that San Francisco City has the best bakeries in the entire United States? Were you aware about that fact?

I was not aware of that.

Oh, yeah. You should-

Go on.

Yeah.

Go on. Tell me more.

Um, I have, I've done some bakery crawls, uh, around in the city, and they are so fun. Like, you should definitely check out all the amazing bakeries out here.

Okay.

Yeah.

Any favorites?

Butter & Crumble. Yeah, my favorite one. Although I couldn't go there.

Awesome.

So as of now, we have J Gray Codex at the top. So guys, please catch up with him or her.

So with the init command, you could actually register your names. I hope some of you have figured that out.

Do you want to talk about the first half?

Yeah.

Five

minutes.

Five

minutes,

okay.

Do

you want to know

?

I'll-- Once I plug it in, I'll put these things.

Okay.

Okay.

Wanna go ahead and-

Okay.

Here it is.

Cool. Um, let's move on to some presentations. Meanwhile, you can start working, I mean, you'll continue working with the CLI that you have. But as, as a first hint, you know, it's, it's a good idea to start using Tmux. How many of you know about Tmux here? Okay, quite a fair amount of audience out here. Um, so yeah, um, in short, start with running Tmux and, um, start by

putting help command. Now, help is really important because every AI agent in today's world relies on the help command before it even starts using that command.

So...

Yeah. Send keys and capture pane gets you far. Now, Tmux actually, um, for those who actually don't know about it, is like a multiple... You could actually use multiple terminal sessions, uh, with the Tmux, and it's like a split pane, uh, feature that you could leverage with Tmux. So, with this example, you can see here, um, I mean, the screen is very short for me, so let me just zoom in a little bit. Yeah.

Tmux-d. Now, what this flag does is it detaches the entire, uh, CLI and puts it under a different Tmux instance all together for you.

So, you could see size. Uh, Tmux capture pane, like I said, you could actually have split panes in Tmuxes, so you could actually take care of, um, working around with, um, in, in the CLI. And, uh, that's one of those cool features that can help your agents perform a lot of activities. Um, think of it when you want to run multiple agents,

you could actually spin up different sessions, and Tmux actually takes a log of those things where it knows what are you working on and resumes those sessions whenever it's needed. So it's actually helpful for agents to communicate and start communicating when they have multiple sessions open, and these are some long-running tasks that you could actually run inside Tmux.

Second good idea is IPC side channels. So, like I mentioned before, Tmux was a good upgrade. So instead of doing screen scraping, which Tmux actually does, um, for visual in-intelligence, you could start using side channels. Now, side channels would help you define your arrow keys that you have and help you understand, and I'll go in the

next slide, uh, how it works. But the agent should know what is the output JSON. Now, this becomes really critical in the entire talk, is output JSONs, and agents want that. Since we have multiple personalities now, earlier we used to only cater to human beings, now we have agents. So think of both these personas so that your CLI can actually take advantage of both these use cases.

So side channel, for those who don't know about it, let me explain to you in, in simple words. Think of it like adding a REST API to your web application, and the website gives humans a visual interfe-- uh, interface. The API gives you a machine-structured interface, right? So think of side channel as exactly the same. Now, that's one of those analogies that I can refer.

Read state, send input as a data.

So we have the barista brew --hyphen, hyphen agent. Now, th-this is a command that will help you run- Your CLI with background agents, right? Um, you want your agents to be running in the background and, uh, you put all of these commands, they are gonna help you. Try using it in your current CLIs. See if you are able to, um, make some progress in

the Barista Brew application that is out there for-- in your laptops. Now, there are a couple of things that you could actually provide as an example out here. You can see Barista Brew ask, and then there is a bunch of information out there. Oh, what is it? Okay. Fields, then small, large, medium, maybe the sizes of the coffee, and yeah, default as large. What does that even mean? That's input data. So all you have to do

is feed in this input data so that agent actually knows that, and you gotta do it in machine-readable formats. So this is where, um, that thing comes in play.

So don't just tell the agent what the human sees, okay? Think it out of the box. What are the agents going to think? They have different skill sets, and you have to start leveraging those skill sets.

Same prompt with a different brain now. So on the left side, you see I was talking about different personas. One is the human beings, and the other one is the agents.

The CLI is the same, but when you look at the way it is structured, human beings want set of options. But on the other hand, you have agents wherein you could provide all this information in one single input file. Okay? So this is where we call this as context injection comes in play. You provide context in such a way that... Let me take an example.

You, you are actually low on oat milk, okay? And, uh, you have only two gallons of, uh, oat milk left. Tell your agent what kind of information it needs to ask the user. Confirm it. If two ounces are good enough, do you want me to pick a backup for you? This information should be provided using context in-injection as a technique,

and you can leverage this in the CLI that you are actually building. Try using it. Maybe you'll be on the highest on the leaderboard.

So we started a journey from Tmux, which is basically screen scraping to injecting skills.

So Tmux is low effort. I think a lot of you might have tried it, but this is where side channels also will help you in providing good, um, sort of feedback to your agents, and context injection will provide you those different scenarios that you can think about, which doesn't happen in the human case.

So yeah, keep pushing your scores.

Do you wanna show this now?

Yeah.

It does take some time.

There you go.

Okay. Cool. We do see a lot of traffic out here.

Oh, somebody's really close to five thousand. Is it Bang? Uh-

Hong

Hong. Okay.

Any questions so far anybody wants to get it answered? Any-

Over there

... queries? What happens if multiple people get five thousand points? Sorry. What happens if multiple people get five thousand points? Okay, let me repeat that question for you. What happens if multiple people reach a score of five thousand? Mark, you wanna take that?

So the, the idea was we would look at score, and if there's a tie, we would look at which one took the least amount of time. Now, based on the resolution of this, I see several that are-- that have completed three of three in zero seconds. I see a bunch of these that have completed it in zero seconds. So I need to consult with someone because that's

not supposed to be possible.

But the idea is that we would look at the high score, the, the score first, and then the time it takes to respond second, so that an accurate response returned quickly would be biased over an accurate response returned slowly.

D one for now

Okay. I'm sorry?

We have only one task to do, but eventually-

Yeah

... if they become under tasks, we might have to think about...

This is the bottom of the leaderboard. Nicholas, game up.

I think everybody has that zero second right now.

Yeah.

What time do you wanna end this?

A couple more minutes.

Let's wait two more minutes, and then we can open it up for ideas, and maybe do ten or twenty minutes of ideas according to how engaged they are.

Is that all right?

Yeah. Absolutely.

Did we have any?

In just a few minutes, we'll open up the floor for some idea sharing. So I'm, I'm hoping that we'll hear some great ideas that go above and beyond our good ideas, 'cause I know that there's some out there.

Mark, were you expecting so many people to be in the four thousands?

Yes. Absolutely. At this conference?

Yeah.

Absolutely.

Awesome.

Were you not?

No. I was.

Okay.

What do you think, anybody can make it to five thousand?

I don't know. We'll hold it open.

Why don't we open the floor up now?

Yeah. For sure.

So we'd love to hear ideas. So did anybody in the room come up with other ideas that were able to get them to such a high score so quickly?

Nobody wants to leave the desk.

Oh, we have... Yes.

It's, it's not an idea, but it's a very gameable benchmark because you only-- You really only have to fill, like, one order that has to be instant. So it's very easy to, like, crank up score without really doing better

Let me, let me come over to you because I can barely hear you.

Oh, sorry.

Okay.

Go ahead.

It, it's a very easy to game benchmark because the only thing that matters is you fulfill one order within, like, small-- like, a very small latency. So if you do a deterministic parser, um, it... Which defeats the spirit of this challenge entirely, but you det- do a deterministic parser, and you have something, like, really close to the server, you can get the latency down, uh, really quick if you just brute force it.

Okay.

Which-

Is, is that what you did?

That's what I'm doing right now.

Okay. All right. Now you said something interesting. The, the deterministic parser that's running in your CLI. Okay. I actually like the idea of the determinism in the CLI. And the reason is that if the CLI is a tool to be used by an LLM, then you

absolutely want it to be deterministic. The LLM is the non-deterministic player here. We want the CLI to be, as someone described it, a small, sharp instrument. Should be very, very focused on what it does, and every time you give it a command, you want it to come back with the same response So you don't want to build an agent into the CLI and then

hand that CLI off to another agent because you don't know what you're gonna get.

Any other interesting ideas? Obviously, we've-- s- a lot of people have figured out how to game the system.

Where there's a Mac Mini, there's a way, I guess. Yeah.

Mark, begin it. This, end this one, wait five minutes or-

Yeah, five minutes. The, the score hasn't changed in a while.

Before the site crashes.

Did you do refresh?

Yeah.

Why does it say reconnecting? Oh, there we go. Okay.

I think the network is a little laggy. The network is a little laggy out here.

Yeah.

There are a lot of com- a lot of computers.

Yeah. Sounds good.

Yeah.

It shouldn't crash.

Yeah, we're frozen right now.

Here we go.

Been no change in the results.

Yeah.

Okay.

Do we wanna wait a few more minutes or call it?

Yeah, like five minutes.

Five minutes. Okay.

Do you wanna bring the PPT to... PPT for the closing arguments or?

Yeah.

We're gonna give it about five more minutes. Yeah. And then we'll call it.

Where's Bon- where's Bonk? Raise your hand. Who is Bonk? Okay, we have something here. All right. Big smile, big smile.

And we do wanna hear how you did it. So once, once we call it, then we'll, we'll, we'll, we'll let you explain what you did. I'd love to hear.

The agent. That's why we're here, right? Um, no, I think what it-- what the agent ended up doing was building a deterministic solver, uh, to the CLI, which is normally deterministic. And then through, um, basically it has a skill. It already has a skill when you use this for multiple use cases.

Um, I need to go over here. Yeah. H-h-hold, hold on.

Sorry, I should have asked for another, I should have asked for another microphone for the room for questions.

Um, yeah. So Bonk is just the, an agent, uh, thing built on top of Claude Agent SDK. Um, it built the whole thing. Prior to this, I already had a sort of eval bench skill that will create sort of a, an environment and, um, optimize. So it basically ended up building a, uh, deterministic CLI solver, and then now it's just running on a tmux and slamming your APIs.

Okay.

Yeah.

All right.

Also, I think I got lucky because I hit it at one point at, like, twenty-nine milliseconds, uh, probably because we're in the same region.

Okay.

It's just a network latency.

Right. And this is all done on Claude?

Uh, Claude Agent SDK.

Okay.

I'm just driving it through Slack.

Now, which model?

Opus four dot eight.

Okay.

Yeah.

Did you try it with any other models?

For the build? Uh, for building the CLI or?

For r- for your a- for your agent, how would we do that? How would-

The, the CLI is completely deterministic.

Sure, sure, sure.

There's like no L- So the agent built the CLI and is just running it on tmux now.

Okay.

Yeah.

Awesome.

Yeah.

Congrats.

Thanks.

I'm gonna go to the next one. Danny. Who's Danny here?

We have one in the back.

You see him? He's back there. Thanks.

Last one.

What, what do we wanna know?

Yeah. How'd you build it? What was your technique?

Yeah. So I don't know anything about, uh, coding at all. I'm, I'm using, uh... I'm basically a vibe coder, and I have a couple agents, and I just said, "Hey, here's the problem. Read it, and then fix it, and fix it." So I just have, yeah, Claude code up on, uh, VSC.

Have you ever built a CLI before?

Uh, no. I, I saw a couple hours ago that obviously there was a prize to be won, so I got in and I said, "Hey- "Uh, figure this out. Explain it to me like I'm, you know, like I don't know anything," and then-

Explain like I'm five?

Yeah. I, I, I don't know. So... And then we came up with, uh, yeah, we came up with some good results and I said, "Hey, we need to make it faster, and give me some suggestions." And yeah, seriously. So I, I guess we have, uh, two scripts running, and the first one got the, uh, forty-nine ninety-eight, and then I said, "Hey, make a copy and give me some suggestions for how we can get the forty-nine ninety-nine."

Right. Did you get any suggestions?

Uh, it's-- this agent's pretty automated, so I just say like, "Hey, here's the start, here's the end. Give me a couple, you know, questions at the beginning if you need it, but then fix the problems so I don't have to deal with it," kind of thing. So sorry, that's not a very good...

No, no, it's good. I'm, I'm sure a lot of people are wondering, like, writing a CLI, is that something that you would ever do? Or would you ask your agent to do it for you? So clearly, we have many people in the room who are just gonna point Claude or Codex at this and say, "Build a CLI. Make it fast. Make it deterministic." And those are the three ideas, right?

Yeah.

Okay. Do you wanna change the-

Yeah. Change the slide.

There it is.

Yeah. Okay. Uh, where was the...

So we talked about all of our suggestions. We heard some ideas. The predominant idea was let your agent figure it out for you. I love that idea. Um, anything else before we close it out?

All right. There you go.

Takeaway for us, based on our instructions, our recommendations: give it good context, give it a side channel, and tell it to use Tmux.

Those are three-

Good ideas to start with.

Thanks, everyone. Monk, you wanna come on up?

We still have Monk as the winner, right?

Yeah.

Okay.

No, no. We called it. We called it five minutes ago.

Yeah, okay.

Yeah, yeah. I'm not gonna change, I'm not gonna change it.

Yeah.

We, we called it.

Monk. Whoo. Awesome.

You wanna take the-

Oh, that is-

Oh, yeah.

Oh, we do.

We'll get-- We'll just get a picture.

You, you get your-

You know, social media.

That's right.

It's a real Mac Mini.

It's a real Mac Mini? Oh my God, these exist?

All right. Awesome.

Okay, cool.

Where to come this-