Unlocking AI Powered DevOps Within Your Organization

[upbeat music] Well, my name is Jon Peck.

Um, I am a developer advocate for GitHub. Uh, I've been a software developer, ah, since the late '90s, uh, and in developer advocacy for about the last 10 years.

So it's been a fun ride. Um, AI is, of course, all the new hotness. Um, I wanna talk to you all today about really how you wanna approach learning AI inside your organization, right?

A lot of us are at that state where, uh, even some of us old fogies are coming around and accepting that there's a few LLMs that can do things maybe better or at least faster than us.

Um, but there's also a lot of confusion about how exactly we start approaching this and, and what kind of motivations we can give our teams and what kind of approaches to make sure that it's efficient.

Um, these are what I'd say you're looking for, you know, kinda best-case scenarios, right? Um, I like to focus on this third column here, efficiency, because we know, we know that LLMs can pump out huge amounts of code.

So what? The question is, is that actually getting you more feature points? What we're seeing in best-case scenarios is up to, uh, one and a half X in how many feature points you can push out in the same period of time.

I'm seeing averages more around 30% in most companies. I'm seeing a lot of improvement on the increase in successful builds. Basically, that's I've got an existing DevOps pipeline, I've got existing tests and that sort of thing, and when people are using AI, those tests are clearing much more often.

Basically, we're hitting the problems early on. I'm also seeing general increases in developer happiness, which is really buoying to see. Also, quick note, you'll see in the lower right there's a URL, gh.io/fair/unlock.

These slides are all available up there. Please feel free to take pictures. But I'll keep that URL up here and there so you can always just find everything I'm talking about.

All right. So first off, how do we ensure that we're actually gonna use AI efficiently? Um, a lot of people's first approach is go out, find some chat tool, s- throw some things into it, get responses, copy and paste it into your IDE, right?

Terribly slow, hard to specify context, hard to iterate in that model. Also kind of exposed, relatively insecure, right? So step one, kind of obvious, but let's just say it, is make sure you're using an IDE that has your AI incorporated directly, hopefully one that's flexible enough to use drag-and-drop options for pulling in just the right context and

helping it narrow down so you're not just doing everything and looking at all the files all the time. I'll get into more specifics about what we look for in an IDE.

In this particular case, I'll be spending a decent amount of time, uh, using VS Code, that's my primary, and with GitHub Copilot, 'cause that's of course my primary being an employee.

Um, the other thing is that you wanna make sure that you're still ensuring model choice. We may get to a point where oc- orchestrators are good enough to always pick the right model at the right time.

We're not there yet. Um, the advantage of using a combined tool that has an all-in subscription kind of thing is the ability to go and pick the right model at the right time.

So I'm doing something really fast. I just need quick, low-cost responses. I'm using a Flash model. I'm doing deep research, and I'm trying to explore a problem space before I actually begin coding.

I'm pulling out a reasoning model. There's lots of different cases where you'd use these different things. All right. Now, when you're ready to jump in, start with the developer as the operator interactive with the IDE.

And the temptation here in a lot of cases is to start greenfield, 'cause it's so cool and fun. Hey, I hop into an AI. I say, "Hey, I wanna build, uh, Minecraft, except I want it to be using the Doom engine, and I want it to run on my dishwasher."

Okay, cool. So I can do that, but it's not really teaching me a lot about how to interact with the AI, and it's also not what I'm doing on a daily basis.

Most of the time, we're really working brownfield. We're working on modifying an existing application with a decent amount of code already in it, and it's important for us to get the patterns we need as we start doing that work.

So I can jump into something. I can say, "Hey, I wanna do something like I wanna swap out, uh, my ORM from this one version to another," or, "I wanna create tests for the things that I've got already, and I wanna use this data source to do it," and that sort of thing.

And in that way, we start to learn effectively the right pairings of how we create those prompts and how we shape the context that we're handing to the model so it's doing the right work at the right time.

So really important to get people invested in that direct daily interaction and how to create and shape those prompts and find the right places to work. All right. Then you might wanna go back and you might wanna play around with greenfield development now that you have those practices and say, "Okay, how fast can I create something new?"

A big step in improvement with that, and also with the brownfield work, is the existence now of agent mode, right? So Copilot agent mode, others have this sort of thing too.

And what you'll find with agent mode, if you're working in a greenfield context, you can use it to sort of iterate on what the initial scoping should be, get it to help you create your prompt before you actually send the full prompt.

The way that I do this is I'll open up a README markdown or something like that, and then I'll say to the LLM, "Hey, here's the kind of application I want.

What are the kinds of specs I should create?" It'll spit something out, which, which is usually nonsense. I'll refine it. I'll be like, "No, I really want you to use these technologies.

I really want it to have this particular file structure in the trees. Here's an example of what an API call should look like," et cetera. And when I'm then at a point where I have a well-scoped document, something I'd feel good handing to another engineer, then I can hand that to the actual model to do my build.

In brownfield mode, the advantage of agent mode is that it's going to be able to take a large amount of context and narrow it down so I'm not overloading my prompt, so I'm not putting too much into it.

That's gonna have the benefit of it working correctly because it's narrowly scoped enough that it'll actually fit inside the call. It's gonna have the second benefit of having it not overrun, because if I tell it only focus on these files, on these folders, then it's not gonna try and do a whole bunch of other stuff I don't

want it to do, right? So in that case, I'll do something like I was talking about with migrating the ORM. I'll pull in just the models folder. I'll pull in just the back end Python Flask app or whatever it is.

I might pull in a few settings files, and then I'll tell it to go ahead and do that task. The first thing I should see my agent do in that case is it'll look at those and it'll say, "Hey, I'm further narrowing the context down.

Here's my plan for the kinds of actions I'm going to take." And then it's going to go and start building out each separate section of the plan. I can pause it.

I can give it new instructions and tell it continue from there if it's going down the wrong path. Or, this is something we need to get comfortable with as developers, if it's just made 500 changes but I think it's working completely the wrong way, immediately just revert all of those and try again, right?

There's not as much cost to blowing away huge amounts of code as there used to be, so I should be comfortable with just erasing and starting over. Obviously, make sure you're doing your commits in between at each set point so you have a good Git history of things that actually worked, right?

On that note, I'll talk about Copilot instructions in a minute, but also when I'm doing my prompts, if I'm just doing it that way, telling it to keep a change log of what it's been doing can be really, really handy.

So I just have a file where I can see all of the edits that it made. Okay. Now, how do we get into team mode? How do we start taking these practices and sharing them with our colleagues and making sure we're working together well as a team?

Um, so first off, team practices and docs don't go away, but they get better, right? Yes, I'm always gonna have my standards. I'm always gonna have everything from, like, what's my committed dependency file, all the way to my manuals and best practices and playbooks of how we do things.

But with the addition of Copilot instructions files, I can now codify this so when the AI is working with me, it's also obeying those best practices. In GitHub, the instantiation of that is a file called copilot-instructions.md inside the .github folder in your repo.

Use that locally, but also commit that file up into your repo. And then it's part of your job as, say, a lead developer or a team manager to make sure that thing continues to be revised properly so that when the team makes a decision, "Hey, we're gonna go with this linting practice.

We're gonna go to this way of doing our model accessors. We're gonna have a different kind of pattern, and we're always gonna check for this specific problem," it's my job to make sure that ends up in the Copilot instructions file, so as they're working with AI, it's keeping all those same best practices in mind.

Second, find a way to make your institutional knowledge accessible to the AI. In a limited sense, that can be things like providing API specifications, existing patterns, and ways of working to the context as you work.

If you're using Copilot Enterprise, you have the additional ability to create what are called knowledge bases inside your GitHub org. These are basically collections of repositories, each with special names.

So I might have one or two or three repositories that are my best examples of how to write an accessible front end that'll work, work well with my blind operators.

I might have another which is our best practices for using Python in certain specific machine learning cases. And yet another that defines all of our internal APIs, which the LLM wouldn't know automatically unless I was provided that context.

I can create these knowledge bases as named items, and then when I as an operator am working on a piece of code, I can say, "Add the knowledge base for accessibility.

Add the knowledge base for my internal finance APIs," or whatever it is. And now the LLM is informed as I'm working and will obey those best patterns.

Also, this is a, a subtle thing, it's a human thing, but make it okay to be wrong. Make it okay to ask questions and admit your failures and bring them to the team.

'Cause we're all in learning mode here. There's a lot of unknowns out there. And if we try to be perfect and always wait until we've got everything right to push it out, we're not learning and we're not helping the team grow.

If I find a flaw, I find a way that a particular model is working wrong, or I'm having a problem I just can't solve in a particular context, but I think someone else might have on my team, throw it out there.

Put it in your Slack. Put it in your Teams chat. And then after you have solved that problem, take that knowledge that was generated, put it into a centralized guide that everybody can access, maybe in the repo, maybe in a centralized team knowledge store, so that there's a history there of what problems we found, what we solved,

and other developers can quickly reference those. Lastly, just keep an eye on your team and how they're using their AI tool use, right? Uh, in GitHub world, this surfaces as the Copilot metrics APIs, but you're gonna look for other things as well.

You know, whether you've got internal tracking on usage, whether you're looking at volume of code and the type of code submitted, that sort of thing. You wanna make sure that people who just don't happen to come across the right tool, come across the right pattern, or feeling a little bit hesitant are not falling behind the rest of

your team. So it's important to keep track of that so you can help those people who might not have had access. All right. Now we can talk a little bit about governance at a high level, right?

These are further concerns you have inside your organization. One, safety, privacy, right? Make sure whatever tool you're using is actually going through Proxies going through sanitization, et cetera, so what you're throwing out there is not being reused in other contexts, reused for training.

Get guarantees from that provider. Uh, GitHub itself, for example, uh, will actually provide an indemnica- indemnification clause in its enterprises, and also it's fully opt-in for any tracking. So when you come to us as an enterprise, we're not retraining based on your data or anything like that.

Every prediction's getting trashed in its memory context. Also, make sure that you've got the option to do general high-level governance. So for example, we can say at a repository level, "Always exclude these items from the Copilot context."

Then it's not accidentally leaking in environment secrets and that sort of thing as it's making predictions, right? Um, and then lastly, make sure that you have org-wide policy governance so that every single maintainer of every repository doesn't have a huge overhead, that you can say at an org level, "These are the sets of policies we want in

terms of new edge features, in terms of which models people can access, et cetera," and quickly get your wide governance under control.

All right. Coming back to how we work as operators. The other thing I want to remind folks is that while it's, it's really cool and obvious to generate new actual production code inside our IDEs using AIs, what we're gonna find in many cases is that you get good lift there on the development side, but there's a whole

set of other use cases you wanna be considering that you're actually gonna get more lift early on, right? So let's step through these. In the planning phase, I can work directly with an AI to figure out what my large-scale plan is for a feature or for a new product.

That includes brainstorming with it about what technologies are best, what are the latest cool things, what kind of infrastructure might I consider, et cetera. So bring that in so you know you're going down the right track early.

On the security side, we can use these tools to quickly go through large existing code bases and say, "What are you seeing for potentially bad dependencies? What are you seeing for bad patterns?

Is there something like, you know, cross-site scripting that exists inside this product? Can we find those and remove those quickly?" Testing, I find this to be probably the, the number two speed gain, that for a lot of us, we have a huge backlog, and one of those things that gets dropped is do you have full coverage?

Do you have all of your test cases? I can quickly and easily go in to an existing code base, say, "Hey, I want you to generate tests for these modules," drag them in, "and I want you to use this," pull in some JSON samples or use MCP to connect to your database as data examples for how to

test that, "and then I want you to build me a bunch of unit tests using these frameworks." And boom, usually in about 10% of the time it would normally take me to build all those tests, I've got at least a functioning skeleton that I can update a little bit.

Deployment. Most of these models are actually pretty good at infrastructure. So when we get to that point where our application's ready to deploy, I can just say, "Hey, how do I put this on whatever?"

Generate a GitHub Actions file, generate a Terraform file, et cetera, and move those out. So now we're moving even beyond the basic developer scope. You're thinking about, I hope, other roles in your organization that you can bring in and give access to this power who won't necessarily be thinking on the day-to-day about the application, but they can

still benefit. And then I find that the, the number one speed gain, and actually additional role that I want to include, is documenters, right? Because we all get to that point, "Hey, I built the code.

It... The, the code is the comment, and I don't really care if the end users know how to use it. That's somebody else's problem." Just going in and spending that 15 minutes of, "Hey, document what this application does.

Produce a markdown file. Give me a full API spec. Give me an executive summary. Give me examples of how I would actually interact with this. If I'm using a vision-capable model, give me some graphs, or if not, mermaid charts, about what the actual workflows are inside the application."

Those are a huge benefit, possibly directly to your end developers, but at the very least to the people curating the documentation inside your company. You're giving them a leg up.

All right. I've got four minutes left, so this will continue to be quick. Um, let's talk about autonomous AI now. So everything I've talked about so far is semi-autonomous, right?

Even when I'm working with an agent in the IDE, it's doing a bunch of work, but it's pausing every now and then saying, "Am I allowed to access the terminal to do this?

Does this plan look good?" Et cetera. But we're getting to the era now where there are some things we can move into fully autonomus- autonomous modes, right? A couple things to keep in mind here.

A, you're using this as a time saver. You wanna use automation where it makes sense, but you don't wanna drop it in everywhere blindly. And you wanna make sure that there's always a human in the loop in some way.

As an example, in GitHub world, this means that when we're using autonomous agents, they're always creating a new branch to work on, and its execution's in a protected environment so that it's not gonna mess up your main branch of code and it's not gonna leak things and destroy existing environments.

It's gonna work in a box until I say it's ready, and I'm like, "That branch looks good. Complete the PR," and then we'll actually merge it in. Don't skip that step.

So a fuse ways we-- a few ways we can do this. Dead simplest, auto-generating PR descriptions, for example. I've completed a bunch of work. I wanna provide a good description to the people downstream.

I can now go in, I can open a PR, I can just say, "Hey, Copilot, create a summary for this." And it's gonna give me a high level, and then, like, key file, file details.

So that's an easy win that doesn't require any depth. But then we get into more complicated stuff. So about six months ago, we added the ability for Copilot code review.

I can go into an existing pull request, no matter how it was created. I can assign Copilot, set them as an assignee on that pull request the same way that I would set a human as an assignee.

And Copilot asynchronously is gonna come along, take a look at that pull request, and it's gonna start generating comments and code suggestions. Now, these won't be automatically injected. They'll show up, and this is our major philosophy, the same way anything a human would do.

We're still using that same skeleton of all the existing GitHub methodology. So if another human comes along and they make a code suggestion on your PR, it shows up as something you can accept or reject.

Same thing with Copilot. It'll say, "Hey, I don't think that this particular function is as efficient as it could be. Here's a suggested rewrite. Do you want to accept it?"

And you get, as an operator, the choice of whether or not to do it. But the benefit is it can do all of that while you're working on something else, and you just come back half an hour later and look at all its comments, kind of the same way you'd...

It work asynchronously with another developer. Um, the other I would point you to is this new ability to say, "All right, assign an issue directly to Copilot." Now, this sounds dangerous, but again, everything happens in isolation on its own branch, and you wanna be judicious.

You know? There's only so many giant, high-scoped issues that you'd necessarily want to assign to something totally anonymous... uh, autonomous. Uh, but when you are doing something like, "Hey, there's not enough documentation.

Hey, I need to add tests for this. Hey, I wanna add this one function that does XYZ," writing a well-shaped issue the same way you would write a well-shaped prompt, say, "Focus on these.

Do these exact things. These should be the outcomes. This should be the test generated." And then take that issue and try assigning it to Copilot. Copilot will take that.

It'll create a new branch, create a new PR, work autonomously for a while, provide you with what it thinks is a completed PR that's ready to go. And then you get to take a look at that and say either, "This was terrible.

Just destroy the whole PR and try again. I'll refine my issue and reassign it." Or, "Uh, you did one little thing wrong. Put a comment into that PR." Copilot will come back, make some more modifications on the branch, and present you with that set of changes.

Or it's good to go, in which case, boom, run my standard CI/CD pipeline, make sure everything's good, merge it on in.

All right. And I'm just about at time, so I'll leave you with this one last thing, which is MCP servers, right? So MCP is now available in two places.

One is in VS Code itself, where you can go in and you can add an MCP. Go to that URL, github.com/model-context-protocol/servers, and pick ones that are applicable to your workflow, the technologies you're using, the kinds of things you wanna do.

An obvious one I'd point out is GitHub itself has an MCP server. So now you can say things to your agent after you've added it like, "Hey, uh, okay, I love all the work you just did.

Uh, put it on a new branch. Uh, go ahead and commit it with a sensible commit. Uh, and then create a new PR for it and assign so-and-so to approve that PR."

And the agent will just do it for you. You don't have to flip context and come back out to GitHub and work through the web API. Um, the other thing is that this is also now available, for those of you using Copilot Enterprise, to use with the other agentic interactions I just talked about.

So you can actually configure your MCP at the repository level, and then when you're doing something like assigning Copilot to an issue, it can take advantage of those MCP connections right there, and it can reach out to other systems as it's trying to accomplish its work.

I'll leave you with this last slide, which is again there on that URL, gh.io/fair.unlock. Just a bunch of convenience links for training, cheat sheets, that sort of thing. Thanks very much for your time, and I'll be out in the hallway if anyone wants to chat more later. [audience applauds] [upbeat music]