AI Engineer World's Fair 2024
GitHub Next Explorations
Read the talk
GitHub Next: From Completing Code to Completing Tasks
Factory electrification offers a useful lens for two GitHub Next explorations: predicting related edits and helping developers turn an issue into verified code.
From a talk by Rahul Pandita
Before you start: Basic familiarity with functions, GitHub issues, pull requests, and terminal commands will help you follow the demonstrations.
What changes when the machinery changes?
How much of a workflow exists to accommodate the technology underneath it? That question frames GitHub Next’s search for new ways to develop software. Rahul Pandita describes a team of about 20 researchers, senior developers, and tool builders working outside regular product development and reporting directly to GitHub’s CEO. Its purpose is to explore possibilities, then pass the learning to product teams building tools such as Copilot.
Pandita introduces the problem through Andrew Ng’s analogy of AI as the new electricity. Before factory electrification, a central steam engine drove large shafts, which drove auxiliary shafts, which workers connected to through belts and pulleys. The power system dictated the factory’s layout. Work had to happen where mechanical power could reach it; workers organized themselves around the machinery.
Small electric motors changed that constraint because they retained their efficiency at smaller sizes. Power could be distributed around the factory, making a different floor plan possible. Yet the new layout did not appear automatically with the new motor. In Pandita’s historical framing, roughly forty years separated early electric motors in the 1880s from mainstream adoption in the 1920s. He uses that interval to emphasize experimentation: finding useful applications, improving them, and reducing risk until using the technology becomes normal practice. A new capability creates room for a new workflow; discovering that workflow is separate work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Testing an idea before making it a product
If the right AI development workflow were already known, the efficient move would be to build it. Uncertainty makes rapid prototypes useful instead. GitHub Next puts an idea through progressively broader feedback:
- Build a functional prototype and use it heavily within Next.
- Expand internal use to the rest of GitHub if it survives that first test.
- Release a technical preview to early adopters and learn from their use.
- Consider productization if the evidence continues to support it.
These are gates, not a guaranteed release pipeline. An exploration can be stopped or shelved at any stage when the signals do not justify continuing.
Copilot began as one of these experiments. Other explorations included Copilot for CLI, Copilot Voice, GitHub Blocks, and SpecLang. Some became products, some contributed capabilities to existing products, and others were shelved because the timing or the idea was wrong. Shelving an implementation does not discard what the team learned from it.
The two explorations examined here are Next Edit Suggestions and Copilot Workspace. The recording captures them as active explorations in 2024; the current GitHub Next catalog labels Next Edit Suggestions as Product and Workspace as Completed. The demonstrations below follow the workflow shown in the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Predicting what needs to change beyond the cursor
Ordinary ghost text extends code at the current cursor. That is useful when creating new code, but much development consists of modifying existing code: adding, deleting, and changing lines in several locations. Next Edit Suggestions begins with a broader question: after this edit, what else needs to change? Its unit of assistance is a related edit, potentially somewhere other than the place where the developer is typing.
The Python demonstration follows a chain of dependencies. Pandita adds a parameter, and the system suggests updating the method definition. After that update, it suggests related argument changes. Once the code changes are in place, it identifies documentation that no longer matches the implementation and suggests correcting that too. Adding another parameter repeats the sequence; the visible later suggestion adds a color parameter and changes the greeting.
A small Python example makes the same dependency concrete. If a call changes to greet("Mona", "green"), the definition, behavior, and documentation must agree with it:
python
def greet(name: str, color: str) -> str:
"""Return a greeting that includes the person's favorite color."""
return f"Hello, {name}! Your favorite color is {color}."
print(greet("Mona", "green"))
Here, adding color to the signature alone would leave the intended behavior incomplete. Updating the returned greeting without updating the documentation would leave another inconsistency. The demonstration’s value is the sequence of suggestions across those dependent locations, rather than a longer completion at one cursor position.
Suggest correction
An edit prediction also needs a navigation interface
Predicting a useful change does not settle how to present it. At the time of the talk, the team was still investigating several connected problems:
- Presentation: Is ghost text the right interface for replacing or deleting existing code?
- Navigation: How should a suggestion appear when its location is outside the viewport, or in a file that is not open?
- Accuracy: How should models be fine-tuned for this particular task so that suggestions are relevant enough to help?
The location problem expands the interaction beyond autocomplete: the developer must understand both the proposed change and where it belongs.
Internal dogfooding remained the release gate. Pandita anticipated an IDE appearance in the following months, either through a standalone Next technical preview or within an existing Next product, conditional on those investigations producing a satisfactory experience. That was a forecast, not an announcement that the demonstrated behavior was already available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expanding assistance to the development inner loop
Related edits still operate at the level of code. Moving to task completion means helping with the reasoning that precedes implementation. A developer starts with an assignment, works out what it requires, examines the repository’s languages and frameworks, and determines how new behavior can fit the existing code. The desired result then has to become a concrete set of file changes.
| Stage | Question to resolve | Artifact |
|---|---|---|
| Task | What am I trying to accomplish? | Defined problem |
| Specification | What exists, and what should exist? | Current and desired states |
| Plan | Which files need which changes? | Implementation plan |
| Implementation | How do those changes become code? | Modified files |
Code is one artifact in this loop, not the entire loop. Copilot Workspace aims to make AI useful across these stages rather than limiting assistance to the final act of writing code.
Workspace itself emerged from smaller explorations. SpecLang investigated natural language as a functional specification for programs. A separate effort investigated improving completions by re-prompting models with runtime information. Those experiments, combined with user feedback, became a broader environment for completing tasks.
Developer interviews exposed several requirements. Getting started on an assigned issue was difficult. Trusting AI output was another obstacle. Developers also wanted to retain the problem-solving part of their work: AI should act as a thought partner they could collaborate with, while they remained in control.
Workspace maps those needs to concrete interactions. A one-click proposal provides a starting point. A built-in runtime lets the developer check generated code. Editable intermediate artifacts allow course corrections, and collaboration can happen through a GitHub pull request or by sharing the workspace itself with colleagues.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a missing Rust example to a file-level plan
The Workspace demonstration uses Monaspace, another GitHub exploration: a family of coding fonts with a website demonstrating their features. Its playground shows syntax highlighting across languages, but Rust is missing. The task is specific and observable: add a Rust example to that playground.
Pandita starts from a GitHub issue requesting that addition and opens it in Workspace. The interface presents a specification divided into Current and Proposed states. The specification, plan, and implementation shown during this walkthrough are cached to keep the demonstration within its allotted time. Pandita describes uncached generation as taking minutes rather than hours; the fast transitions on screen are not a latency measurement.
The current specification identifies missing Rust support. The proposed specification describes the target: add the syntax-highlighting package, include it in package.json, and update the related files. This gives the developer something to assess before asking for code. Does the system understand the gap, and does its proposed destination match the issue?
The generated plan identifies three files to update. Pandita reviews that plan, judges it to be at a useful level of detail, and requests implementation. Workspace then displays the resulting file changes. The progression matters: an issue becomes a description of the desired behavior, that description becomes a file-level plan, and the reviewed plan becomes an implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Correcting the proposal and checking the result
The generated artifacts remain editable. Pandita points to instructions for adding Rust to the language mappings and adding code documentation, then shows that chat can also target a particular location for revision. He does not run that regeneration path during the presentation because it would leave the cached sequence and require another iteration. The interaction is available for steering the work; the recording does not demonstrate its regenerated result.
Verification moves the task from plausible file changes to observable behavior. In the integrated terminal, Pandita asks to install and run the repository. The first synthesized command is unsuitable. He edits it to the command he wants before executing it in an actual terminal. This correction is part of the workflow: natural language proposes a command, but the developer still inspects what will run.
The terminal runs the corrected command, and Pandita reports that the code compiles. He then opens a live preview, which takes longer to load than its displayed estimate suggests. Compilation provides one check; seeing the requested change in the application provides another.
Why generate a simple npm command instead of typing it? On a desktop, direct entry may be easier. On a phone, command syntax and symbols are more tedious to enter through a mobile keyboard. Natural-language command synthesis addresses that input constraint, allowing a developer to work from a mobile device without manually composing every shell command. The preview eventually loads with a Rust example, completing the demonstrated path from issue to a running preview—not to a production deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learning and review become the next workflow questions
Workspace’s continuing improvements include runtime support for synthesizing terminal commands and faster file completions. Beyond Workspace, Next is investigating how developers learn with AI and how code review should change if AI writes most of the code. The closing research slide also places editors, terminals, planning, and tracking within that broader agenda. Some investigations may reach technical preview; others may be stopped when they fail to find a useful direction.
That returns the discussion to the factory floor. The opportunity is larger than inserting a new source of power into an unchanged arrangement of work. It includes discovering where developers should inspect a proposal, revise a plan, navigate to a related edit, or run a check. Pandita closes by inviting collective experimentation to shorten the distance between AI’s capabilities and everyday workflows that make those capabilities useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
GitHub Next's exploration of maintaining program behavior in natural-language specifications while a toolchain manages implementation.
Five compatible coding typefaces, with font downloads, installation instructions and typography features.
Further reading
Paul A. David's May 1990 study of factory electrification, organizational change and delayed productivity gains.
Read the complete timestamped transcript
- 0:00
[on-hold music] My name is Rahul Pandita, and I am a researcher at GitHub Next.
- 0:19
Uh, and today we're gonna talk about some of the GitHub Next explorations. Uh, now, before we begin, who among you have heard of GitHub Next?
- 0:28
Oh, cool. Quite a few of you. That will make it go much easier and much faster. All right. For those of you who don't know us, we are, uh, about 20 bunch of researchers, senior-ish level, uh, developers, and mostly code build-- uh, tool builders, uh, who work outside of the regular product, uh, and report directly to our
- 0:50
CEO, uh, and that's by design. And, and our goal is to explore the future of software engineering like you all are doing in, in your day-to-day jobs. And the-- and, and the reason for exploring that is that, like, once we do our explorations, we toss it on and we pass it on our learnings to the product and
- 1:10
development teams so that they can build really compelling products like the Copilot that you all have used, hopefully at some point of time. As an aside, uh, for people who are following us on Twitter, uh, I don't look anything like my picture over here.
- 1:24
I'm the one in the green background. But we do have Devin in our team. He's not an automated AI. He's a very real person, and he looks exactly like the person on the top right corner on that slide.
- 1:35
All right. Since we have gotten that out of the way, let's talk about-- let's get back to the future of software engineering with regards to gen, gen AI. So here's what Andrew Ng, uh, who single-handedly trained a whole generation of machine learning engineers, uh, has to say about, uh, AI.
- 1:55
That it's just as electricity. It's the new electricity. It's gonna transform the software development and almost every other field, just like electricity did a hundred years ago.
- 2:06
So what does that mean? Here's a picture of what a manufacturing facility looked like before electrification. There used to be a giant, uh, mostly coal-powered steam turbine or steam engine located centrally, which used to turn these giant, uh, giant shafts, which will turn these auxiliary shafts, so forth and so on, and individual workers would connect to these
- 2:29
shafts using the belt and pulley system, right? And, and these engines were, like, really, really huge. So, so it was the workers. The whole architecture of the factory were designed around this steam engine.
- 2:41
And, and the whole workflow was around the steam engine. And, and it was the workers who were working around the technology rather than the technology working for people, right?
- 2:54
And along in nineteen-- uh, eighteen eighty came these electric motors, uh, and, and they had the potential to revolutionize, uh, the, the manufacturing sector. Why? Because unlike steam engines or steam, uh, motors, they retained their efficiency when they were smaller, right?
- 3:11
Even-- So, so you could basically redesign the entire factory floor plan. So we would think that, wow, this is great, and everyone would jump on this. But it was not until nine- nineteen twenties where these became the mainstream.
- 3:26
So early eighteen eighties to late nineteen twenties. What was happening about these forty years? What was happening was exploration and experimentation. People were trying to figure out, uh, how to use this technology, how to make it better, how to de-risk it to a point that,
- 3:45
that the use of this technology becomes the norm rather than the exception. And that's what we do at GitHub Next, right? Our charter is to explore the future of software engineering, and with the emphasis on the word explore, right?
- 3:59
Because if we knew what the future of software engineering in context of AI looks like, we would just build it. That's more efficient. But unfortunately, we do not. So what we have to resort to is exploration.
- 4:10
We just try out different things, rapidly prototype, experiment, and figure out whether something works or not. And if it works, then we put it out in front of our customers or end, end users, and we learn from them, and then we finally transform into a product.
- 4:25
Oftentimes, an idea begins as inside our Next, uh, as a functional prototype, which goes through heavy dogfooding inside the Next team. If it survives that, then we move on to the next level of dogfooding that is inside the company.
- 4:39
If it survives that, then we move on to the next level, which is releasing it as a tech preview, uh, to other early adopters. We learn from that. If it survives that, then it may have a chance to become a product like that, a product in the future.
- 4:52
And we can kill or we can shelve any of these exploration at any point of time if we are not getting the right signal so that we can explore other areas.
- 5:01
We did that with the Copilot. So yes, Copilot started off as a Next experiment, and since that we have created many other experiments like Copilot for CLI, Copilot Voice, GitHub Blocks, Spec Lang, so forth and so on.
- 5:15
A lot of these have transformed into a product of their own, so you can see some of them as, uh, GitHub product offerings. A lot of them have been absorbed into existing products, uh, and, and you will see them as a part of the existing products.
- 5:30
And a significant number of them have been shelved. We've learned what we learned from those experiments and figured out that this is not the right time for that kind of exploration, or the exploration itself was flawed.
- 5:39
So-- But we learned from them, and we will keep that learning and use that in our next, uh, Next explorations.
- 5:48
So that was an overview of GitHub Next, and today I'm gonna talk about two sp- uh, specific explorations. Uh, one is the Next Edit Suggestions in the Copilot Workspace that are currently active, uh, from, from GitHub Next perspective.
- 6:01
And, uh, specifically, I'm talk-- I'm gonna talk about what their motivations was and, and how they came to be and what are the future plans for that.
- 6:09
So first off, uh, Copilot Next Edit Suggestions, right? So what if-- It started off with this question: What if ghost text could be more intelligent, right? So we all know what Copilot does.
- 6:22
Uh, it provides you the code completions in your current context, right? While it's, like, really, really good at creating new code, but that's not what we all do, right?
- 6:33
We, we, we almost always edit existing code, which involves, uh, editing, adding, deleting lines at multiple locations in a program, right? What if ghost text was good at that as well?
- 6:46
And that's what this exploration is. We call it Next Edit Suggestion, which provides you suggestions not only at the current cursor level, but provides you suggestions what else needs to change in a program.
- 6:57
But enough talking. Let's jump onto a demo.
- 7:02
Right. Here, I am going to add this parameter in this Python program, and the next edit suggestion automatically picks it up and says that, "Hey, you need to update your method definition."
- 7:13
Once we update the method definition, it says that, "Hey, you need to add these, uh, these, these arguments." And once that has been updated, then it will go back and say, "Hey, uh, now the code document, uh, is not, is not in line with what the code is actually doing," and it goes ahead and edits that and
- 7:29
updates that as well. And the same thing repeats when I add one more, uh, parameter.
- 7:39
All right. So that was Copilot Next, uh, Edit Suggestions experiment. Uh, we are-- we're still not ready yet. We are still, uh, experimenting with a bunch of other stuff like, you know, uh, is the ghost text completion the right, uh, modality for it, or do we need to figure out a bit differ-different way of presenting those suggestions?
- 8:01
What if the location of the next edit is not visible in the current viewport, or what if the location is in a file that is not even open in an editor?
- 8:11
Uh, most importantly, we are also working on fine-tuning the models specifically for this use case, the idea being that, like, if we want the next edit suggestions to be, uh, accurate, and we want it to be very useful, then the suggestions needs to be on point.
- 8:25
And once we are done with these further sub-explorations, and we feel that it has gotten through our internal dogfooding standard, Next Edit Suggestions would be coming out either as a standalone, uh, tech preview from Next or as a part of an existing Next product, uh, sometime in your IDE, uh, in next few months.
- 8:45
All right. So there was code completions. But let's move from the code completions to the task completions land. Uh, why do we ask? Why, why move from the task completions?
- 8:55
It just turns out, uh, that while code is, like, an important artifact, uh, that comes out of software development, but it's not the only artifact. Software development involves this inner loop where you begin with a task.
- 9:08
The idea is, like, what am I supposed to do? Uh, how am I-- Uh, what, what is the specific thing that I'm trying to do? And followed by, uh, how do I go about doing that thing?
- 9:18
What are the frameworks that are at my disposal? What are the programming languages that are at, that are at my disposal? What are the kind of, uh, what, what is the existing code that's there?
- 9:27
What-- How do I write a new code that is consistent with those codes? So that's becomes a sort of a specification. And once you understand where you are, then you sort of try to decide, like, where am I going with it?
- 9:37
Like, how does the final product look like? Once you have zeroed in on that, then you go about what specific file changes do I need to make to, to, to get to that final product, and that sort of becomes a plan.
- 9:49
And once you get to the plan, then you go to the implementation part. And that forms this loop of software development, and we call it inner loop, and we would like the AI to be helpful in all those aspects of that inner loop, and that's why we built Copilot Workspace.
- 10:04
And mind you, like all Next explorations, it did not start as Copilot Workspace. It started as individual explorations. For instance, we started to figure out, can we use natural language to, as a functional specification of programs?
- 10:16
So there is a Spec Lang exploration. We, in parallel, we were trying to figure out if we can improve the code completions by providing, re-prompting the model with the runtime information.
- 10:26
And all of those things combined, and with the user feedback, combined into this one bigger exploration called Copilot Workspace. And we were also talking to our users. Like, we, we wanted to talk to developers, and we wanted to ask that, "Hey, we are building this thing.
- 10:40
How would you like AI to support you? What are your major pain points?" And one-- And a few things became very, very clear while talking to our users, right?
- 10:49
So first thing is that the most, uh, difficulty that people faced was getting started on a task. Like, how do I... I know that a issue is assigned to me, how do I get started on it?
- 10:59
Followed by, how do I trust the output of the AI? I don't trust it. And more importantly, they figured out that problem-solving is what software development is about, and I-- they would like to retain that problem-solving, uh, aspects of it.
- 11:11
And they would like the help of AI in the form of a thought partner or a sparring partner or a second brain which they can collaborate with to solve a problem.
- 11:20
And lastly and most importantly, they would like to retain control. Developers are in control, not the other way around. And with this feedback,
- 11:30
we built Copilot Workspace. So what is it? It allows you to s-- It simplifies getting started, so one-click proposal on, on your tasks. It has a built-in runtime that allows you to ver-quickly verify what the, the code that has been provided by the AI.
- 11:46
It has an environment which is built for iteration, so if you feel that AI is going in the wrong direction, you can just go and quickly correct it. And most importantly, it is designed for collaboration, so you can just share, uh, your code or your work as a part of the GitHub pull request, or you can share
- 12:00
your work or share your workspace with your colleagues if you're not comfortable with it. But let's-- Enough talking. Let's just get into a demo about it, right? So this is Monaspace, which is another GitHub exploration.
- 12:14
So if we are to write code, let's write code in style, and these are the four f-- uh, is a family of Monaspace fonts that has been released by GitHub.
- 12:21
And, and this is a website that outlines a bunch of, uh, features of these fonts.
- 12:28
And over here, somewhere over here, is this playground which says that, uh, h- that here are how the syntax highlighting looks across different languages. Notice that it is missing Rust, and Rust appears to be the next cool thing that all the cool kids are doing, so we would like to update this Monaspace website with a Rust example
- 12:46
as well. So how do I get started?
- 12:50
So I've created this issue, or somebody has created this issue, it just happens to be me for the purpose of this demo, that I would like to create... I would like to add a Rust example to the font play-playground.
- 13:02
And I can just click this button over here, and it will open the Copilot Workspace for me. And through the magic of caching, you can see that it quickly generates the specification and propo-- uh, current specification and the proposed specification.
- 13:18
Uh, why caching? Uh, because I had to finish this demo in time. But trust me, it's not a matter of hours, it does happen in a matter of minutes, right?
- 13:26
And, and for the, those of you who are interested, I would like to do a live demo for you in the Microsoft booth after this task. All right. So what is the current specification?
- 13:34
It just goes and figures out, does the website have this playground that contains a, a Rust package? And it says it doesn't, and it goes to the target state.
- 13:43
Would, where would the target, what does the target state look like? And it would say that, yes, the website will have the specific package for syntax highlighting, the website will have, uh, this package in, in package.json, and then I will update a bunch of other files.
- 13:58
It looks nice, and I'll go and generate a plan for it. Again, through the magic of caching, a plan has been generated, and it will tell you that these three files, these three files need to be updated.
- 14:07
And I will... It, it appears that this seems to be at the right level of modality, then I will go ahead and implement it. And yes, magic of caching again.
- 14:16
What we see is the files that are over here. Uh,
- 14:21
now, this seems nice and, but what about the iterate part? What you can do is, at any given point or time, if you feel that something is not right, you can just go ahead and say that, "Okay, add Rust to the language mappings," and say, "Add code
- 14:38
documentation." And you can edit at any given point of time. And what you can also do is that you can edit via chat over here, and you can say that, "Hey, I want to edit this one specific location.
- 14:49
How do I go about in doing this?" I'm not gonna do this because it's gonna go through the whole iteration loop, and then the illusion of the caching will break, and it will take a lot of time.
- 14:59
But I would like to do, show that in live demos afterwards. But how do I trust whether this is in fact the right thing? So I will open up this integrated terminal and I will say, uh,
- 15:12
install and run this repo. All right. So what's gonna happen is, uh, that a suggestion is gonna load, and apparently not the right thing. But I can quickly go and edit it and say that, all right, this is the command that I'm specifically looking for, and I can go and run.
- 15:35
Now, this will run this command in an actual terminal, and we'll see the output in some, some point of time. Uh, and, and you can see that actually this, this code does compile.
- 15:45
What we also have is a preview. What we can do is open the live preview. I don't trust it. It's, uh, it will say that it's just gonna be a second, but it takes longer than that while that loads.
- 15:58
What are the other things? Uh, one of the things that you would say is that, "Hey, you wrote a very simple command at the terminal. You said NPM. You could actually type that thing in the terminal."
- 16:05
And yes, you're right, I can type that thing. But think about that in a mobile setting, when you can open Copilot Workspace in a mobile plat-- uh, in, in on your phone.
- 16:13
It becomes very tedious to type those symbols, right? And if you have used the, the mobile keyboard, it's not very useful for that. So what I'm gonna... So, so that's why we use this natural language way of, uh, writing these commands in the terminal, uh, so that it can help you when you're on the go.
- 16:31
It can c-- it synthesizes commands. And hopefully the website has loaded, and there is a Rust example, right? Cool.
- 16:42
That was a demo and- [audience applauding] Thank you. We are working... We are not stopping there. We are working on bunch of these improvements, and I can talk about these improvements, uh, on one-on-one basis with you.
- 16:53
And, uh, and, and you already saw some of the improvements, like the runtime support to synthesize the terminal commands and, and faster file completions using, uh, to, to make the Copilot Workspace better.
- 17:05
But there are other next explorations that are also active, like how do we rethink the developer learning with AI, and how does the code review change if majority of the code that, that is now being written is by AI?
- 17:15
So what does that mean? And some of these explorations will, will work out, and some of these exploration we will see as tech previews, and some of these exploration will kill because we don't know where they're going.
- 17:24
So in summary, I'm saying that we do not know what the future of AI is, but what we know is explorations is the way to get it. And with all your help, we'll jointly explore the space so that we don't have to wait, like electricity, we don't have to wait for forty years to get to a place
- 17:40
where, to, to get to a place with software development where we enjoy the benefits of AI. You have been a lovely audience. That is my time. I really appreciate you.
- 17:49
And if you have more questions, if you want to have live demos, I'm available in the Microsoft booth, uh, in like two salons over that side. Thank you so much. [upbeat music]