GitHub Next Explorations

Talk transcript

1. 0:00

   [on-hold music] My name is Rahul Pandita, and I am a researcher at GitHub Next.

2. 0:19

   Uh, and today we're gonna talk about some of the GitHub Next explorations. Uh, now, before we begin, who among you have heard of GitHub Next?

3. 0:28

   Oh, cool. Quite a few of you. That will make it go much easier and much faster. All right. For those of you who don't know us, we are, uh, about 20 bunch of researchers, senior-ish level, uh, developers, and mostly code build-- uh, tool builders, uh, who work outside of the regular product, uh, and report directly to our

4. 0:50

   CEO, uh, and that's by design. And, and our goal is to explore the future of software engineering like you all are doing in, in your day-to-day jobs. And the-- and, and the reason for exploring that is that, like, once we do our explorations, we toss it on and we pass it on our learnings to the product and

5. 1:10

   development teams so that they can build really compelling products like the Copilot that you all have used, hopefully at some point of time. As an aside, uh, for people who are following us on Twitter, uh, I don't look anything like my picture over here.

6. 1:24

   I'm the one in the green background. But we do have Devin in our team. He's not an automated AI. He's a very real person, and he looks exactly like the person on the top right corner on that slide.

7. 1:35

   All right. Since we have gotten that out of the way, let's talk about-- let's get back to the future of software engineering with regards to gen, gen AI. So here's what Andrew Ng, uh, who single-handedly trained a whole generation of machine learning engineers, uh, has to say about, uh, AI.

8. 1:55

   That it's just as electricity. It's the new electricity. It's gonna transform the software development and almost every other field, just like electricity did a hundred years ago.

9. 2:06

   So what does that mean? Here's a picture of what a manufacturing facility looked like before electrification. There used to be a giant, uh, mostly coal-powered steam turbine or steam engine located centrally, which used to turn these giant, uh, giant shafts, which will turn these auxiliary shafts, so forth and so on, and individual workers would connect to these

10. 2:29

    shafts using the belt and pulley system, right? And, and these engines were, like, really, really huge. So, so it was the workers. The whole architecture of the factory were designed around this steam engine.

11. 2:41

    And, and the whole workflow was around the steam engine. And, and it was the workers who were working around the technology rather than the technology working for people, right?

12. 2:54

    And along in nineteen-- uh, eighteen eighty came these electric motors, uh, and, and they had the potential to revolutionize, uh, the, the manufacturing sector. Why? Because unlike steam engines or steam, uh, motors, they retained their efficiency when they were smaller, right?

13. 3:11

    Even-- So, so you could basically redesign the entire factory floor plan. So we would think that, wow, this is great, and everyone would jump on this. But it was not until nine- nineteen twenties where these became the mainstream.

14. 3:26

    So early eighteen eighties to late nineteen twenties. What was happening about these forty years? What was happening was exploration and experimentation. People were trying to figure out, uh, how to use this technology, how to make it better, how to de-risk it to a point that,

15. 3:45

    that the use of this technology becomes the norm rather than the exception. And that's what we do at GitHub Next, right? Our charter is to explore the future of software engineering, and with the emphasis on the word explore, right?

16. 3:59

    Because if we knew what the future of software engineering in context of AI looks like, we would just build it. That's more efficient. But unfortunately, we do not. So what we have to resort to is exploration.

17. 4:10

    We just try out different things, rapidly prototype, experiment, and figure out whether something works or not. And if it works, then we put it out in front of our customers or end, end users, and we learn from them, and then we finally transform into a product.

18. 4:25

    Oftentimes, an idea begins as inside our Next, uh, as a functional prototype, which goes through heavy dogfooding inside the Next team. If it survives that, then we move on to the next level of dogfooding that is inside the company.

19. 4:39

    If it survives that, then we move on to the next level, which is releasing it as a tech preview, uh, to other early adopters. We learn from that. If it survives that, then it may have a chance to become a product like that, a product in the future.

20. 4:52

    And we can kill or we can shelve any of these exploration at any point of time if we are not getting the right signal so that we can explore other areas.

21. 5:01

    We did that with the Copilot. So yes, Copilot started off as a Next experiment, and since that we have created many other experiments like Copilot for CLI, Copilot Voice, GitHub Blocks, Spec Lang, so forth and so on.

22. 5:15

    A lot of these have transformed into a product of their own, so you can see some of them as, uh, GitHub product offerings. A lot of them have been absorbed into existing products, uh, and, and you will see them as a part of the existing products.

23. 5:30

    And a significant number of them have been shelved. We've learned what we learned from those experiments and figured out that this is not the right time for that kind of exploration, or the exploration itself was flawed.

24. 5:39

    So-- But we learned from them, and we will keep that learning and use that in our next, uh, Next explorations.

25. 5:48

    So that was an overview of GitHub Next, and today I'm gonna talk about two sp- uh, specific explorations. Uh, one is the Next Edit Suggestions in the Copilot Workspace that are currently active, uh, from, from GitHub Next perspective.

26. 6:01

    And, uh, specifically, I'm talk-- I'm gonna talk about what their motivations was and, and how they came to be and what are the future plans for that.

27. 6:09

    So first off, uh, Copilot Next Edit Suggestions, right? So what if-- It started off with this question: What if ghost text could be more intelligent, right? So we all know what Copilot does.

28. 6:22

    Uh, it provides you the code completions in your current context, right? While it's, like, really, really good at creating new code, but that's not what we all do, right?

29. 6:33

    We, we, we almost always edit existing code, which involves, uh, editing, adding, deleting lines at multiple locations in a program, right? What if ghost text was good at that as well?

30. 6:46

    And that's what this exploration is. We call it Next Edit Suggestion, which provides you suggestions not only at the current cursor level, but provides you suggestions what else needs to change in a program.

31. 6:57

    But enough talking. Let's jump onto a demo.

32. 7:02

    Right. Here, I am going to add this parameter in this Python program, and the next edit suggestion automatically picks it up and says that, "Hey, you need to update your method definition."

33. 7:13

    Once we update the method definition, it says that, "Hey, you need to add these, uh, these, these arguments." And once that has been updated, then it will go back and say, "Hey, uh, now the code document, uh, is not, is not in line with what the code is actually doing," and it goes ahead and edits that and

34. 7:29

    updates that as well. And the same thing repeats when I add one more, uh, parameter.

35. 7:39

    All right. So that was Copilot Next, uh, Edit Suggestions experiment. Uh, we are-- we're still not ready yet. We are still, uh, experimenting with a bunch of other stuff like, you know, uh, is the ghost text completion the right, uh, modality for it, or do we need to figure out a bit differ-different way of presenting those suggestions?

36. 8:01

    What if the location of the next edit is not visible in the current viewport, or what if the location is in a file that is not even open in an editor?

37. 8:11

    Uh, most importantly, we are also working on fine-tuning the models specifically for this use case, the idea being that, like, if we want the next edit suggestions to be, uh, accurate, and we want it to be very useful, then the suggestions needs to be on point.

38. 8:25

    And once we are done with these further sub-explorations, and we feel that it has gotten through our internal dogfooding standard, Next Edit Suggestions would be coming out either as a standalone, uh, tech preview from Next or as a part of an existing Next product, uh, sometime in your IDE, uh, in next few months.

39. 8:45

    All right. So there was code completions. But let's move from the code completions to the task completions land. Uh, why do we ask? Why, why move from the task completions?

40. 8:55

    It just turns out, uh, that while code is, like, an important artifact, uh, that comes out of software development, but it's not the only artifact. Software development involves this inner loop where you begin with a task.

41. 9:08

    The idea is, like, what am I supposed to do? Uh, how am I-- Uh, what, what is the specific thing that I'm trying to do? And followed by, uh, how do I go about doing that thing?

42. 9:18

    What are the frameworks that are at my disposal? What are the programming languages that are at, that are at my disposal? What are the kind of, uh, what, what is the existing code that's there?

43. 9:27

    What-- How do I write a new code that is consistent with those codes? So that's becomes a sort of a specification. And once you understand where you are, then you sort of try to decide, like, where am I going with it?

44. 9:37

    Like, how does the final product look like? Once you have zeroed in on that, then you go about what specific file changes do I need to make to, to, to get to that final product, and that sort of becomes a plan.

45. 9:49

    And once you get to the plan, then you go to the implementation part. And that forms this loop of software development, and we call it inner loop, and we would like the AI to be helpful in all those aspects of that inner loop, and that's why we built Copilot Workspace.

46. 10:04

    And mind you, like all Next explorations, it did not start as Copilot Workspace. It started as individual explorations. For instance, we started to figure out, can we use natural language to, as a functional specification of programs?

47. 10:16

    So there is a Spec Lang exploration. We, in parallel, we were trying to figure out if we can improve the code completions by providing, re-prompting the model with the runtime information.

48. 10:26

    And all of those things combined, and with the user feedback, combined into this one bigger exploration called Copilot Workspace. And we were also talking to our users. Like, we, we wanted to talk to developers, and we wanted to ask that, "Hey, we are building this thing.

49. 10:40

    How would you like AI to support you? What are your major pain points?" And one-- And a few things became very, very clear while talking to our users, right?

50. 10:49

    So first thing is that the most, uh, difficulty that people faced was getting started on a task. Like, how do I... I know that a issue is assigned to me, how do I get started on it?

51. 10:59

    Followed by, how do I trust the output of the AI? I don't trust it. And more importantly, they figured out that problem-solving is what software development is about, and I-- they would like to retain that problem-solving, uh, aspects of it.

52. 11:11

    And they would like the help of AI in the form of a thought partner or a sparring partner or a second brain which they can collaborate with to solve a problem.

53. 11:20

    And lastly and most importantly, they would like to retain control. Developers are in control, not the other way around. And with this feedback,

54. 11:30

    we built Copilot Workspace. So what is it? It allows you to s-- It simplifies getting started, so one-click proposal on, on your tasks. It has a built-in runtime that allows you to ver-quickly verify what the, the code that has been provided by the AI.

55. 11:46

    It has an environment which is built for iteration, so if you feel that AI is going in the wrong direction, you can just go and quickly correct it. And most importantly, it is designed for collaboration, so you can just share, uh, your code or your work as a part of the GitHub pull request, or you can share

56. 12:00

    your work or share your workspace with your colleagues if you're not comfortable with it. But let's-- Enough talking. Let's just get into a demo about it, right? So this is Monaspace, which is another GitHub exploration.

57. 12:14

    So if we are to write code, let's write code in style, and these are the four f-- uh, is a family of Monaspace fonts that has been released by GitHub.

58. 12:21

    And, and this is a website that outlines a bunch of, uh, features of these fonts.

59. 12:28

    And over here, somewhere over here, is this playground which says that, uh, h- that here are how the syntax highlighting looks across different languages. Notice that it is missing Rust, and Rust appears to be the next cool thing that all the cool kids are doing, so we would like to update this Monaspace website with a Rust example

60. 12:46

    as well. So how do I get started?

61. 12:50

    So I've created this issue, or somebody has created this issue, it just happens to be me for the purpose of this demo, that I would like to create... I would like to add a Rust example to the font play-playground.

62. 13:02

    And I can just click this button over here, and it will open the Copilot Workspace for me. And through the magic of caching, you can see that it quickly generates the specification and propo-- uh, current specification and the proposed specification.

63. 13:18

    Uh, why caching? Uh, because I had to finish this demo in time. But trust me, it's not a matter of hours, it does happen in a matter of minutes, right?

64. 13:26

    And, and for the, those of you who are interested, I would like to do a live demo for you in the Microsoft booth after this task. All right. So what is the current specification?

65. 13:34

    It just goes and figures out, does the website have this playground that contains a, a Rust package? And it says it doesn't, and it goes to the target state.

66. 13:43

    Would, where would the target, what does the target state look like? And it would say that, yes, the website will have the specific package for syntax highlighting, the website will have, uh, this package in, in package.json, and then I will update a bunch of other files.

67. 13:58

    It looks nice, and I'll go and generate a plan for it. Again, through the magic of caching, a plan has been generated, and it will tell you that these three files, these three files need to be updated.

68. 14:07

    And I will... It, it appears that this seems to be at the right level of modality, then I will go ahead and implement it. And yes, magic of caching again.

69. 14:16

    What we see is the files that are over here. Uh,

70. 14:21

    now, this seems nice and, but what about the iterate part? What you can do is, at any given point or time, if you feel that something is not right, you can just go ahead and say that, "Okay, add Rust to the language mappings," and say, "Add code

71. 14:38

    documentation." And you can edit at any given point of time. And what you can also do is that you can edit via chat over here, and you can say that, "Hey, I want to edit this one specific location.

72. 14:49

    How do I go about in doing this?" I'm not gonna do this because it's gonna go through the whole iteration loop, and then the illusion of the caching will break, and it will take a lot of time.

73. 14:59

    But I would like to do, show that in live demos afterwards. But how do I trust whether this is in fact the right thing? So I will open up this integrated terminal and I will say, uh,

74. 15:12

    install and run this repo. All right. So what's gonna happen is, uh, that a suggestion is gonna load, and apparently not the right thing. But I can quickly go and edit it and say that, all right, this is the command that I'm specifically looking for, and I can go and run.

75. 15:35

    Now, this will run this command in an actual terminal, and we'll see the output in some, some point of time. Uh, and, and you can see that actually this, this code does compile.

76. 15:45

    What we also have is a preview. What we can do is open the live preview. I don't trust it. It's, uh, it will say that it's just gonna be a second, but it takes longer than that while that loads.

77. 15:58

    What are the other things? Uh, one of the things that you would say is that, "Hey, you wrote a very simple command at the terminal. You said NPM. You could actually type that thing in the terminal."

78. 16:05

    And yes, you're right, I can type that thing. But think about that in a mobile setting, when you can open Copilot Workspace in a mobile plat-- uh, in, in on your phone.

79. 16:13

    It becomes very tedious to type those symbols, right? And if you have used the, the mobile keyboard, it's not very useful for that. So what I'm gonna... So, so that's why we use this natural language way of, uh, writing these commands in the terminal, uh, so that it can help you when you're on the go.

80. 16:31

    It can c-- it synthesizes commands. And hopefully the website has loaded, and there is a Rust example, right? Cool.

81. 16:42

    That was a demo and- [audience applauding] Thank you. We are working... We are not stopping there. We are working on bunch of these improvements, and I can talk about these improvements, uh, on one-on-one basis with you.

82. 16:53

    And, uh, and, and you already saw some of the improvements, like the runtime support to synthesize the terminal commands and, and faster file completions using, uh, to, to make the Copilot Workspace better.

83. 17:05

    But there are other next explorations that are also active, like how do we rethink the developer learning with AI, and how does the code review change if majority of the code that, that is now being written is by AI?

84. 17:15

    So what does that mean? And some of these explorations will, will work out, and some of these exploration we will see as tech previews, and some of these exploration will kill because we don't know where they're going.

85. 17:24

    So in summary, I'm saying that we do not know what the future of AI is, but what we know is explorations is the way to get it. And with all your help, we'll jointly explore the space so that we don't have to wait, like electricity, we don't have to wait for forty years to get to a place

86. 17:40

    where, to, to get to a place with software development where we enjoy the benefits of AI. You have been a lovely audience. That is my time. I really appreciate you.

87. 17:49

    And if you have more questions, if you want to have live demos, I'm available in the Microsoft booth, uh, in like two salons over that side. Thank you so much. [upbeat music]