Beyond Code Coverage: Functionality Testing with Playwright

Talk transcript

1. 0:00

   [upbeat music] Okay.

2. 0:17

   Hi, everyone. Um, my name is Marlene, and I am a senior developer advocate at both Microsoft and GitHub. So I work at-- I work in a group called Core AI, which looks at how developers are using AI across our products.

3. 0:34

   Um, so this is kind of new. To start off today, I wanted to show you some stats about GitHub from GitHub Octoverse, uh, last year's GitHub Octoverse report in twenty twenty-five, which shows data about how developers are using GitHub.

4. 0:50

   What we saw from our report was that more code was added to GitHub last year than ever before. So about a billion commits were pushed to the platform in twenty twenty-five, which is GitHub's most active year ever.

5. 1:08

   Okay? What we know now in twenty twenty-six is that this growth is accelerating. So a couple of-- We haven't actually released any official stats yet, but a couple of days ago, our COO, Kyle Daigle, tweeted that we're seeing about two hundred and seventy-five million commits to the platform every week.

6. 1:31

   And if we extrapolate that over time, we're going to see about fourteen billion commits by the end of the year. So that's fourteen times the amount of growth we saw last year, or commits we saw last year, which, by the way, last year again, was our biggest year ever with a, a billion commits.

7. 1:49

   One thing that we know is that there's a growing share of these commits that are co-authored by AI agents. We haven't released the data yet, but we can actually track and see or, you know, some Claude, for example, cosigns commits, and so does Copilot, but Codex doesn't.

8. 2:06

   But we can also kind of track based off of, uh, some wording in, in the code.

9. 2:13

   I had a question when I saw all of this growth in terms of how much code we were seeing, and that question is: Does AI actually make developers more productive?

10. 2:25

    We're seeing all of this code. Does it actually correlate with productivity?

11. 2:30

    One of the best resources I've seen that tries to answer this question is actually from AI engineer from a talk last year, and this graph shares findings from that talk that's from a Stanford University study of a hundred and twenty thousand developers.

12. 2:47

    And in this study, um, it found that while, yes, AI can make developers more productive, it's actually how the developers are using AI that matters the most. So this graph from the study actually shows us that clean code bases amplify AI gains and AI productivity, while unchecked AI in a code base is going to amplify

13. 3:12

    entropy. To illustrate this point, the speaker from this talk gave a case study example of a company that used AI in an unchecked way in their database. And what you can see is that the number of PRs that the team was pushing out increased, but at the same time, the code quality that this team was, was seeing

14. 3:32

    decreased, and actually, they spent a lot more time reworking that code, refactoring that code. And so overall, though there was effective output increase of like one percent, AI didn't really improve the productivity from this team.

15. 3:49

    So what we learned from this study is that a lot of value that we are wanting to see as developers from AI relies or hinges on us having a clean code base.

16. 4:01

    So for developers that are using AI tools, we wanna focus on things like good test coverage, type coverage, and things like good documentation, modularity, and so on.

17. 4:12

    So I'd actually also argue that we need to start standardizing some practices across our teams and across our industry, and this is something that's a bit of a controversial topic because some people at this conference believe in just closing their eyes and shipping, and that's also okay.

18. 4:27

    But I think in my ideal world, and from this study we've seen, I would recommend standardized practices for keeping a code base clean. So how can developers maintain-- create and maintain clean code?

19. 4:41

    This question is actually not a new question. In our industry over time, we have seen several methods that have tried to make, uh, maintaining a clean code base a central part of their philosophy.

20. 4:54

    One of those approaches that I've actually seen a lot of developers that are doing, uh, agentic coding with coding agents talking about is test-driven development or TDD. Uh, Simon Willison, who's very popular, just recently published a blog post about how he's using this specific flavor of TDD called Red-Green TDD.

21. 5:13

    And here, what happens is a developer-- the first thing that happens is the developer gets an incoming feature request. As soon as they get the, the request, they immediately start by writing a failing test because the feature doesn't exist.

22. 5:27

    After that, the developer focuses on getting the test to pass. And in this green phase, when they're trying to get the test to pass, historically, you should not be focusing on the quality of the code.

23. 5:39

    All you're focusing on is speed and getting the test to go green. So in the past, developers maybe would copy code from, uh, Stack Overflow and so on and get the test passing.

24. 5:49

    But then after that, the final phase of this is the refactor phase. And in this phase, you're just focusing on code quality. So you're taking that code that you made pass and refactoring it so that it follows all the best practices.

25. 6:07

    So not everyone is a fan of TDD, and like many things in this industry, TDD was pronounced dead in two thousand and fourteen. Um, and one of the most common complaints that I've seen on the internet about TDD is that it focuses too much on code coverage with unit tests and that it doesn't actually test the system.

26. 6:28

    So DHH, who created Rails, published this blog post in, in two thousand and fourteen and was kind of talking about this, that is an over-focus on unit tests.

27. 6:39

    And we know that when we over-index on code coverage, there are several issues that come up. One of the issues is that there's a tendency to test implementation details.

28. 6:49

    So, um, take an example like we see on the screen where we are having an order calculation with a discount. If the test is tied directly to a method like calculate, just simply renaming the name of the test, even if the functionality is still fine, is going to break those unit tests.

29. 7:07

    So that's not be- going to be great. But even if we test specifically, uh, the behavior of the system, like the final end result of, of the price we're looking for, um, or we test on something like a stable contract like our API or a module that doesn't change, but we export it, it should survive any refactors

30. 7:27

    of our internal code. I would say that if you're interested in learning more about this and behavior-driven, uh, TDD, I would recommend the talk by Ian Cooper called TDD: Where it All Went Wrong.

31. 7:39

    It's a very good talk. Another thing that we see is that in the age of AI, that, uh, many developers are using AI to generate tests. What they've noticed is that AI sometimes generates self-affirming tests.

32. 7:56

    So while the code coverage test might pass and you- your unit test suite is all green, the behavior of the system is not being validated, and that's where the problem lies.

33. 8:09

    So for the rest of this talk, I'm gonna be focusing on showing you how you can avoid these problems and start to test for functionality using Playwright. Playwright is an open source testing framework that's built by Microsoft, and it automates end-to-end testing in the browser by simulating user interactions.

34. 8:27

    And the link that you see on the screen there is, uh, is the documentation. So Playwright supports a number of different, uh, languages right now, Python, TypeScript, C#. And the example scripts that you can see on the screen is what a test would typically look like.

35. 8:43

    So you have that line that says "page.goto" telling the, um, the script that it starts at the toys pa- page is where we wanna start, and then we're going to look for the placeholder search, and then we're gonna fill that search bar with that, uh, let-- with the word Furby, and that will actually do-- run the search

36. 9:03

    for us automatically in the browser, for example. You can also use headed or headless mode, so you don't necessarily have to look at the browser while your tests are running.

37. 9:13

    You can actually just have them running in the background as well.

38. 9:18

    Um, why? Okay. So going back to that idea of TDD, when we're using Playwright with AI, it actually should speed up the full process of TDD for us. So a lot of developers in the past have really complained about how TDD is slow and that it, it's not effective for teams that want to move fast.

39. 9:38

    But if we have AI, then what happens is that red part and the green part are fast. So we're focusing on getting our agents to generate these behavioral tests, the Playwright tests.

40. 9:51

    Then we're focusing on getting the agent to quickly generate as fast as it can code that's going to make the test pass. And then I would recommend that developers are going to spend the most amount of time, so it grows bigger, on that refactoring stage.

41. 10:04

    So they're spending time looking at the code the agent has generated and making that code better.

42. 10:12

    There's a number of ways you can connect your coding agents today with Playwright. One of those ways is through the Playwright MCP server. You can use the CLI tool if you'd like that instead.

43. 10:23

    Or you can use something, uh, called Playwright Agents. And when you're using Playwright Agents, you'll run the command that you can see on the screen. And once you run that command, it's going to install for you, um, three agent.md files.

44. 10:37

    So the first one is gonna be a planner, second is a generator, and the third is a healer. So the planner will plan which tests to run, the, uh, generator's gonna actually generate the, the tests, and then the healer will fix those tests for you.

45. 10:51

    Okay. So I do want to show you a demo, and I am going to hope the demo gods are smiling today. Uh, so we will-- we'll give this a try.

46. 11:07

    Oh. Oh, no. Okay. Here we go. So I wanna give us a scenario. The scenario is that... Oh, you can't see my--

47. 11:19

    You are only looking at my PowerPoint right now, and I don't know how to stop that. [laughs] Uh, let me close the PowerPoint maybe and see if that will help.

48. 11:32

    Um.

49. 11:34

    What we can do is-

50. 11:34

    And I want to just show this screen.

51. 11:37

    Sorry. Hopefully, they'll give me more time. [laughs]

52. 11:44

    More time.

53. 11:44

    Okay. Perfect. That, that's working as expected. Okay. Perfect. So the scenario that we're gonna imagine today is imagine I'm a developer. I'm working at a toy company called Tailspin Toys.

54. 11:55

    And a few days ago, I got an email from the search product management team, and they asked me to add some new search and filter features to the site.

55. 12:03

    They asked me to add in a search bar with text search for simple searches and Azure AI Search for more complex ones, and they've also asked me to add in a sidebar so customers can filter by category and price.

56. 12:15

    So I'd like for Copilot to help me with this task and also- For us to use this, uh, TDD first style of development. So this is GitHub Copilot CLI.

57. 12:27

    And the first thing that we can do is we're going to try to get the agent to get the information that we saw in that email and bring in the features here into our terminal.

58. 12:39

    And, uh, for this, we're going to use something called Work IQ, which is Microsoft... It's a skill that Microsoft has developed that lets developers connect to the M365 suite.

59. 12:50

    So Outlook, PowerPoint, whatever it is you would like, and to bring that information here into the terminal. So if you're using the M365 suite for work, I can definitely recommend it.

60. 13:00

    And what I will also mention with TDD, in the past when we've done things like unit tests, um, typically people, what would trigger writing a unit test is adding a new method to a class.

61. 13:13

    But actually, in this new world, what we wanna focus on is the behavior. So we wanna focus on a feature. So if a feature request comes, that is what the trigger is for the test to be written.

62. 13:25

    So now we have our list of what needs to be actually developed. And I'm passing in a second, uh, I'm passing in the second prompt, and I'm asking Copilot to help me develop these features using Red-Green TDD to start by writing the Playwright tests that fail for each feature.

63. 13:44

    And I'm telling it not to commit the changes just for the sake of this example. And I do wanna point out that the first thing the agent is gonna do is it's going to start to examine my code base.

64. 13:55

    So it's gonna understand what's in my code base. I have the Playwright MCP server already installed in my CLI, uh, into Co-Copilot CLI, and it knows what it needs to do to create the tests to be able to test for these functions.

65. 14:10

    So the agent is gonna understand what the ca- code base is going to look, uh, look like, and then gonna write the test for it. Uh, this process is actually going to take a while, so in the meantime, I'm gonna switch over to a new tab, and I'm gonna run the command to get Playwright, the Playwright test.

66. 14:30

    So earlier today, I asked, uh, I got the agent to generate those failing tests, and then I got it to do the green phase, where the agent just creates the code to get the test to pass.

67. 14:42

    And then now I'm asking my agent to go ahead and run the Playwright test to actually test for that search bar and filter feature for us. So like I mentioned before, I have the Playwright MCP server already installed.

68. 14:57

    You can see it installed here. And our agent is just going to look for the test file, and if everything works correctly, it's gonna start writing some tests, running some tests.

69. 15:08

    So we see it's opened the correct page. It's typing in different inputs, which is testing for the search bar is working. We saw Furby was correctly found. Simon was correctly found.

70. 15:18

    And now it's clicking buttons, so also testing the category filter is working correctly. Again, my hands are not on the keyboard. This is all Playwright and Copilot, so super cool.

71. 15:29

    And now it's correctly finding all of the toys in the specific price range. So when I run these functionality tests, all I can see actively that, okay, the agent has written this code.

72. 15:42

    The code is working as I expected. The app is working as I expected. So there's so many different ways that you can test your app by functionality, and all of our tests pass.

73. 15:52

    Uh, now, once our tests have passed, that's when I would say we step into the next phase of actually going ahead and, um, and, and running out our, uh, writing our refactor, so refactoring the code the agent has created to generate these tests that pass.

74. 16:09

    So a final thing I will do is I will give you some best practices,

75. 16:16

    um, with Playwright. The first thing I would say is that when Playwright runs those functionality tests, it's going to take screenshots of all the tests that it's run. I've gotten into the practice with adding those screen- screenshots to a PR.

76. 16:32

    So if I've made some changes, I'll add them to a PR. The second thing is that you don't have to run it where it launches the browser like you saw in the example.

77. 16:40

    You can run it in headless mode so it runs in the background. And then a final thing is I would say commit your code, uh, before you actually get it to fix the test, or, you know, commit before it starts to make changes to your code.

78. 16:53

    Because if you don't commit, it might not remember what it, what happened in the past. So that's something to do. And then I would also say to generate one feature, one test per feature as well.

79. 17:06

    As a final note, these are some resources you could take a look at. Ah, I forgot to add the link to the GitHub repo. But all of these slides are gonna be available at that link there.

80. 17:16

    You can check out the documentation, and you can also connect with me on social media as well. So yeah. Thanks everyone. I think that's all the time I have today. [clapping]

81. 17:29

    I think we have two minutes for questions. Does anyone have any questions

82. 17:35

    about this? Yes, I see a question there.

83. 17:38

    Um, like, I, I use Playwright together with Storybook because you need to, like, set up to a certain state. Uh, like, how would you... Like now, this example, e-com website is quite simple and like-

84. 17:49

    Yeah

85. 17:50

    ... easy to run. How would you-- What are tips for more complex apps and admin panels where you have a lot of, uh-

86. 17:59

    Where you have a lot of, uh-

87. 18:00

    State management and-

88. 18:01

    I mean, I think if you have a lot of state management, I would focus on maybe... I would recommend using Playwright agents where it downloads the specific agent.md file because that's going to have some specialized instructions that are better at handling state and things like that.

89. 18:20

    So I found that agents, um, uh, Playwright agents specifically has a lot of good instructions already built into it that should help with that. Another thing that you could do is if you didn't wanna use Playwright for everything, you could also just directly test your APIs.

90. 18:35

    If there's an API available, that's something you could do. Um, so yeah. That's what I would recommend. Thanks. Are there any other questions? Maybe one more question. Uh, uh, I'm not sure.

91. 18:49

    Yeah. [laughs]

92. 18:51

    Um, can Playwright also check, like, different, uh, like, sizes, like from like-

93. 18:56

    Mm-hmm

94. 18:56

    ... desktop size to, like-

95. 18:58

    Yeah.

96. 18:58

    Okay. I heard that.

97. 18:59

    Yes. Yes. It can. It can check your d- mobile versus on desktop. It should just work. Yeah. One more. Yeah.

98. 19:08

    Is it only for browser-based, uh, checking? So if I'm developing a Mac app or a iPhone app.

99. 19:14

    It's browser-based for the moment. Yeah. For the moment, it's only browser-based. Yeah.

100. 19:19

     Okay. I think that is all, all the time I have for today. Thanks, everyone.

101. 19:26

     Thanks. [clapping]

102. 19:26

     Sorry about the-- no link to the GitHub. [outro music]