Using Azure Test Plans with Playwright

In 2020, I blogged about associating automated tests with Azure Test Cases. The post had 18 questions, which indicates there is still confusion on how this works, especially how to set it up with Playwright (which was pre-stable release at the time).

In this post, I’ll walk through how to configure both Playwright Test (JavaScript/TypeScript) and Playwright .NET to get test results in Azure Test Plans. Each option uses abstractions built on the Azure DevOps REST API so you don’t have to write additional code to accomplish this.

Why?

Azure Test Plans is a popular service that many teams are using for manual testing. By publishing your automated Playwright tests to the service, you get a couple of benefits:

  1. Traceability. This gives you the option to link your requirements (Azure Boards) to automated tests and the pipeline that ran them. By mapping the two, you can establish the quality of the requirements based on test results. Ideally, a test case is created for each of the acceptance criteria listed for the requirement.
  2. History. Drilling into every pipeline run to see test results over time is a pain. Azure Test Plans allows you to see results through features like the progress report and charts.
  3. Test inventory. By tracking automated AND manual test cases, you can do things like track the status of a test case (not automated, planned to be automated, or automated). This makes it easy to track the progress of automated testing efforts, e.g. how many manual tests have been converted to automated, how many remain, etc.

What are the options?

I’ll show working code examples for both Playwright Test (TypeScript) and Playwright .NET using NUnit. If you’re already sick of reading and want to see them in action, here are some links.

TypeScript: tests, pipeline to run tests, test plan

.NET: tests, build to publish binaries, release to run tests, test plan

Playwright Test (TypeScript)

playwright-azure-reporter is a custom reporter (npm package) that allows you to post test results by annotating your test case name with the Azure test plan ID. The README has instructions for installing the package and adding the reporter to playwright.config.ts

My example project’s config looks like this: playwright.config.ts.

Once that is in place:

  1. Manually create new test cases in Azure Test Plans taking note of the ID (planID in query string of URL)
  2. Add the ID in brackets to the test case title. 444, 445 in this example:

When these tests get run, you will then be able to see the outcome for each test case:

My example pipeline runs these tests for every commit on main and also uses the JUnit reporter to publish results to the pipeline’s Test tab:

Playwright .NET

This option works out of the box but has some caveats and complexity: A Windows runner and a release pipeline are required to use the Visual Studio test platform installer and Visual Studio Test tasks. Also, Visual Studio must be used to associate test cases.

Here is how I set this up in my example project:

  1. Manually create new Azure Test Plans test cases
  2. Use Visual Studio’s test explorer to associate the automated test cases:

This will change the Automation status field on the test case work item to automated:


Once the test cases are configured, we can set up our pipelines to run the tests.

  1. Create a build pipeline that runs dotnet publish (using Windows agent) in order to create an artifact with the Playright binaries: playwright-dotnet.yml
  2. Create a release pipeline referencing the artifact created in the previous step:

3. Add install tasks (that run on Windows agent) for “Visual Studio Test Platform Installer” (prereq for VS Test task), .NET, and Playwright browsers:

4. Add the VS Test task and reference your test plan:

5. Create a new release to run the tests. Example results: Test tab, test plan results.

Summary

Hopefully, you were able to follow my examples to get this set up in your own environment. I’d love to hear feedback on anything I may have missed, new features you’d like to see from the product team at Microsoft, or interesting use cases you have experience with.

Happy testing,

Marcus

13 thoughts on “Using Azure Test Plans with Playwright

  1. Thanks for sharing!

    I have two questions:
    – What Azure License do you need to have for making use of the Testplan?
    – Why do you need the [443] in the title of your test? I don’t really like that.

  2. Hey Marcus,

    Great blog post. I’m a big fan of Azure Test Plans and great to see the integration with Playwright. The link to Azure Test Plans above requires access. I’d like to see your test plan , 444 and 445, with either read access to your URL or screen shots.

    Thank you for sharing!

    Randy

      1. Thank you Marcus, I’ll check it out soon. I’m also curious, will this work with the hybrid approach? GitHub Actions >>> ADO Boards + Test Plans?

      2. Hello,

        Thanks for the post, I have some questions:
        – Would it be possible to add screenshots of the test plans & a test case ?
        – Isn’t there anything more to add, to link a test case to the playwright project ?
        – What about the configurations (ex: for configurationIds) ?

        Again, thank you

  3. Hi There,

    Thanks for the post. I had one question The ‘Azure Token’ mentioned is it ‘Personal Access Token’?

    Rahul.

  4. Thank you so much for sharing this awesome article.
    I have the tests running and reporting to test plans though I have a challenge accessing the ADO token when set as a secret variable, only works when the token is set as a regular variable. Could you kindly share your pipeline YAML file for reference on how you are setting/accessing the token.

    I would also appreciate if you could write an article on setting up the pipelines for the Playwright Test (TypeScript).

Leave a Reply

Your email address will not be published. Required fields are marked *