So I wired together GitHub Copilot, Work IQ, and Azure DevOps to pull meeting transcripts, extract action items, and draft work items for me to review before anything gets created or modified.

TL;DR: I connected Work IQ MCP (reads Microsoft 365 meeting data) and Azure DevOps MCP (creates work items) to GitHub Copilot in VS Code. I record most of my meetings, so GitHub Copilot reads the transcript, extracts action items, and drafts ADO work items for me to review. I sign off before anything gets created or modified.

The Toolchain

Four things wired together:

1. GitHub Copilot in VS Code: the orchestration layer. Runs the prompt, calls the MCP tools, handles the back-and-forth.
2. Work IQ MCP: reads my Microsoft 365 data (meetings, emails, Teams messages). This is where meeting context comes from.
3. Azure DevOps MCP Server: creates work items, sets fields, assigns owners.
4. Custom instructions and prompt files: the glue that tells GitHub Copilot what to extract and how to structure the output.

Work IQ: Reading Meeting Data

Work IQ is a Microsoft-built, first-party MCP server that exposes your Microsoft 365 work data — mail, calendar, Teams, files, people — as MCP tools.

How it works: you ask a natural-language question about your M365 data, the Work IQ MCP server translates that into Microsoft Graph requests behind the scenes, and you get structured data back. It runs under user-delegated permissions, so you’re scoped to what you already have access to.

It’s a single MCP server that handles all M365 domains, not separate servers per data type. You query it the way you’d ask a person: “what happened in my last meeting with the platform team?” and it figures out the right Graph calls to make. Work IQ is currently in public preview, so the specific tools and APIs may shift.

For this project, meeting transcripts are what matter. I record most of my meetings, so the transcripts are available. Work IQ can pull the full transcript, which gives GitHub Copilot much richer context for extracting action items. Who said what, what was agreed on, what was left open. That detail matters when you’re trying to turn a conversation into structured work items.

Azure DevOps MCP: Creating Real Work Items

The other half is the Azure DevOps MCP server. It gives GitHub Copilot the ability to perform actual ADO operations: creating work items, setting fields like title, description, assigned to, area path, iteration, priority. Not generating JSON that you paste somewhere.

How It Actually Works

The flow:

1. A meeting happens. I try to record most meetings, the transcript is available in M365.
2. I ask GitHub Copilot to pull the meeting transcript via Work IQ MCP.
3. GitHub Copilot extracts the action items: who’s responsible, what needs to happen, any deadlines mentioned.
4. GitHub Copilot drafts the ADO work items and presents them to me for review.
5. I review the draft. Edit titles and descriptions, remove anything that shouldn’t be there. Then I approve.
6. Only after I sign off does GitHub Copilot call the ADO MCP server to create the work items.

I don’t let GitHub Copilot create work items unsupervised. Meetings have nuance: speculative ideas, tangential discussions, someone volunteering someone else for work they don’t know about yet. An AI reading a transcript doesn’t know the difference between a decision and someone thinking out loud. I do.

A couple of things I learned that help a lot:

Reference your ADO org and project in your instructions file. If you put your default org URL and project name in your custom instructions or .github/copilot-instructions.md, GitHub Copilot doesn’t have to ask you every time. It just knows where to create work items. One less round-trip in every conversation.

Point to a parent work item. I usually reference the Epic I’m currently working under in my prompt. That way Copilot knows to create child Features, User Stories, or Tasks underneath it instead of dumping loose items into the backlog. Without this, you end up with orphaned work items that need manual reparenting. If your team structures work under Epics or Features, give GitHub Copilot that context upfront.

In practice, a prompt looks something like:

Read the transcript from my most recent meeting with 
the platform team. Extract action items and draft 
Azure DevOps work items in the Platform project under 
Epic #4521. Show me the draft before creating anything.

GitHub Copilot makes the Work IQ tool call, gets the transcript back, parses out action items, and shows me a structured draft. I make edits, say “looks good,” and then GitHub Copilot makes the ADO MCP calls. You can watch the tool calls happening in real time in the GitHub Copilot chat.

The interesting part isn’t any single tool call. It’s that GitHub Copilot handles the orchestration between two completely separate MCP servers in one conversation. Read from one system, get human approval, write to another.

Why GitHub Copilot Over Copilot Cowork

This is the question I keep getting. Microsoft has Copilot Cowork, built on Claude Cowork, an enterprise work orchestration agent that can plan multi-step tasks across M365 (calendar, Teams, Excel, email) with basically zero setup. It supports skills, plugins, and MCP connectors, so it’s not short on extensibility. For one-off knowledge work across M365, it works well.

But I chose GitHub Copilot + MCP for this, and here’s why: it lives where my dev tools already are.

Everything is in the repo. My custom instructions, prompt files, and skill definitions are files that live in version control. I can diff them, review them in PRs, and share them across the team through normal git workflows. Cowork’s skills and plugins exist in their own ecosystem outside of source control.

Built for developer workflows. Cowork is designed for knowledge workers doing non-technical tasks across M365, and it’s good at that. GitHub Copilot + MCP is designed for people who want to wire systems together programmatically, inspect tool calls, and iterate on prompts the way they iterate on code.

They solve different problems. Cowork gives you zero-friction M365 orchestration with guardrails. GitHub Copilot + MCP gives you a developer-native platform where the customization lives in your repo and the workflow runs in your IDE. For repeatable workflows that need to reach into ADO and stay under version control, I’ll take GitHub Copilot + MCP.

Where I’m Currently At

The setup works. I’ve used it on a handful of real meetings and the output is good enough that I keep using it. The drafts are usually 80-90% right. I fix a title here, remove a non-actionable item there, adjust an assignment. But the baseline is solid and the review step keeps me in control.

The prompt file is versioned, the instructions are tuned for my ADO setup, and the whole thing runs in about 30 seconds of GitHub Copilot chat plus a minute of review. Beats the 15 minutes of copy-paste-and-forget that it replaced. Or the zero minutes I spent when I just forgot entirely.

What’s Next

Right now this lives in my prompt files and custom instructions. It works for me, but if I want other people on the team to use it, I need to package it up somehow.

GitHub Copilot has three mechanisms for this: custom agents (.agent.md), prompt files (.prompt.md), and skills (SKILL.md). They all live in the repo, version in git, and give the team a single file to iterate on. But they work differently, and the differences matter for something that creates real work items in an external system.

I’m leaning toward a custom agent. The main reason is tool restriction. An .agent.md file can restrict Copilot to only the tools you declare in frontmatter, so when someone invokes @meeting-actions, Copilot operates with only the Work IQ and ADO MCP tools. No stray tool calls against your codebase when the intent is “process a meeting.” For a workflow that writes to a real system, scoped tool access matters from day one.

Agents also let you bake in custom instructions, model preferences, and the human review step as part of the agent definition itself. The team invokes it with @meeting-actions, and everything about how the workflow runs is encapsulated in one file. The extraction logic, the field mappings, what counts as an action item vs. a discussion point, the “show me the draft before creating anything” gate.

A prompt file (.prompt.md) is the lighter alternative. It can declare MCP tool dependencies in frontmatter and you invoke it with /meeting-work-items. The difference is that prompt files don’t restrict Copilot to only those tools. If you want to prototype the instructions first and don’t need tool scoping yet, a prompt file is less conceptual overhead. And upgrading from a prompt file to an agent later is low-cost since the instructions body is the same Markdown either way.

Skills were actually my first instinct, but they’re the weakest fit here. A SKILL.md gets auto-discovered by Copilot when it matches what you’re asking about, which is useful for guidance and reusable patterns. But auto-discovery is a liability when the end result is writing work items to ADO. You can disable auto-discovery with disable-model-invocation: true in the frontmatter, but at that point you’ve turned a skill into a slash command without the tool restriction or model preference support that agents and prompt files offer. Skills also can’t declare tool dependencies in machine-readable frontmatter. They reference tools in body text only.

Squad is an open-source framework conceived by Brady Gaster that creates an AI development team through GitHub Copilot. You describe what you’re building, and Squad proposes a team of specialists that live in your repo as files. They persist across sessions, learn your codebase, share decisions, and get better the more you use them.

It’s not a chatbot wearing hats. Each team member runs in its own context, reads only its own knowledge, and writes back what it learned.

I set it up for the repo of this blog. Here’s what I learned.

Installing Squad

Getting started takes two commands:

npm i -g @bradygaster/squad-cli
squad

For additional installation options (including npx and cloning from source), see the official installation guide.

Then open Copilot in VS Code, type @squad, and tell it what you’re building:

I'm starting a new project. Set up the team.

Squad proposes a team, each member named from a persistent thematic cast. You say yes. They’re ready.

The Team

Squad generated a team tailored to my blog’s needs:

• 🏗️ Mal / Lead: architecture, code review, cross-agent coordination
• ⚛️ Kaylee / Designer/Dev: CSS, layout, responsive design
• 📝 Wash / Content Dev: blog posts, frontmatter, SEO
• 🧪 Zoe / Tester: Playwright tests, accessibility, performance
• 📋 Scribe / Session logger: decisions, session summaries
• 🔄 Ralph / Work monitor: backlog, issue triage, CI monitoring

The names come from a persistent casting system. Once assigned, they stick. Anyone who clones the repo gets the same team with the same cast.

Each agent has a charter (charter.md) that defines scope and boundaries: what they own, what files they can modify, and critically, what they don’t touch. Kaylee owns CSS but never writes tests. Wash owns blog posts but never touches layout. These boundaries prevent agents from stepping on each other.

What Gets Created

Everything lives in a .squad/ directory:

.squad/
├── team.md            # Roster
├── routing.md         # Who handles what
├── decisions.md       # Shared brain
├── ceremonies.md      # Design reviews, retros
├── casting/
│   ├── policy.json
│   ├── registry.json
│   └── history.json
├── agents/
│   ├── kaylee/
│   │   ├── charter.md # Identity, expertise, voice
│   │   └── history.md # Project-specific learnings
│   ├── wash/
│   │   ├── charter.md
│   │   └── history.md
│   └── zoe/
│       ├── charter.md
│       └── history.md
└── log/               # Session history

Commit this folder. Your team persists. Names persist. It’s all in git.

Parallel Agents, Not Sequential

This is the part that surprised me most. When you give Squad a task, the coordinator launches every agent that can usefully start, simultaneously:

You: "Team, redesign the blog"
  🏗️ Mal    → analyzing architecture requirements
  ⚛️ Kaylee → building new layout             (all launched
  🧪 Zoe    → writing test cases from spec     in parallel)
  📋 Scribe → logging everything

When agents finish, the coordinator immediately chains follow-up work. Tests reveal edge cases, another agent picks them up, no waiting for you to ask.

Each agent gets its own context window. With Claude Sonnet 4 or Claude Opus 4’s 200K token window, and the coordinator kept thin, each agent has ~78–83% of its context available for actual work. Fan out to 5 agents and you’re working with ~1M tokens of total reasoning capacity.

Knowledge That Compounds

Every time an agent works, it writes lasting learnings to its history.md. After a few sessions, agents know your conventions, your preferences, your architecture. They stop asking questions they’ve already answered.

Team-wide decisions live in decisions.md, which every agent reads before working. Personal knowledge stays in each agent’s history.md. The Scribe keeps session logs searchable in log/.

Frontend agent knowledge over time: Stack, framework → Components, routing → Design system, a11y conventions

Lead agent knowledge over time: Scope, roster → Trade-offs, risks → Full project history, tech debt map

Tester agent knowledge over time: Framework, first cases → Edge case catalog → Regression patterns, coverage gaps

Issue Integration

Squad ties into GitHub Issues with a labeling workflow:

1. Label an issue squad. The Lead auto-triages it, determines who should handle it, and applies the right squad:{member} label.
2. The assigned member picks up the issue in their next Copilot session (or automatically if Copilot coding agent is enabled).
3. Labels sync automatically from your team roster via the sync-squad-labels workflow.

What I Learned

The first session is the least capable. Knowledge compounds. By the third or fourth session, agents were making decisions based on prior context without me having to repeat anything.

Boundaries matter more than capabilities. Clear charters that define what an agent doesn’t do are more important than what it can do. Overlap is the enemy.

The Scribe is secretly the most important agent. Coming back the next day to a searchable log of every decision and session is invaluable. Context is never lost.

If you’re using GitHub Copilot for your repo(s) today, give Squad a try. The jump from one agent to a coordinated team is pretty awesome, and it all lives in git.

To set some context, our team builds on Microsoft Fabric (data analytics SaaS platform) which recently announced Fabric IQ that included new features/capabilities such as Ontology and Data Agents, so the timing worked out great to kick the tires on what’s actually possible in practice.

TL;DR

• Start with the real problem, not the tools
• Cut scope ruthlessly and ship something valuable, not everything
• Keep the delivery simple while you’re figuring out if it even works
• Break big agents into smaller, more focused sub agents
• Engage stakeholders throughout the process
• Test often and iterate fast

Begin With a Clear Business Problem

Instead of jumping straight into tool selection, we focused on understanding the problem first. Before the hack, we had a brainstorming session to come up with a list of high impact use cases we could tackle. We landed on preparing monthly business reviews, specifically, analyzing KPI shifts, and writing up narratives. This takes our team a lot of time (hundreds of hours) and a lot of it gets repeated every single month. That helped us figure out what actually needed automating and what value an AI agent could realistically add in three days.

When we talked to stakeholders, they were pretty clear about what they needed: spot the biggest KPI moves, connect those changes to contextual factors, and provide concise summaries for leadership review. Those requirements told us which agent capabilities actually mattered and confirmed that building KPI and commentary prototypes was worth the effort.

Reduce Scope Early and Create a Practical MVP

There are a ton of options when it comes to user experience and how stakeholders would interact with our agents. To keep it simple, we decided the MVP would generate insights on a schedule and deliver them by email. It was more important that we get the outputs in the hands of the stakeholders ASAP to provide feedback and validate, than spend a bunch of time on a polished interface that provided garbage outputs.

From a tooling perspective, this allowed us to leverage tools we already had experience with to move fast: we chose Logic Apps to orchestrate Fabric data agents, passing outputs as variables, and merging them together to send the final email. There was quite a learning curve that came with ramping up on AI native orchestrators (Copilot Studio, AI Foundry, etc.) and environment setup, that just wasn’t realistic for a 3 day timeline.

Break Big Agents Into Smaller, More Focused Sub Agents

We started with a plan for two big agents: one for KPI analysis, one for writing narratives. As we threw more and more data sources at our agents, we found their accuracy turned to slop. We moved to finely scoped sub agents that were specialized for their tasks. We stripped out tables and measures we didn’t need, cleaned up the filtering logic, and set the agents to use narrower data sources. Those tweaks stabilized things and made sure each piece was actually doing what it was supposed to do. When we split up the work and trimmed unnecessary complexity from the agents, accuracy went way up. We now have 6 (and growing) specialized agents…

Lessons Learned: Fabric Data Agents

Specific to Fabric data agents, we experienced issues like using incorrect DAX queries and tables/columns that weren’t even in scope. To overcome this, we implemented a few practices:

Good Practices for Fabric Data Agents:

• Keep the Semantic Model as clear as possible, especially custom DAX measures
• Use the Prepare your data for AI tool on the published semantic model
• Select only the essential measures and tables when creating the agent
• Provide explicit instructions to the agent about when it should perform its own calculations

Engage Stakeholders Throughout the Process

Having stakeholders involved throughout the hackathon was extremely valuable. This helped us continuously validate what we were building and speed up the feedback loop. Which KPI moves actually mattered, how they wanted the summaries written, and what context was missing, etc. Bringing them in early meant we didn’t waste time building stuff nobody wanted. We could pivot fast based on what they told us.

Key Insight: Early stakeholder engagement turned potential weeks of rework into hours of quick adjustments. Their feedback directly shaped which agents we prioritized and how we structured the outputs.

Closing Thoughts

Three days gave us a chance to see what AI-powered agents could actually do. And honestly, it showed that with a tight timeline, you can still ship something real if you nail the problem, keep scope tight, and iterate like crazy.

Since the hackathon, we’ve come a long way with our solution but this gave us a jump start on the work.

I’ll go through some optimization techniques I’ve found to be effective:

• Caching Dependencies
• Optimize Docker Usage
• Strategic Job Splitting and Parallelization
• Matrix Builds for Testing Efficiently
• Selective Testing with Path Filtering
• Self-hosted Runners for Specialized Workloads
• Monitoring Your Workflow Performance
• Conclusion

Caching Dependencies

The most immediate win for most workflows is proper dependency caching:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16'
          
      - name: Cache dependencies
        uses: actions/cache@v3
        with:
          path: |
            ~/.npm
            node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-
            
      - name: Install dependencies
        run: npm ci --prefer-offline --no-audit

The above example includes several optimizations:

• Caching both ~/.npm and node_modules
• Using lockfile hashing for cache keys
• Fallback cache keys for partial hits
• --prefer-offline flag to prioritize cached packages

Optimize Docker Usage

For workflows using Docker, image layering and caching are critical:

jobs:
  docker-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
            
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max

After your build, add this step to prevent cache size explosion:

      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

Strategic Job Splitting and Parallelization

Instead of sequential steps, split work into parallel jobs when possible:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        # ...lint setup and execution
        
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        # ...test setup and execution
        
  build:
    runs-on: ubuntu-latest
    needs: [lint, test]
    # This job only runs after lint and test complete successfully

Matrix Builds for Testing Efficiently

For multi-environment testing, use matrix builds:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [14, 16, 18]
    steps:
      - uses: actions/checkout@v3
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test

Selective Testing with Path Filtering

Avoid running unnecessary jobs by filtering based on changed paths:

jobs:
  frontend-tests:
    if: |
      github.event_name == 'pull_request' &&
      (github.event.pull_request.base.ref == 'main' ||
       contains(github.event.pull_request.labels.*.name, 'run-all-tests'))
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            frontend:
              - 'frontend/**'
              - 'shared/**'
              
      - name: Frontend Tests
        if: steps.filter.outputs.frontend == 'true'
        run: npm test

Self-hosted Runners for Specialized Workloads

For specific needs like GPU access or large compute requirements, self-hosted runners can dramatically improve performance:

jobs:
  gpu-training:
    runs-on: self-hosted-gpu
    steps:
      - uses: actions/checkout@v3
      # GPU-intensive tasks here

Monitoring Your Workflow Performance

GitHub provides insights into workflow run times. Use the GitHub API to track this data over time:

async function getWorkflowRunTimes(owner, repo, workflow_id) {
  const { Octokit } = require("@octokit/rest");
  const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
  
  const runs = await octokit.actions.listWorkflowRuns({
    owner,
    repo,
    workflow_id,
  });
  
  return runs.data.workflow_runs.map(run => ({
    id: run.id,
    duration: new Date(run.updated_at) - new Date(run.created_at),
    conclusion: run.conclusion
  }));
}

Conclusion

Optimizing GitHub Actions workflows is an ongoing process. Start with the high-impact items like caching and job parallelization, then iteratively improve based on metrics. The reward is not just faster builds but happier developers and lower costs.

Why I Built This

The more I can do without touching my mouse, the happier I am. Switching VPN connections was one of those small but frequent tasks that bugged me - too many clicks just to turn a VPN on or off. Since I’m often already in VS Code when a VPN connection is required, why not control it from there?

What It Does

The VPN Toggle extension lets you:

• Toggle any Windows VPN connection on/off with commands
• Works with any VPN connection set up in Windows

How to Use It

1. Install the extension from the VS Code Marketplace
2. Press Ctrl+Shift+P and type “VPN” to see the available commands:
   • VPN: Select and Connect - Shows a list of available VPN connections to choose from
   • VPN: Connect to Last Used - Connects to the most recently used VPN
   • VPN: Disconnect Current - Disconnects the current VPN connection

Under the Hood

If you’re curious about how it works, the extension uses PowerShell commands to interact with Windows VPN connections. All the code is open source on GitHub if you want to take a peek or contribute.

Feedback Please!

Let me know what you think! Feel free to open an issue if you have any suggestions or run into problems.

I was recently notified that an old API key was discovered in one of the repos I own. Even if you remove the sensitive data in a new commit, it can still be found in the Git history.

To remove the API key, I decided to use the BFG Repo-Cleaner for cleansing bad data out of your Git repository history. However, I found myself fumbling around with the BFG CLI and spending way too much time trying to remove the key from the Git history.

That’s when I realized there had to be a better way. As a frequent user of Visual Studio Code, I thought, “Why not create a VS Code extension that simplifies this process?”.

Introducing the BFG VS Code Extension

The BFG VS Code Extension is a wrapper for the BFG Repo-Cleaner that makes it easy to remove credentials from your Git history. It guides you through the process step by step using the Command Palette, so you don’t have to remember complex CLI commands.

Here is how it works:

1. Clones a fresh copy of your repo using the –mirror flag.
2. Installs BFG: This step downloads the BFG jar file from the official repository and saves it in the workspace folder.
3. Enter credential to remove: This step prompts the user to enter the credential to remove, writes this credential to a file in the workspace folder, and uses the --replace-text option of BFG Repo-Cleaner to replace this credential with ***REMOVED*** in the repository’s history.
4. Remove credentials: This step runs the BFG Repo-Cleaner with the --replace-text option to replace the specified credential with ***REMOVED*** in the repository’s history.
5. Clean your repository: This step runs the git reflog expire --expire=now --all && git gc --prune=now --aggressive command to clean the repository.
6. Push the changes: This step runs the git push --force command to push the changes to the remote repository

You can find the BFG VS Code Extension on the VS Code Marketplace and the source code on GitHub. If you have any questions or feedback, feel free to open an issue on GitHub.

The best way to prevent sensitive data from being exposed in your Git history is to never commit it in the first place. Always use environment variables or configuration files that are ignored by Git to store sensitive data. But, if you do accidentally commit sensitive data, the BFG VS Code Extension is here to help you clean it up!

In this post, I’ll walk through how to configure both Playwright Test (JavaScript/TypeScript) and Playwright .NET to get test results in Azure Test Plans. Each option uses abstractions built on the Azure DevOps REST API so you don’t have to write additional code to accomplish this.

Why?

Azure Test Plans is a popular service that many teams are using for manual testing. By publishing your automated Playwright tests to the service, you get a couple of benefits:

1. Traceability. This gives you the option to link your requirements (Azure Boards) to automated tests and the pipeline that ran them. By mapping the two, you can establish the quality of the requirements based on test results. Ideally, a test case is created for each of the acceptance criteria listed for the requirement.
2. History. Drilling into every pipeline run to see test results over time is a pain. Azure Test Plans allows you to see results through features like the progress report and charts.
3. Test inventory. By tracking automated AND manual test cases, you can do things like track the status of a test case (not automated, planned to be automated, or automated). This makes it easy to track the progress of automated testing efforts, e.g. how many manual tests have been converted to automated, how many remain, etc.

What are the options?

I’ll show working code examples for both Playwright Test (TypeScript) and Playwright .NET using NUnit. If you’re already sick of reading and want to see them in action, here are some links.

TypeScript: tests, pipeline to run tests, test plan

.NET: tests, build to publish binaries, release to run tests, test plan

Playwright Test (TypeScript)

playwright-azure-reporter is a custom reporter (npm package) that allows you to post test results by annotating your test case name with the Azure test plan ID. The README has instructions for installing the package and adding the reporter to playwright.config.ts

My example project’s config looks like this: playwright.config.ts.

Once that is in place:

1. Manually create new test cases in Azure Test Plans taking note of the ID (planID in query string of URL)
2. Add the ID in brackets to the test case title. 444, 445 in this example:

When these tests get run, you will then be able to see the outcome for each test case:

My example pipeline runs these tests for every commit on main and also uses the JUnit reporter to publish results to the pipeline’s Test tab:

Playwright .NET

This option works out of the box but has some caveats and complexity: A Windows runner and a release pipeline are required to use the Visual Studio test platform installer and Visual Studio Test tasks. Also, Visual Studio must be used to associate test cases.

Here is how I set this up in my example project:

1. Manually create new Azure Test Plans test cases

2. Use Visual Studio’s test explorer to associate the automated test cases:

   

   This will change the Automation status field on the test case work item to automated:

   

   Once the test cases are configured, we can set up our pipelines to run the tests.

3. Create a build pipeline that runs dotnet publish (using Windows agent) in order to create an artifact with the Playright binaries: playwright-dotnet.yml

4. Create a release pipeline referencing the artifact created in the previous step:

   

5. Add install tasks (that run on Windows agent) for “Visual Studio Test Platform Installer” (prereq for VS Test task), .NET, and Playwright browsers:

   

6. Add the VS Test task and reference your test plan:

   

7. Create a new release to run the tests. Example results: Test tab, test plan results.

   

Summary

Hopefully, you were able to follow my examples to get this set up in your own environment. I’d love to hear feedback on anything I may have missed, new features you’d like to see from the product team at Microsoft, or interesting use cases you have experience with.

Happy testing, Marcus

Playwright already has useful features built in to report on things like the HTML report test step timing and the trace viewer that includes the call duration of each action.

HTML report test step duration

I wanted to take this a step further by using the Navigation Timing API, measuring start to loadEventEnd. All of the examples I found online used performance.timing, which is now deprecated. This is a very simple code snippet, but posting this will hopefully help others find a solution faster.

Here we have a function measurePerformance that can be called inside any test case to get navigation start to load event end times. This could easily be wrapped in a conditional to fail the test based on times. In my case, I just wanted it to be surfaced in the HTML report as a custom annotation to compare between sites, by toggling baseURL.

As a result, this is what the HTML report looks like:

Happy testing!

1. Create separate environments for development, staging, and production

GitHub Actions has an environments feature to describe a deployment target such as dev, staging, or production. By referencing the environment in a job, you can take advantage of protection rules and/or secrets that get scoped to the environment. Some potential use cases include requiring a particular person or team to approve workflow jobs that reference an environment (e.g. manual approval before production deploy), or limiting which branches can deploy to a particular environment. I also like to set the environment URL so it’s easily accessible from the summary page:

2. Establish workflow breakpoints with dependencies

By default, GitHub Actions runs multiple commands simultaneously. However, you can utilize the needs keyword to create dependencies between jobs, meaning that if a job fails (e.g. tests), dependent jobs won’t run. This also helps you control jobs which jobs run in parallel; if there aren’t dependencies between steps, break them out into separate jobs, then set their needs to the next step in the process.

e.g. my app, database, and infra as code projects can be built at the same time before deploying to dev:

3. Use secrets to store sensitive workflow data

GitHub’s secrets allow you to securely store sensitive data, including passwords, tokens, certificates, etc. You can directly reference secrets in workflows, meaning that you can create and share workflows with colleagues that employ secrets for secure values without hardcoding them directly into YAML workflow files. I like to scope the secrets close to the steps that require them. For example, rather than setting a secret for the entire workflow to access, it can be set for the job that contains steps that reference the secret.

e.g. Only the Playwright test job needs to reference AzureAD creds:

4. Conditionals can aid in differences between environments

GitHub Actions allows you to use conditionals that employ the “if” keyword to decide whether a step should run. You can use this feature to develop dependencies so that if a dependent job fails, the workflow can continue running. You can also use specific built-in functions for data operations, as well as leverage status check functions to determine whether preceding steps have succeeded, failed, canceled, or disrupted. Moreover, you can use conditionals to share workflow data among different branches and forks, with steps tailored to different triggers or environments. The conditions can also be set in reusable workflows to toggle different steps between environments:

e.g. I want reusable workflows to be uniform across environments, with the exception of steps that are only based on environmentName conditionals:

5. Share data between jobs to aid in “build once, deploy many”

GitHub Actions enables you to share data between jobs in any workflow as artifacts, which are linked to the workflow run where they are produced. This can help simplify the creation of workflows and facilitate the development of more complex automation where one workflow informs another via dependencies or conditionals. This also helps enable the mantra “build once, deploy many”. In other words, build projects in an environment-agnostic fashion, upload them as artifacts, then all deployment jobs use the same set of artifacts across environments.

6. Use contexts to access workflow information

Contexts represent a group of variables that can access details about workflow runs, runner environments, jobs, and steps to help derive key information about workflow operations. Contexts use expression syntax such as ${{ }}, and you can use most of them at any point in the workflow. I like to dump the entire context at the beginning of jobs to aid in troubleshooting:

Wrapping up

I’m curious to learn about other ways folks are leveraging GitHub Actions features. Add a comment to this post with any tips or tricks you’ve used with GitHub Actions!

NOTE: the creepy feature image for this post was generated via DALL-E

Environment Variables

Storing secrets in plain text in our code or configuration files can pose a significant security risk, especially if we share our code with others or publish it on public repositories like GitHub. Instead, we can store the credentials of our test accounts using environment variables. The environment variables are then referenced in our tests using the process core module of Node.js:

To set the values of these variables we can use our CI system’s secret management. For GitHub Actions, setting the values in the pipeline would look something like this:

example GitHub Actions workflow setting env vars scoped to job

To make local development easier, we can use .env files that are added to .gitignore to make sure they don’t get committed to source control.

example .env file with key-value pairs

Tips

• As a starting point, use codegen to walk through logging in, then refactor.
• Create a new tenant for testing and turn off MFA and security defaults. MFA cannot be fully automated and requires manual intervention.
• Optionally, set conditional access policies on your test environment to bypass login, then have a separate environment and tests for the login scenario itself.

• The test account will need to be granted permission to the app under test for the first time. You can either add conditionals to your test script (if X locator is present, then click Yes) to account for this or manually log in once to grant permissions. This is a one-time step.

  

• Auth can be set up to run at various stages of test execution. If all of your tests require auth, I’d recommend logging in once and re-using the signed-in state via [global setup](https://playwright.dev/docs/auth#reuse-signed-in-state). If only a subset of tests requires auth, you can use a [beforeAll hook](https://playwright.dev/docs/auth#reuse-the-signed-in-page-in-multiple-tests) or [fixture](https://playwright.dev/docs/test-fixtures).
  **EDIT**: As of 1.31, Playwright now has test project dependencies, which allows you to perform setup in a more advantageous approach compared to a global setup script (e.g. produce traces and HTML report). Docs now have example scripts to walk through this: https://playwright.dev/docs/auth#basic-shared-account-in-all-tests

Example setup

Create auth.setup.ts

Update playwright.config.ts with project dependencies, so the script above gets run before tests that need to be authenticated:

When the tests get run, the following will happen:

1. auth.setup.ts logs into AAD/Entra ID using creds from env variables
2. the signed-in state is saved to file storageState.json
3. the browser context for all of the test cases is created using the already logged-in state via storageState.json

With this setup, we reduce the test execution time by only logging in once, rather than in every individual test case.

Wrapping up

I’m curious to learn about other ways that people are handling AAD/Entra ID authentication in their Playwright tests. If you have experience with this, I’d love to hear about the challenges you faced and the solutions you came up with. Your insights could be valuable to others who are also working on Playwright test automation and facing similar issues.

Happy testing!