I built an AI analyst to keep the speed and expose the work. For a focused question, it finds the governed semantic model that owns the answer. For a broader briefing, it can pull evidence together across the data landscape. The analyst runs queries under the user’s identity and includes its sources, periods, and filters with each answer. Type show dax to see the query. The LLM handles the conversation; institutional knowledge defines what a correct answer requires.

I started this as a side project around three months ago. People across our org use it, and I want to explain how it works.

I work at Microsoft on an analytics team supporting a large ecommerce business. We have plenty of data. Finding where it lives, stitching it together across reports and one-off spreadsheets, and crafting a story can be a pain in the ass.

The AI analyst complements reports and human analysts. It makes the governed semantic models built over our Microsoft Fabric gold layer easier to use, cuts the manual stitching, captures the institutional knowledge that makes a number trustworthy, and gives us more time to make decisions.

Using it

I use it for three sizes of question in one place.

Small: a single metric. “What was yesterday’s value for Metric A?” It picks the right semantic model and hands you the number.

Medium: a health check. One command gets you a cross-source briefing across a set of business areas, with each line graded against a threshold. It answers “Is anything on fire right now?” on one screen.

Large: an executive readout. One prompt pulls data across each relevant model, grades each area against a performance framework, and writes a decision brief: what happened, why it matters, and what to do about it, ending in a ranked action list with owners.

The executive readout saves me the most time. Before the agent, putting one together meant opening a stack of reports, lining up the numbers, writing the narrative, and formatting it for leadership. The agent can do that in a couple of minutes and attach the sources. I can inspect the query behind any answer and verify its filters and measures.

Ask a plain-English question and the answer includes its receipt: the source, filters, and period used.

you: what was the value of Metric A in Region A last quarter?

analyst: Metric A in Region A was X.X units. Source: semantic model A | Filters: Region = Region A, Scenario = Actuals | Period: Previous quarter

Type show dax and it returns the query it ran. I replaced the real table and measure names in this example:

EVALUATE
ROW( "Metric A",
    CALCULATE(
        [Metric A],
        'Geography'[Region] = "Region A",
        'Calendar'[Relative Quarter] = "Previous Quarter"
    )
)

How it works

The LLM handles the conversation. Routing, model guidance, and query execution are what make the answers useful.

I support three hosts: GitHub Copilot, Microsoft 365 Copilot Cowork, and Microsoft Scout. I started with a GitHub Copilot custom agent wired up with several MCP servers. MCP, the Model Context Protocol, is the open standard that lets an agent talk to external tools in a consistent way.

The MCP servers

• The core is Power BI’s remote MCP server (in preview). It separates two steps: generating a DAX query from a plain-English question and executing it against a semantic model under your identity. For a one-off question, the server generates a query within guardrails supplied by the model guidance: the right measures and required filters. For bigger briefings, the agent runs prewritten DAX from those files.
• Other tools extend the workflow:
  • The Azure DevOps MCP Server files a bug when something looks broken.
  • Web IQ fetches external market intelligence.
  • Work IQ brings in work context from Microsoft 365, such as files, meetings, mail, and chats. It runs under your identity and reaches content you can access.

The governed model

I rely on governed semantic models for trustworthy answers. We encode certified business logic and agreed definitions for core business measures in each model. Row-level security limits each user to the data they can access.

In a medallion architecture, data engineers store raw data in bronze tables, clean and conform it in silver, then shape it into business-ready gold tables. My team’s semantic models use that Microsoft Fabric gold layer. I built a thin natural-language layer that queries those models in place.

The analyst follows this path for each request:

Question → agent routing → registry and model guidance → governed semantic model → query result and source metadata → answer or briefing

The main components

The agent file. Defines routing, workflow, and every mode and command.

A model registry. Maps each report to its exact workspace and semantic-model IDs. After the agent chooses a model, the registry resolves the target up front, so the agent does not wander around discovering what exists at runtime.

Per-model guidance. One prompt file per report, in plain Markdown, encodes the institutional knowledge: which filters must always be present, how measure names differ across tables, which dimension value looks like a total but is not, and which measures must never feed a narrative. These files carry the accumulated product knowledge; the LLM provides a flexible interface.

The exact rules vary by model, but a sanitized sketch of the shape looks like this:

## Required filters
- Scope: required on every query
- Scenario: Actuals

## Measure guidance
- Metric A: use the certified measure
- Never narrate from measures listed as unsafe

The execution layer. The Power BI remote MCP server runs the query in GitHub Copilot; Fabric IQ runs it in Cowork.

One brain, three front doors

I shipped it first as the GitHub Copilot custom agent. The first practical question was, “Great, how do I use this without opening an IDE?”

It has three front doors:

• The GitHub Copilot custom agent in VS Code, with the full set of MCP servers.
• A plugin for Microsoft 365 Copilot Cowork, which uses Fabric IQ for Power BI access.
• A plugin for Microsoft Scout, an OpenClaw-like desktop app for Windows and macOS that acts across your files, shell, browser, and Microsoft 365. Scout supports a compatible agent-and-plugin shape built on the GitHub Copilot CLI, so I ported the Markdown files with few changes; the plugin adds the MCP servers the host does not ship.

The agent file and per-report prompt files form one canonical spec. A script generates the Scout desktop plugin from it by rewriting the front matter, re-anchoring file paths, and transforming the MCP configuration for that host. The Cowork edition reuses the same prompt files. These hosts are converging on the same basic shape: MCP for tools plus similar agent and skill formats. I can keep the host-specific glue small.

Where it is going

Next, I plan to host it as a Foundry hosted agent. Foundry Agent Service can run a packaged agent on managed infrastructure that handles scaling and session state. I could publish a hosted version to Microsoft Teams and Microsoft 365 Copilot Chat, where people already work.

I also plan to add data sources and connect the analyst to daily workflows.

Stuff I learned

Institutional knowledge took most of the work. A bare LLM can return a plausible but wrong number. I spent most of the effort writing down or referencing the institutional knowledge that makes an answer correct.

“Show me the query” makes review concrete. People do not trust a black box, and they are right not to. Letting someone inspect the filters and measures does more for trust than a claim about accuracy can.

Static checks protect the contract today. For relevant changes, CI checks that golden questions route to the right model, model files keep their required filters, known-bad measures stay quarantined, and safety sections remain intact. The checks need no credentials or live queries and run in seconds. They catch regressions before the analyst starts returning garbage.

End-to-end trace evals are next. The next tier will record real runs, check generated DAX for required filters, and trace each narrative number to a query result. I need unattended agent runs to build those evals, another reason to host it on Foundry.

Adding another supported semantic model requires configuration. It needs a registry entry, per-model guidance, and routing and eval cases. I wrote a CONTRIBUTING.md with step-by-step instructions and practices that make the process repeatable.

Wrapping up

This is how I want data questions to work: ask in plain English, pull the answer from the right governed models, and keep the evidence attached. Each answer keeps its source, filters, and query attached for review.

If you are building something similar, I would love to compare notes on how you encode business rules and keep the work reviewable.

I wired together GitHub Copilot, Work IQ, and Azure DevOps to pull meeting transcripts, extract action items, and draft work items for my review. I approve each draft before GitHub Copilot creates or changes a work item.

TL;DR: I connected Work IQ MCP (reads Microsoft 365 meeting data) and Azure DevOps MCP (creates work items) to GitHub Copilot in VS Code. I record most of my meetings, so GitHub Copilot reads the transcript, extracts action items, and drafts ADO work items for me to review. I sign off before GitHub Copilot creates or changes anything.

The Toolchain

The workflow combines four pieces:

1. GitHub Copilot in VS Code: the orchestration layer. Runs the prompt, calls the MCP tools, handles the back-and-forth.
2. Work IQ MCP: reads my Microsoft 365 data (meetings, emails, Teams messages). This is where meeting context comes from.
3. Azure DevOps MCP Server: creates work items, sets fields, assigns owners.
4. Custom instructions and prompt files: the glue that tells GitHub Copilot what to extract and how to structure the output.

Work IQ: Reading Meeting Data

Work IQ is a Microsoft-built, first-party MCP server that exposes your Microsoft 365 work data as MCP tools: mail, calendar, Teams, files, and people.

Work IQ takes a natural-language question about your M365 data, translates it into Microsoft Graph requests, and returns structured data. It uses user-delegated permissions and accesses content available to you.

One MCP server handles all M365 domains. You query it the way you’d ask a person: “what happened in my last meeting with the platform team?” and it figures out the right Graph calls to make. Work IQ remains in public preview, so the specific tools and APIs may shift.

I use meeting transcripts for this project. I record most of my meetings, so the transcripts sit in M365. Work IQ pulls the full transcript, giving GitHub Copilot context about who said what, what people agreed to, and which items remained unresolved. I get enough detail to act on a draft three weeks later.

Azure DevOps MCP: Creating Real Work Items

The Azure DevOps MCP server lets GitHub Copilot create work items and set fields such as title, description, assigned to, area path, iteration, and priority. It writes the items to the backlog, so I do not have to paste JSON somewhere and tidy it up by hand.

How It Works

I use this six-step flow:

1. I record most meetings, which makes their transcripts available in M365.
2. I ask GitHub Copilot to pull the meeting transcript via Work IQ MCP.
3. GitHub Copilot extracts the action items: who’s responsible, what needs to happen, any deadlines mentioned.
4. GitHub Copilot drafts the ADO work items and presents them to me for review.
5. I review the draft, edit titles and descriptions, and remove anything that should not be there.
6. After I sign off, GitHub Copilot calls the ADO MCP server to create the work items.

I don’t let GitHub Copilot create work items unsupervised. Meetings are messy: speculative ideas, tangents, and the time-honored move of someone volunteering a colleague for work that colleague hasn’t heard about yet. I was in the room, so I can separate real decisions from speculation during review.

Two details reduce cleanup:

Reference your ADO org and project in your instructions file. Add your default organization URL and project name to your custom instructions or .github/copilot-instructions.md. GitHub Copilot can then target the right project without another round trip.

Point to a parent work item. I include the Epic I’m working under in my prompt. Copilot can then create child Features, User Stories, or Tasks beneath it. Otherwise, Copilot may create orphaned work items that I have to reparent. If your team structures work under Epics or Features, give GitHub Copilot that context up front.

My prompt looks like this:

Read the transcript from my most recent meeting with 
the platform team. Extract action items and draft 
Azure DevOps work items in the Platform project under 
Epic #4521. Show me the draft before creating anything.

GitHub Copilot makes the Work IQ tool call, gets the transcript back, parses out action items, and shows me a structured draft. I make edits, say “looks good,” and GitHub Copilot makes the ADO MCP calls. You can watch the tool calls in GitHub Copilot chat.

GitHub Copilot orchestrates two separate MCP servers in one conversation: it reads from one system, waits for my approval, and writes to another.

Choosing GitHub Copilot Over Copilot Cowork

Microsoft has Copilot Cowork, built on Claude Cowork, an enterprise work orchestration agent that can plan multi-step tasks across M365 across calendar, Teams, Excel, and email with little setup. It also supports skills, plugins, and MCP connectors. For one-off knowledge work across M365, it works well.

I chose GitHub Copilot + MCP because it lives where my dev tools are.

Everything is in the repo. My custom instructions, prompt files, and skill definitions are files that live in version control. I can diff them, review them in PRs, and share them across the team through normal git workflows. Cowork’s skills and plugins exist in their own ecosystem outside of source control.

Built for developer workflows. Cowork is designed for knowledge workers doing non-technical tasks across M365, and it’s good at that. GitHub Copilot + MCP is designed for people who want to wire systems together through code, inspect tool calls, and iterate on prompts the way they iterate on code.

Cowork handles M365 orchestration with built-in guardrails. GitHub Copilot + MCP gives you a developer-native platform where the customization lives in your repo and the workflow runs in your IDE. For repeatable workflows that need to reach into ADO and stay under version control, I’ll take GitHub Copilot + MCP.

Current Results

I’ve run the setup against a handful of real meetings. The drafts land around 80-90% right: I fix a title, delete an item that was never an action item, and tweak an assignment or two. The review step keeps me in control.

I keep the prompt file in version control and tune the instructions for my ADO setup. The workflow takes about 30 seconds of GitHub Copilot chat plus a minute of review. I used to spend 15 minutes copying and pasting, or zero minutes when I forgot the work.

Packaging It for the Team

The workflow lives in my prompt files and custom instructions. To share it with the team, I need to package it as a reusable Copilot customization.

GitHub Copilot offers three mechanisms for this: custom agents (.agent.md), prompt files (.prompt.md), and skills (SKILL.md). All three live in the repo, use git for versioning, and give the team a single file to iterate on. Their differences matter for a workflow that creates work items in an external system.

I plan to use a custom agent because it supports tool restriction. An .agent.md file can restrict Copilot’s tool set to declarations in frontmatter. When you invoke @meeting-actions, Copilot uses the Work IQ and ADO MCP tools without access to unrelated codebase tools. For a workflow that writes to a real system, scoped tool access matters from day one.

Agents also let you include custom instructions, model preferences, and the human review step in the agent definition. The team invokes it with @meeting-actions. One file contains the extraction logic, field mappings, definition of an action item, and the “show me the draft before creating anything” gate.

A prompt file (.prompt.md) is the lighter alternative. It can declare MCP tool dependencies in frontmatter, and you invoke it with /meeting-work-items. Prompt files leave Copilot’s other tools available. If you want to prototype the instructions first and don’t need tool scoping yet, a prompt file has less conceptual overhead. Moving from a prompt file to an agent costs little because the instructions body uses the same Markdown.

I considered skills first, but they fit this workflow least well. Copilot auto-discovers a SKILL.md when it matches your request, which helps with guidance and reusable patterns. Auto-discovery adds risk when the result writes work items to ADO. You can disable it with disable-model-invocation: true in the frontmatter. Doing so turns a skill into a slash command without the tool restriction or model preference support that agents and prompt files offer. Skills also can’t declare tool dependencies in machine-readable frontmatter. They reference tools in body text.

Brady Gaster created the open-source framework to build AI development teams through GitHub Copilot. You describe what you’re building, and Squad proposes a team of specialists that live in your repo as files. They persist across sessions, learn your codebase, share decisions, and improve as you use them.

Each team member runs in its own context, reads its own knowledge, and writes back what it learned.

I set it up for the repo behind this blog.

Installing Squad

Getting started takes two commands:

npm i -g @bradygaster/squad-cli
squad

For additional installation options (including npx and cloning from source), see the official installation guide.

Then open Copilot in VS Code, type @squad, and tell it what you’re building:

I'm starting a new project. Set up the team.

Squad proposes a team with names from a persistent thematic cast. Once you approve it, the team is ready.

The Team

Squad generated a team tailored to my blog’s needs:

• 🏗️ Mal / Lead: architecture, code review, cross-agent coordination
• ⚛️ Kaylee / Designer/Dev: CSS, layout, responsive design
• 📝 Wash / Content Dev: blog posts, frontmatter, SEO
• 🧪 Zoe / Tester: Playwright tests, accessibility, performance
• 📋 Scribe / Session logger: decisions, session summaries
• 🔄 Ralph / Work monitor: backlog, issue triage, CI monitoring

The casting system keeps each name after assignment, so a teammate who clones the repo gets the same team and cast.

Each agent has a charter (charter.md) that defines scope and boundaries: what they own, what files they can modify, and what they don’t touch. Kaylee owns CSS but never writes tests. Wash owns blog posts but never touches layout. These boundaries prevent agents from stepping on each other.

What Gets Created

Everything lives in a .squad/ directory:

.squad/
├── team.md            # Roster
├── routing.md         # Who handles what
├── decisions.md       # Shared brain
├── ceremonies.md      # Design reviews, retros
├── casting/
│   ├── policy.json
│   ├── registry.json
│   └── history.json
├── agents/
│   ├── kaylee/
│   │   ├── charter.md # Identity, expertise, voice
│   │   └── history.md # Project-specific learnings
│   ├── wash/
│   │   ├── charter.md
│   │   └── history.md
│   └── zoe/
│       ├── charter.md
│       └── history.md
└── log/               # Session history

Commit this folder, and the team’s names and knowledge persist in git.

Parallel Agents, Not Sequential

I did not expect Squad to run work in parallel. When you give it a task, the coordinator launches all relevant agents at once:

You: "Team, redesign the blog"
  🏗️ Mal    → analyzing architecture requirements
  ⚛️ Kaylee → building new layout             (all launched
  🧪 Zoe    → writing test cases from spec     in parallel)
  📋 Scribe → logging everything

As agents finish, the coordinator chains follow-up work. A test can expose an edge case, and another agent can pick it up without waiting for you to ask.

Each agent gets its own context window. With Claude Sonnet 4 or Claude Opus 4’s 200K token window, a lightweight coordinator leaves each agent ~78–83% of its context for project work. Fan out to 5 agents and you’re working with ~1M tokens of total reasoning capacity.

Knowledge That Compounds

Every time an agent works, it writes lasting learnings to its history.md. After a few sessions, agents know your conventions, your preferences, your architecture. They stop asking questions they’ve already answered.

Team-wide decisions live in decisions.md, which every agent reads before working. Personal knowledge stays in each agent’s history.md. The Scribe keeps session logs searchable in log/.

Frontend agent knowledge over time: Stack, framework → Components, routing → Design system, a11y conventions

Lead agent knowledge over time: Scope, roster → Trade-offs, risks → Full project history, tech debt map

Tester agent knowledge over time: Framework, first cases → Edge case catalog → Regression patterns, coverage gaps

Issue Integration

Squad ties into GitHub Issues with a labeling workflow:

1. Label an issue squad. The Lead triages it, determines who should handle it, and applies the right squad:{member} label.
2. The assigned member picks up the issue in their next Copilot session. Copilot coding agent can pick it up sooner when enabled.
3. The sync-squad-labels workflow syncs labels from your team roster.

Lessons From Using It

The first session is the least capable. Knowledge compounds. By the third or fourth session, agents were making decisions based on prior context without me having to repeat anything.

Clear charters need explicit exclusions. Define what each agent owns and what it cannot touch. That separation keeps agents from duplicating or conflicting with one another.

The Scribe does the most valuable work for me. Its searchable log of decisions and sessions saves me from reconstructing what past-me was thinking the next day.

If you’re using GitHub Copilot for your repo today, give Squad a try. It splits work across separate contexts while keeping the team configuration in git.

Our team builds on Microsoft Fabric, a data analytics SaaS platform. Microsoft announced Fabric IQ, including Ontology and Data Agents, before the hackathon, so we tested those capabilities in a real scenario.

TL;DR

• Start with the business problem
• Cut scope to a useful MVP
• Use a simple delivery path for the first test
• Break big agents into smaller, more focused sub agents
• Engage stakeholders throughout the process
• Test often and iterate fast

Begin With a Clear Business Problem

We began by defining the problem. Before the hack, we listed candidate use cases and chose monthly business review preparation: analyzing KPI shifts and writing narratives. The team spends hundreds of hours on this work each month. We used that scope to identify what needed automation and what an AI agent could deliver in three days.

Stakeholders asked us to spot the biggest KPI moves, connect those changes to contextual factors, and provide concise summaries for leadership review. We used those requirements to prioritize KPI analysis and commentary prototypes.

Reduce Scope and Create a Practical MVP

We needed a way for stakeholders to receive the agent’s output. For the MVP, we scheduled insight generation and delivered the result by email. Email put outputs in stakeholders’ hands fast, so they could validate the content before we invested in a polished interface.

We also chose tools we knew. Logic Apps orchestrated the Fabric data agents, passed outputs as variables, and merged them into the final email. Learning Copilot Studio or AI Foundry and setting up their environments would have consumed most of the three-day hackathon.

Break Big Agents Into Smaller, More Focused Sub Agents

We started with a plan for two big agents: one for KPI analysis and one for writing narratives. As we added data sources, their accuracy turned to slop. We split them into specialized subagents with narrow scopes. We stripped out tables and measures we didn’t need, cleaned up the filtering logic, and set the agents to use narrower data sources. Narrowing the scope this way stabilized the output and improved accuracy. The prototype grew to six specialized agents.

Lessons Learned: Fabric Data Agents

Fabric data agents sometimes generated incorrect DAX or referenced tables and columns outside their scope. We responded with four practices:

Good Practices for Fabric Data Agents:

• Keep the semantic model and its custom DAX measures clear
• Use the Prepare your data for AI tool on the published semantic model
• Select the essential measures and tables when creating the agent
• Provide explicit instructions to the agent about when it should perform its own calculations

Engage Stakeholders Throughout the Process

Stakeholders reviewed the work throughout the hackathon, which shortened the feedback loop. They told us which KPI moves mattered, how they wanted the summaries written, and what context was missing. Their early feedback kept us from building features nobody wanted and let us adjust the prototypes during the hackathon.

Closing Thoughts

In three days, we tested AI agents against a real monthly-review workflow. A clear problem and tight scope let us ship a working prototype. Stakeholder feedback guided each adjustment.

The three-day prototype gave us the starting point for the solution we continued to build.

I’ll go through some optimization techniques I’ve found to be effective:

• Caching Dependencies
• Optimize Docker Usage
• Strategic Job Splitting and Parallelization
• Matrix Builds for Testing Efficiently
• Selective Testing with Path Filtering
• Self-hosted Runners for Specialized Workloads
• Monitoring Your Workflow Performance
• Conclusion

Caching Dependencies

The most immediate win for most workflows is proper dependency caching:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16'
          
      - name: Cache dependencies
        uses: actions/cache@v3
        with:
          path: |
            ~/.npm
            node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-
            
      - name: Install dependencies
        run: npm ci --prefer-offline --no-audit

The above example includes several optimizations:

• Caching both ~/.npm and node_modules
• Using lockfile hashing for cache keys
• Fallback cache keys for partial hits
• --prefer-offline flag to prioritize cached packages

Optimize Docker Usage

For workflows using Docker, image layering and caching are critical:

jobs:
  docker-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
            
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: false
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max

After your build, add this step to prevent cache size explosion:

      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

Strategic Job Splitting and Parallelization

Instead of sequential steps, split work into parallel jobs when possible:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        # ...lint setup and execution
        
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        # ...test setup and execution
        
  build:
    runs-on: ubuntu-latest
    needs: [lint, test]
    # This job only runs after lint and test complete successfully

Matrix Builds for Testing Efficiently

For multi-environment testing, use matrix builds:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [14, 16, 18]
    steps:
      - uses: actions/checkout@v3
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test

Selective Testing with Path Filtering

Avoid running unnecessary jobs by filtering based on changed paths:

jobs:
  frontend-tests:
    if: |
      github.event_name == 'pull_request' &&
      (github.event.pull_request.base.ref == 'main' ||
       contains(github.event.pull_request.labels.*.name, 'run-all-tests'))
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            frontend:
              - 'frontend/**'
              - 'shared/**'
              
      - name: Frontend Tests
        if: steps.filter.outputs.frontend == 'true'
        run: npm test

Self-hosted Runners for Specialized Workloads

For specific needs like GPU access or large compute requirements, self-hosted runners can dramatically improve performance:

jobs:
  gpu-training:
    runs-on: self-hosted-gpu
    steps:
      - uses: actions/checkout@v3
      # GPU-intensive tasks here

Monitoring Your Workflow Performance

GitHub provides insights into workflow run times. Use the GitHub API to track this data over time:

async function getWorkflowRunTimes(owner, repo, workflow_id) {
  const { Octokit } = require("@octokit/rest");
  const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
  
  const runs = await octokit.actions.listWorkflowRuns({
    owner,
    repo,
    workflow_id,
  });
  
  return runs.data.workflow_runs.map(run => ({
    id: run.id,
    duration: new Date(run.updated_at) - new Date(run.created_at),
    conclusion: run.conclusion
  }));
}

Conclusion

Optimizing GitHub Actions workflows is an ongoing process. Start with the high-impact items like caching and job parallelization, then iteratively improve based on metrics. The reward is not just faster builds but happier developers and lower costs.

Why I Built This

The more I can do without touching my mouse, the happier I am. Switching VPN connections was one of those small but frequent tasks that bugged me - too many clicks just to turn a VPN on or off. Since I’m often already in VS Code when a VPN connection is required, why not control it from there?

What It Does

The VPN Toggle extension lets you:

• Toggle any Windows VPN connection on/off with commands
• Works with any VPN connection set up in Windows

How to Use It

1. Install the extension from the VS Code Marketplace
2. Press Ctrl+Shift+P and type “VPN” to see the available commands:
   • VPN: Select and Connect - Shows a list of available VPN connections to choose from
   • VPN: Connect to Last Used - Connects to the most recently used VPN
   • VPN: Disconnect Current - Disconnects the current VPN connection

Under the Hood

If you’re curious about how it works, the extension uses PowerShell commands to interact with Windows VPN connections. All the code is open source on GitHub if you want to take a peek or contribute.

Feedback Please!

Let me know what you think! Feel free to open an issue if you have any suggestions or run into problems.

I was recently notified that an old API key was discovered in one of the repos I own. Even if you remove the sensitive data in a new commit, it can still be found in the Git history.

To remove the API key, I decided to use the BFG Repo-Cleaner for cleansing bad data out of your Git repository history. However, I found myself fumbling around with the BFG CLI and spending way too much time trying to remove the key from the Git history.

That’s when I realized there had to be a better way. As a frequent user of Visual Studio Code, I thought, “Why not create a VS Code extension that simplifies this process?”.

Introducing the BFG VS Code Extension

The BFG VS Code Extension is a wrapper for the BFG Repo-Cleaner that makes it easy to remove credentials from your Git history. It guides you through the process step by step using the Command Palette, so you don’t have to remember complex CLI commands.

Here is how it works:

1. Clones a fresh copy of your repo using the –mirror flag.
2. Installs BFG: This step downloads the BFG jar file from the official repository and saves it in the workspace folder.
3. Enter credential to remove: This step prompts the user to enter the credential to remove, writes this credential to a file in the workspace folder, and uses the --replace-text option of BFG Repo-Cleaner to replace this credential with ***REMOVED*** in the repository’s history.
4. Remove credentials: This step runs the BFG Repo-Cleaner with the --replace-text option to replace the specified credential with ***REMOVED*** in the repository’s history.
5. Clean your repository: This step runs the git reflog expire --expire=now --all && git gc --prune=now --aggressive command to clean the repository.
6. Push the changes: This step runs the git push --force command to push the changes to the remote repository

You can find the BFG VS Code Extension on the VS Code Marketplace and the source code on GitHub. If you have any questions or feedback, feel free to open an issue on GitHub.

The best way to prevent sensitive data from being exposed in your Git history is to never commit it in the first place. Always use environment variables or configuration files that are ignored by Git to store sensitive data. But, if you do accidentally commit sensitive data, the BFG VS Code Extension is here to help you clean it up!

In this post, I’ll walk through how to configure both Playwright Test (JavaScript/TypeScript) and Playwright .NET to get test results in Azure Test Plans. Each option uses abstractions built on the Azure DevOps REST API so you don’t have to write additional code to accomplish this.

Why?

Azure Test Plans is a popular service that many teams are using for manual testing. By publishing your automated Playwright tests to the service, you get a couple of benefits:

1. Traceability. This gives you the option to link your requirements (Azure Boards) to automated tests and the pipeline that ran them. By mapping the two, you can establish the quality of the requirements based on test results. Ideally, a test case is created for each of the acceptance criteria listed for the requirement.
2. History. Drilling into every pipeline run to see test results over time is a pain. Azure Test Plans allows you to see results through features like the progress report and charts.
3. Test inventory. By tracking automated AND manual test cases, you can do things like track the status of a test case (not automated, planned to be automated, or automated). This makes it easy to track the progress of automated testing efforts, e.g. how many manual tests have been converted to automated, how many remain, etc.

What are the options?

I’ll show working code examples for both Playwright Test (TypeScript) and Playwright .NET using NUnit. If you’re already sick of reading and want to see them in action, here are some links.

TypeScript: tests, pipeline to run tests, test plan

.NET: tests, build to publish binaries, release to run tests, test plan

Playwright Test (TypeScript)

playwright-azure-reporter is a custom reporter (npm package) that allows you to post test results by annotating your test case name with the Azure test plan ID. The README has instructions for installing the package and adding the reporter to playwright.config.ts

My example project’s config looks like this: playwright.config.ts.

Once that is in place:

1. Manually create new test cases in Azure Test Plans taking note of the ID (planID in query string of URL)
2. Add the ID in brackets to the test case title. 444, 445 in this example:

When these tests get run, you will then be able to see the outcome for each test case:

My example pipeline runs these tests for every commit on main and also uses the JUnit reporter to publish results to the pipeline’s Test tab:

Playwright .NET

This option works out of the box but has some caveats and complexity: A Windows runner and a release pipeline are required to use the Visual Studio test platform installer and Visual Studio Test tasks. Also, Visual Studio must be used to associate test cases.

Here is how I set this up in my example project:

1. Manually create new Azure Test Plans test cases

2. Use Visual Studio’s test explorer to associate the automated test cases:

   

   This will change the Automation status field on the test case work item to automated:

   

   Once the test cases are configured, we can set up our pipelines to run the tests.

3. Create a build pipeline that runs dotnet publish (using Windows agent) in order to create an artifact with the Playright binaries: playwright-dotnet.yml

4. Create a release pipeline referencing the artifact created in the previous step:

   

5. Add install tasks (that run on Windows agent) for “Visual Studio Test Platform Installer” (prereq for VS Test task), .NET, and Playwright browsers:

   

6. Add the VS Test task and reference your test plan:

   

7. Create a new release to run the tests. Example results: Test tab, test plan results.

   

Summary

Hopefully, you were able to follow my examples to get this set up in your own environment. I’d love to hear feedback on anything I may have missed, new features you’d like to see from the product team at Microsoft, or interesting use cases you have experience with.

Happy testing, Marcus

Playwright already has useful features built in to report on things like the HTML report test step timing and the trace viewer that includes the call duration of each action.

HTML report test step duration

I wanted to take this a step further by using the Navigation Timing API, measuring start to loadEventEnd. All of the examples I found online used performance.timing, which is now deprecated. This is a very simple code snippet, but posting this will hopefully help others find a solution faster.

Here we have a function measurePerformance that can be called inside any test case to get navigation start to load event end times. This could easily be wrapped in a conditional to fail the test based on times. In my case, I just wanted it to be surfaced in the HTML report as a custom annotation to compare between sites, by toggling baseURL.

As a result, this is what the HTML report looks like:

Happy testing!

1. Create separate environments for development, staging, and production

GitHub Actions has an environments feature to describe a deployment target such as dev, staging, or production. By referencing the environment in a job, you can take advantage of protection rules and/or secrets that get scoped to the environment. Some potential use cases include requiring a particular person or team to approve workflow jobs that reference an environment (e.g. manual approval before production deploy), or limiting which branches can deploy to a particular environment. I also like to set the environment URL so it’s easily accessible from the summary page:

2. Establish workflow breakpoints with dependencies

By default, GitHub Actions runs multiple commands simultaneously. However, you can utilize the needs keyword to create dependencies between jobs, meaning that if a job fails (e.g. tests), dependent jobs won’t run. This also helps you control jobs which jobs run in parallel; if there aren’t dependencies between steps, break them out into separate jobs, then set their needs to the next step in the process.

e.g. my app, database, and infra as code projects can be built at the same time before deploying to dev:

3. Use secrets to store sensitive workflow data

GitHub’s secrets allow you to securely store sensitive data, including passwords, tokens, certificates, etc. You can directly reference secrets in workflows, meaning that you can create and share workflows with colleagues that employ secrets for secure values without hardcoding them directly into YAML workflow files. I like to scope the secrets close to the steps that require them. For example, rather than setting a secret for the entire workflow to access, it can be set for the job that contains steps that reference the secret.

e.g. Only the Playwright test job needs to reference AzureAD creds:

4. Conditionals can aid in differences between environments

GitHub Actions allows you to use conditionals that employ the “if” keyword to decide whether a step should run. You can use this feature to develop dependencies so that if a dependent job fails, the workflow can continue running. You can also use specific built-in functions for data operations, as well as leverage status check functions to determine whether preceding steps have succeeded, failed, canceled, or disrupted. Moreover, you can use conditionals to share workflow data among different branches and forks, with steps tailored to different triggers or environments. The conditions can also be set in reusable workflows to toggle different steps between environments:

e.g. I want reusable workflows to be uniform across environments, with the exception of steps that are only based on environmentName conditionals:

5. Share data between jobs to aid in “build once, deploy many”

GitHub Actions enables you to share data between jobs in any workflow as artifacts, which are linked to the workflow run where they are produced. This can help simplify the creation of workflows and facilitate the development of more complex automation where one workflow informs another via dependencies or conditionals. This also helps enable the mantra “build once, deploy many”. In other words, build projects in an environment-agnostic fashion, upload them as artifacts, then all deployment jobs use the same set of artifacts across environments.

6. Use contexts to access workflow information

Contexts represent a group of variables that can access details about workflow runs, runner environments, jobs, and steps to help derive key information about workflow operations. Contexts use expression syntax such as ${{ }}, and you can use most of them at any point in the workflow. I like to dump the entire context at the beginning of jobs to aid in troubleshooting:

Wrapping up

I’m curious to learn about other ways folks are leveraging GitHub Actions features. Add a comment to this post with any tips or tricks you’ve used with GitHub Actions!