Playwright Integration with Mergify

This guide explains how to integrate Playwright with Test Insights using the @mergifyio/playwright reporter. Once installed, test results are automatically uploaded to Test Insights without any extra workflow changes.

Install the @mergifyio/playwright package alongside @playwright/test to automatically upload your test results to Test Insights.

npm install --save-dev @mergifyio/playwright

yarn add --dev @mergifyio/playwright

pnpm add --save-dev @mergifyio/playwright

Setting up the reporter takes two steps.

Wrap your playwright.config.ts with withMergify:

import { defineConfig } from '@playwright/test';
import { withMergify } from '@mergifyio/playwright';

export default withMergify(
  defineConfig({
    // ... your existing configuration
  })
);

withMergify adds the Mergify reporter while preserving any reporters, globalSetup, and globalTeardown you already have configured.

In your test files, import test and expect from @mergifyio/playwright instead of @playwright/test:

import { test, expect } from '@mergifyio/playwright';

test('logs in', async ({ page }) => {
  // ...
});

This import enables test quarantine: when a quarantined test fails, its outcome is reported as passing so it doesn’t block your pipeline.

Your workflow should run your tests as usual while exporting the secret MERGIFY_TOKEN as an environment variable.

Add the following to the GitHub Actions step running your tests:

env:
  MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}

For example:

- name: Run Tests 🧪
  env:
    MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
  run: npx playwright test

Set MERGIFY_TOKEN in the environment of the agents running your tests. The step itself then needs no Mergify-specific configuration:

steps:
  - label: "Run Tests 🧪"
    command: npx playwright test

The reporter automatically collects your test results and sends them to Test Insights.

The test you import from @mergifyio/playwright applies test quarantine inside the Playwright run. When a quarantined test ends in any failing status, a timeout included, that test sets the expected status to the outcome, so Playwright counts the result as expected and the result does not change Playwright’s exit code. The exit code of your test step already accounts for quarantine, so the step needs nothing more:

• Do not add continue-on-error: true. The recipes that upload a JUnit report with the mergifyio/gha-mergify-ci action need it because the action decides the job’s result after the tests. Here nothing does, so it would let every real failure through.

• There is no step id to set and no test_step_outcome to pass: both belong to that action, which this setup does not use.

A crash cannot pass for a green run either. If Playwright dies mid-run, it exits non-zero and the step fails. The reporter uploads results when the run ends, so a process killed outright before then, such as by the out-of-memory killer, sends nothing to Test Insights.

If the global setup that withMergify adds cannot fetch the quarantine list, the run quarantines nothing, and a quarantined test that fails makes the step fail as usual.

If your playwright.config.ts defines multiple projects, such as one per browser (chromium, firefox, webkit), the same test runs once per project. By default, Test Insights identifies each test by name only, so every project’s copy of a test shares the same identity and is treated as a single test. A test that passes on chromium but fails on webkit then looks flaky instead of consistently broken on one browser.

To keep each project’s tests separate, set PLAYWRIGHT_MERGIFY_INCLUDE_PROJECT_IN_TEST_NAME to true. The reporter then prefixes each test name with its project, for example [chromium] > login.spec.ts > logs in, so Test Insights tracks flakiness and quarantine per project:

env:
  MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
  PLAYWRIGHT_MERGIFY_INCLUDE_PROJECT_IN_TEST_NAME: "true"

This option is opt-in (off by default) to preserve the history of tests that already report without a project prefix.

After pushing these changes, your next CI run reports its Playwright results automatically.

You can then review your test results, including any failures or flaky tests, directly in the Test Insights dashboard.