Pytest Integration with Mergify

This guide explains how to integrate Pytest with Test Insights using the pytest-mergify plugin. Once installed, test results are automatically uploaded to Test Insights without any extra workflow changes.

You need to install the pytest-mergify plugin to automatically upload your test results to Test Insights. This can be done in different ways depending on your Python dependency setup. Below are a few examples:

pip install pytest-mergify

Or add pytest-mergify to your requirements.txt.

setup(
    name="your-package",
    ...
    install_requires=[
        ...
    ],
    extras_require={
        "dev": ["pytest-mergify"]
    },
    ...
)

Make sure those dependencies are installed when running your tests.

[options.extras_require]
dev =
    pytest-mergify

Make sure those dependencies are installed when running your tests.

poetry add --group dev pytest-mergify

Your workflow should run your tests as usual while exporting the secret MERGIFY_TOKEN as an environment variable.

Add the following to the GitHub Actions step running your tests:

env:
  MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}

For example:

- name: Run Tests 🧪
  env:
    MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
  run: pytest

Set MERGIFY_TOKEN in the environment of the agents running your tests. The step itself then needs no Mergify-specific configuration:

steps:
  - label: "Run Tests 🧪"
    command: pytest

The plugin collects your test results and sends them to Test Insights.

The plugin applies quarantine inside the pytest session. It marks each quarantined test as a non-strict xfail, so a quarantined test that fails is reported as xfailed and does not count against pytest’s exit code. The exit code of your test step already accounts for quarantine, so the step needs nothing more:

• Do not add continue-on-error: true. The recipes that upload a JUnit report with the mergifyio/gha-mergify-ci action need it because the action decides the job’s result after the tests. Here nothing does, so it would let every real failure through.

• There is no step id to set and no test_step_outcome to pass: both belong to that action, which this setup does not use.

A crash cannot pass for a green run either. If pytest dies mid-run or fails outside a test, such as a collection error, it exits non-zero and the step fails. The plugin uploads results when the session ends, so a process killed outright before then, such as by the out-of-memory killer, sends nothing to Test Insights. A run that stops early without the process being killed, such as an interrupted session or one cut short by --maxfail, still uploads the results collected up to that point.

If the plugin cannot fetch the quarantine list, it quarantines nothing for that run, and a quarantined test that fails makes the step fail as usual.

If you’re using Tox to manage test environments, you can still use pytest-mergify by passing the MERGIFY_TOKEN and the rest of the CI environment variables into the test environment. On Buildkite the token comes from the agent’s environment, as above, so only the GitHub Actions step names it.

In your CI workflow:

# GitHub Actions
- name: Run Tox Tests
  env:
    MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
  run: tox

# Buildkite
steps:
  - label: "Run Tox Tests"
    command: tox

In your tox.ini, make sure the plugin is included in your testenv dependencies:

[testenv]
# You need to pass the MERGIFY_*, CI, GITHUB_*, etc variables
passenv = *
deps =
    pytest
    pytest-mergify
commands = pytest

If you’re using multiple environments (e.g. py38, py39, etc.), the plugin will work for all of them as long as the token is set correctly.

If you’re running multiple Tox environments (e.g., py38, py39, etc.), we recommend setting the MERGIFY_TEST_JOB_NAME environment variable to identify each environment’s report in Test Insights:

In your CI workflow:

# GitHub Actions
- name: Run Tox Tests
  env:
    MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
    MERGIFY_TEST_JOB_NAME: tox-${{ matrix.python-version }}
  run: tox

# Buildkite
steps:
  - label: "Run Tox Tests ({{matrix}})"
    command: tox
    matrix:
      - "3.10"
      - "3.11"
      - "3.12"
    env:
      MERGIFY_TEST_JOB_NAME: "tox-{{matrix}}"

After pushing these changes, your next CI run reports its pytest results automatically.

You can then review your test results, including any failures or flaky tests, directly in the Test Insights dashboard.