pytest-xdist makes the suite faster and the flakes weirder

A pytest test that asserts on the contents of /tmp/test-output/report.txt passes locally. CI runs pytest -n auto across 4 workers and the test fails 30% of the time with FileNotFoundError. The file path is hardcoded. The test that creates the file is in a different module. The two tests share the same path string, and that string is the bug.

We see this pattern often enough on Mergify Test Insights that it earned its own slot in our flaky pytest catalog. The cause is shared external state across workers. The fix is per-worker namespacing, plus the right fixture scope.

What you see

WORKDIR = "/tmp/test-output"

def test_writes_report():
    with open(f"{WORKDIR}/report.txt", "w") as f:
        f.write("hello")

def test_reads_report():
    assert os.path.exists(f"{WORKDIR}/report.txt")

Sequential run, single worker: test_writes_report runs first, creates the file, test_reads_report runs next, finds it. Pass.

Under pytest -n auto, pytest-xdist distributes tests across worker processes. The two tests can land on different workers. The reader checks for the file before the writer creates it (or after the writer’s worker cleaned up the directory). The failure looks like a missing file but the actual cause is two parallel processes racing on a shared path.

The frustrating part: the shared path is invisible from the test code. WORKDIR looks like a constant. It is — the constant just refers to a piece of state outside the test process.

Why xdist’s distribution algorithm matters

pytest-xdist’s default scheduler is loadscope, which keeps tests from the same module on the same worker. That helps for module-level fixtures but does not help for shared external resources between modules.

The loadfile scheduler keeps tests from the same file on the same worker. Better for file-level coupling, still does not help for cross-file shared state.

The worksteal scheduler dynamically rebalances. It is the fastest for unbalanced suites but the most likely to expose cross-test races because it cares less about locality.

You can pick the scheduler with --dist:

pytest -n auto --dist worksteal

Pick the one that matches your suite’s reality. None of them help if your tests share a /tmp/test-output path.

The fix that holds

Use pytest’s built-in tmp_path fixture. It gives every test its own directory, scoped to that test:

def test_writes_report(tmp_path):
    (tmp_path / "report.txt").write_text("hello")

def test_reads_report(tmp_path):
    (tmp_path / "report.txt").write_text("hello")
    assert (tmp_path / "report.txt").exists()

Each test owns its directory. There is nothing shared, so there is no race. The directory cleans up automatically at the end of the test.

For test pairs that genuinely need to share a directory (a producer and a consumer), use tmp_path_factory at session scope:

@pytest.fixture(scope="session")
def shared_dir(tmp_path_factory):
    return tmp_path_factory.mktemp("shared")

def test_writes_report(shared_dir):
    (shared_dir / "report.txt").write_text("hello")

def test_reads_report(shared_dir):
    assert (shared_dir / "report.txt").exists()

The session-scoped fixture is created once per worker, and tmp_path_factory.mktemp includes the worker ID in the path. Workers cannot collide. Tests within the same worker see each other’s writes.

When the resource is genuinely process-wide

Some resources are not file paths. A test database, a Redis instance, a Kafka topic — these live outside the worker. pytest-xdist exposes a worker_id fixture so you can namespace by worker:

@pytest.fixture
def redis_namespace(worker_id):
    namespace = f"test-{worker_id}"
    yield namespace
    redis.delete_pattern(f"{namespace}:*")

Each worker gets its own Redis namespace. Cleanup is per-worker. The shared Redis instance no longer becomes a race.

For test databases under pytest-django or similar plugins, the standard pattern is one database per worker. The --reuse-db --create-db flags handle this once configured. Without per-worker databases, every test creates rows the others can see, and --dist worksteal will eventually surface a race.

How Mergify catches this before you ship

xdist races are easy to dismiss as “weird, retry the suite.” The retry usually wins because the next worker assignment is different. Teams learn to ignore the failure category, which is exactly when a real ordering bug ships to production.

Test Insights groups failures by their xdist worker ID. When a test only fails when assigned to worker 2 alongside a specific other test on the same worker, the dashboard surfaces the parallelism dimension: “test_reads_report fails when scheduled on the worker after test_writes_report’s cleanup.” You see the pair without manually bisecting or minimizing the failing set yourself.

Quarantine kicks in once the pattern is clear, so the merge queue keeps moving while you switch to tmp_path or worker_id-scoped resources.

Stop staring at xdist worker logs trying to find the shared resource. Point Mergify at your suite — the native pytest-mergify plugin installs in one pip install.

More patterns like this

Worker races are one of the eight patterns in the flaky-tests-in-pytest guide. The others are variants of the same theme: state that crosses tests in ways the test author did not anticipate. Fixture teardown order, autouse fixtures touching globals, monkeypatch leakage, async event-loop scope mismatches under pytest-asyncio. Different trigger, same root.

Most fixes are pytest’s built-in fixtures (tmp_path, monkeypatch, worker_id) that already exist for exactly this purpose. The bug is usually the test reaching outside what those fixtures cover.