Quarantine Mode

Enable quarantine mode to eliminate false negatives and detect unstable tests. When a test fails, TestCafe quarantines it, and repeats it until the test yields conclusive results.

Note

Quarantine mode may increase the total run-time of your test suite.

Enable quarantine mode with the -q (--quarantine-mode) command line flag, the quarantineMode configuration file setting, or the quarantineMode Test Runner option:

  • Command Line Interface
    testcafe chrome ./tests/ -q
    testcafe chrome ./tests/ -q attemptLimit=5,successThreshold=2
    
  • Configuration File
    {
        "quarantineMode": true
    }
    
  • TestCafe Test Runner API

    await runner
        .browsers('chrome')
        .src('./tests/')
        .run({ quarantineMode: true });
    
    await runner
        .browsers('chrome')
        .src('./tests/')
        .run({ quarantineMode: { successThreshold: 1, attemptLimit: 3 } });
    

See Martin Fowler’s Eradicating Non-Determinism in Tests article for more information on non-deterministic tests.

Quarantine Mode Options

The successThreshold option (default value: 3) is the number of successful attempts necessary to confirm the test’s success. The attemptLimit option (default value: 5) is the maximum number of test attempts.

Note

The attemptLimit has to be greater than the successThreshold.

Success and Failure Conditions

Quarantined tests succeed when the number of successful attempts reaches the successThreshold value.

Quarantined tests fail as soon as it becomes impossible for them to succeed within the attemptLimit.

For example, the test fails immediately after the following conditions are met:

  1. The number of successes is 0.
  2. You need 3 successful attempts to reach the successThreshold.
  3. You have fewer than 3 attempts left (before you reach the attemptLimit).

Stability Status

TestCafe tracks consecutive test results to try and determine if the test outcome is conclusive.

  • If test results do not change over time (the test succeeds or fails several times in a row), the test is stable, regardless of its success. You can trust the results.
  • If a test alternates between successes and failures, TestCafe marks the test as unstable. There is probably something wrong with the test.