Quarantine Mode

Enable quarantine mode to eliminate false negatives and detect unstable tests. When a test fails, TestCafe quarantines it, and repeats it until the test yields conclusive results.

Enable quarantine mode with the -q (--quarantine-mode) command line flag, the quarantineMode configuration file setting, or the quarantineMode Test Runner option:

• Command Line Interface

  testcafe chrome ./tests/ -q
  testcafe chrome ./tests/ -q attemptLimit=5,successThreshold=2
  

• Configuration File

  {
      "quarantineMode": true
  }
  

• TestCafe Test Runner API

  await runner
      .browsers('chrome')
      .src('./tests/')
      .run({ quarantineMode: true });
  
  await runner
      .browsers('chrome')
      .src('./tests/')
      .run({ quarantineMode: { successThreshold: 1, attemptLimit: 3 } });
  

See Martin Fowler’s Eradicating Non-Determinism in Tests article for more information on non-deterministic tests.

Quarantine Mode Options

The successThreshold option (default value: 3) is the number of successful attempts necessary to confirm the test’s success. The attemptLimit option (default value: 5) is the maximum number of test attempts.

Success and Failure Conditions

Quarantined tests succeed when the number of successful attempts reaches the successThreshold value.

Quarantined tests fail as soon as it becomes impossible for them to succeed within the attemptLimit.

For example, the test fails immediately after the following conditions are met:

1. The number of successes is 0.
2. You need 3 successful attempts to reach the successThreshold.
3. You have fewer than 3 attempts left (before you reach the attemptLimit).

Stability Status

TestCafe tracks consecutive test results to try and determine if the test outcome is conclusive.

• If test results do not change over time (the test succeeds or fails several times in a row), the test is stable, regardless of its success. You can trust the results.
• If a test alternates between successes and failures, TestCafe marks the test as unstable. There is probably something wrong with the test.