Run Tests

You can run TestCafe tests from the command line or JavaScript/TypeScript API.

testcafe safari ./tests/my-fixture.js

const createTestCafe = require('testcafe');

const testCafe = await createTestCafe('localhost', 1337, 1338);

try {
    const runner = testCafe.createRunner();

    await runner
        .src('./tests/my-fixture.js')
        .browsers('safari')
        .run();
}
finally {
    await testCafe.close();
}

TestCafe also allows you to create a configuration file where you can define test run settings. You can then omit these settings in the command line or API to use values from the configuration file.

• Specify Tests to Run
  • Run Tests from Multiple Sources
  • Use Glob Patterns
  • Filter Tests and Fixtures by Name
  • Filter Tests and Fixtures by Metadata
• Specify Target Browsers
  • Use Multiple Browsers
  • Run Tests in All Installed Browsers
  • Test in Portable Browsers
  • Use Headless Mode
  • Enable Mobile Device Emulation
  • Test in Cloud Testing Services
  • Test on Remote and Mobile Devices
• Specify the Report Format
• Customize Screenshot and Video Settings
• Run Tests Concurrently
  • Use Concurrency on Remote Devices
• Stage the Tested App
• Specify a Proxy URL
• Live Mode
  • How Live Mode Works
  • Console Shortcuts in Live Mode
• Quarantine Mode
• Troubleshooting
  • Disable the ‘Close Multiple Tabs’ Popup in IE

Specify Tests to Run

You should specify a path to a file or directory with tests you want to run in the second command line argument:

testcafe chrome ./tests/

In the API, use the runner.src method:

await runner
    .browsers('chrome')
    .src('./tests/')
    .run();

Related configuration file property: src

Run Tests from Multiple Sources

You can specify multiple test files or directories:

testcafe safari ./js-tests/fixture.js ./studio-tests/fixture.testcafe

await runner
    .browsers('safari')
    .src(['./js-tests/fixture.js', './studio-tests/fixture.testcafe'])
    .run();

Use Glob Patterns

TestCafe also supports glob patterns to run a set of files that match a specified pattern:

testcafe firefox ./tests/*mobile*

await runner
    .browsers('firefox')
    .src('./tests/*mobile*')
    .run();

Filter Tests and Fixtures by Name

Use the -t (--test) command line argument or runner.filter method to run a test by name:

testcafe safari ./tests/sample-fixture.js -t "Click a label"

await runner
    .browsers('safari')
    .src('./tests/sample-fixture.js')
    .filter(testName => testName === 'Click a label')
    .run();

Related configuration file property: filter.test

You can also use the -T (--test-grep) argument to specify a grep pattern for the test name:

testcafe chrome ./my-tests/ -T "Click.*"

Related configuration file property: filter.testGrep

To run a fixture by name, use the -f (--fixture) argument:

testcafe firefox ./my-tests/ -f "Sample fixture"

The runner.filter method’s predicate accepts the fixtureName parameter:

await runner
    .browsers('firefox')
    .src('./my-tests/')
    .filter((testName, fixtureName) => fixtureName === 'Sample fixture')
    .run();

Related configuration file property: filter.fixture

To use grep patterns for a fixture name, specify the -F (--fixture-grep) option:

testcafe safari ./my-tests/ -F "Page.*"

Related configuration file property: filter.fixtureGrep

Filter Tests and Fixtures by Metadata

You can also run tests whose metadata contains specific values. Use the --test-meta argument to do this:

testcafe chrome ./my-tests/ --test-meta device=mobile,env=production

The runner.filter method’s predicate accepts the testMeta parameter:

await runner
    .browsers('chrome')
    .src('./my-tests/')
    .filter((testName, fixtureName, fixturePath, testMeta) => {
        return testMeta.device === 'mobile' && testMeta.env === 'production';
    })
    .run();

Related configuration file property: filter.testMeta

To filter fixtures with specific metadata, use the --fixture-meta argument:

testcafe firefox ./my-tests/ --fixture-meta device=mobile,env=production

In runner.filter, the fixture metadata is available in the fixtureMeta parameter:

await runner
    .browsers('firefox')
    .src('./my-tests/')
    .filter((testName, fixtureName, fixturePath, testMeta, fixtureMeta) => {
        return fixtureMeta.device === 'mobile' && fixtureMeta.env === 'production';
    })
    .run();

Related configuration file property: filter.fixtureMeta

Specify Target Browsers

You should specify the browsers where you want to run tests in the first command line parameter.

TestCafe automatically detects supported browsers installed on the local machine. Use browser aliases to identify the target browser:

testcafe chrome ./tests/

In the API, use the runner.browsers method:

await runner
    .browsers('chrome')
    .src('./tests/')
    .run();

Related configuration file property: browsers

Use Multiple Browsers

You can run tests in several browsers. In the command line, specify a comma-separated browser list:

testcafe safari,chrome ./tests/

In the API, pass an array of browser identifiers to runner.browsers:

await runner
    .browsers(['safari', 'chrome'])
    .src('./tests/')
    .run();

Run Tests in All Installed Browsers

Use the all alias to run tests in all locally installed browsers TestCafe can detect:

testcafe all ./tests/

await runner
    .browsers('all')
    .src('./tests/')
    .run();

Test in Portable Browsers

You can also specify the path to a browser executable to launch portable browsers. Use the path: prefix followed by the full path:

testcafe path:d:\firefoxportable\firefoxportable.exe ./tests/

await runner
    .browsers('path:d:\firefoxportable\firefoxportable.exe')
    .src('./tests/')
    .run();

Use Headless Mode

TestCafe can run tests in headless mode in browsers that support it. To run tests in headless mode, put the :headless suffix after the browser name:

testcafe firefox:headless ./tests/

await runner
    .browsers('firefox:headless')
    .src('./tests/')
    .run();

Enable Mobile Device Emulation

You can use Google Chrome mobile device emulation to test mobile layout and features on desktops. Specify the :emulation postfix followed by emulation options:

testcafe "chrome:emulation:device=iphone X" ./tests/sample-fixture.js

await runner
    .browsers('chrome:emulation:device=iphone X')
    .src('./tests/sample-fixture.js')
    .run();

Test in Cloud Testing Services

TestCafe can also run tests in cloud testing services such as BrowserStack or SauceLabs. Install the browser provider for your service and specify the browser alias as described in the browser provider documentation.

For instance, to use SauceLabs, install the testcafe-browser-provider-saucelabs module from npm and run tests as follows:

testcafe "saucelabs:Chrome@52.0:Windows 8.1" ./tests/sample-fixture.js

await runner
    .browsers('saucelabs:Chrome@52.0:Windows 8.1')
    .src('./tests/sample-fixture.js')
    .run();

Test on Remote and Mobile Devices

To run tests remotely on a mobile device or a computer with no TestCafe installation, specify remote instead of the browser alias in the command line:

testcafe remote ./tests/sample-fixture.js

TestCafe generates a URL and displays it in the console. When you visit this URL from the remote device, TestCafe runs tests in this browser. To run tests in several remote browsers, specify their number after the remote keyword: remote:2 or remote:4.

In the API, create a remote browser connection with the testcafe.createBrowserConnection method, visit the generated URL and run tests once the connection is initialized:

const createTestCafe   = require('testcafe');
const testCafe         = await createTestCafe('localhost', 1337, 1338);
const runner           = testCafe.createRunner();
const remoteConnection = testcafe.createBrowserConnection();

// Visit this URL from the remote device.
console.log(remoteConnection.url);

// Wait until the remote device's browser connects.
await new Promise(resolve => remoteConnection.once('ready', resolve));

await runner
    .src('./tests/sample-fixture.js')
    .browsers(remoteConnection)
    .run();

Specify the Report Format

A reporter is a module that formats and outputs test run results. TestCafe ships with five basic reporters, including reporters for spec, JSON, and xUnit formats. You can install other reporters as plugins or create a custom reporter.

Use the -r (--reporter) flag in the command line and the runner.reporter method in the API to specify which reporter to use.

testcafe all ./tests/sample-fixture.js -r xunit

await runner
    .browsers('all')
    .src('./tests/sample-fixture.js')
    .reporter('xunit')
    .run();

testcafe all ./tests/sample-fixture.js -r my-reporter

await runner
    .browsers('all')
    .src('./tests/sample-fixture.js')
    .reporter('my-reporter')
    .run();

Related configuration file property: reporter

To define the output target, specify it after a semicolon in the command line or as the second parameter in runner.reporter.

testcafe all ./tests/sample-fixture.js -r json:report.json

await runner
    .browsers('all')
    .src('./tests/sample-fixture.js')
    .reporter('json', 'report.json')
    .run();

You can use multiple reporters, but only one reporter can write to stdout:

testcafe all ./tests/sample-fixture.js -r spec,xunit:report.xml

await runner
    .browsers('all')
    .src('./tests/sample-fixture.js')
    .reporter(['spec', {
        name: 'xunit',
        output: 'report.xml'
    })
    .run();

Customize Screenshot and Video Settings

TestCafe can take screenshots of the tested page automatically when a test fails. You can also capture screenshots at arbitrary moments with the t.takeScreenshot and t.takeElementScreenshot actions.

Use the -s (--screenshots) command line flag or the runner.screenshots API method.

testcafe all ./tests/sample-fixture.js -s path=artifacts/screenshots,takeOnFails=true

await runner
    .browsers('all')
    .src('./tests/sample-fixture.js')
    .screenshots({
        path: 'artifacts/screenshots',
        takeOnFails: true
    })
    .run();

Related configuration file property: screenshots

To record videos of test runs, pass the --video command line flag or use the runner.video API method.

You can also specify video recording options in the --video-options command line argument or a runner.video parameter:

testcafe chrome ./test.js --video ./videos/ --video-options singleFile=true,failedOnly=true

await runner
    .browsers('chrome')
    .src('./test.js')
    .video('./videos/', {
        singleFile: true,
        failedOnly: true
    })
    .run();

Related configuration file properties:

• videoPath
• videoOptions

Run Tests Concurrently

To save time spent on testing, TestCafe allows you to execute tests concurrently. In concurrent mode, TestCafe invokes multiple instances of each browser. These instances constitute the pool of browsers against which tests run concurrently, i.e. each test runs in the first available instance.

To enable concurrency, use the -c (--concurrency) command line option or the runner.concurrency API method.

The following command invokes three Chrome instances and runs tests concurrently.

testcafe -c 3 chrome tests/test.js

This is how the same thing can be done through the API.

var testRunPromise = runner
    .src('tests/test.js')
    .browsers('chrome')
    .concurrency(3)
    .run();

Related configuration file property: concurrency

Note that you can also use concurrency when testing against multiple browsers.

testcafe -c 4 safari,firefox tests/test.js

In this case, tests are distributed across four Safari instances and the same tests are also run in four Firefox instances.

Use Concurrency on Remote Devices

When you run tests on remote devices, create browser connections for each instance of each browser you test against. The total number of instances is c*n, where c is the concurrency factor and n is the number of browsers.

In the command line interface, specify the number of browser instances after the remote: keyword. For example, use -c 2 remote:6 to run tests in Chrome, Firefox, and Edge, with two instances of each browser. If you use the API, create a browser connection for each instance in code.

Launch all the required browser instances manually on a remote device. The total number of instances should be divisible by the concurrency factor c. Otherwise, an exception is thrown.

testcafe -c 2 remote:4 tests/test.js

If you test against multiple remote browsers, open and connect all instances of one browser before you connect the next browser.

Stage the Tested App

TestCafe can execute a specified shell command before it starts tests. For instance, you can run a command that starts a local web server and deploys the tested app. TestCafe automatically terminates the process when tests are finished.

Use the -a (--app) CLI flag or the runner.startApp API method to provide a command:

testcafe chrome ./my-tests/ --app "node server.js"

await runner
    .browsers('chrome')
    .src('./my-tests/')
    .startApp('node server.js')
    .run();

Related configuration file property: appCommand

TestCafe delays tests to allow the shell command to execute. The default timeout is 1000 milliseconds. Use the --app-init-delay CLI flag or a runner.startApp parameter to specify the timeout value.

testcafe chrome ./my-tests/ --app "node server.js" --app-init-delay 4000

await runner
    .browsers('chrome')
    .src('./my-tests/')
    .startApp('node server.js', 4000)
    .run();

Related configuration file property: appInitDelay

Specify a Proxy URL

If your network uses a proxy to access the internet, specify the proxy URL to TestCafe. Use the --proxy command line argument or the runner.useProxy API method:

testcafe chrome ./my-tests/ --proxy proxy.mycompany.com

await runner
    .browsers('chrome')
    .src('./my-tests/')
    .useProxy('proxy.mycompany.com')
    .run();

Related configuration file property: proxy

You can also specify URLs that should be accessed directly. Pass the list of URLs in the --proxy-bypass command line argument or a runner.useProxy parameter:

testcafe chrome ./my-tests/ --proxy proxy.corp.mycompany.com --proxy-bypass localhost:8080

await runner
    .browsers('chrome')
    .src('./my-tests/')
    .useProxy('proxy.corp.mycompany.com', 'localhost:8080')
    .run();

Related configuration file property: proxyBypass

Live Mode

Live mode ensures TestCafe and the browsers remain active while you work on tests. You can see test results instantly because the tests are restarted when you make changes.

How to Enable Live Mode

Use the -L (--live) flag to enable live mode from the command line interface.

testcafe chrome tests/test.js -L

In the API, create a live mode runner with the testcafe.createLiveModeRunner function and use it instead of a regular test runner.

const createTestCafe = require('testcafe');

const testcafe = await createTestCafe('localhost', 1337, 1338);

try {
    const liveRunner = testcafe.createLiveModeRunner();

    await liveRunner
        .src('tests/test.js')
        .browsers('chrome')
        .run();
}
finally {
    await testcafe.close();
}

How Live Mode Works

When you run tests with live mode enabled, TestCafe opens the browsers, runs the tests, shows the reports, and waits for further actions.

Then TestCafe starts watching for changes in the test files and all files referenced in them (like page objects or helper modules). Once you make changes in any of those files and save them, TestCafe immediately reruns the tests.

When the tests are done, the browsers stay on the last opened page so you can work with it and explore it with the browser’s developer tools.

You can use live mode with any browsers: local, remote, mobile or headless.

Console Shortcuts in Live Mode

• Ctrl+S - stops the current test run;
• Ctrl+R - restarts the current test run;
• Ctrl+W - turns off/on the file watcher;
• Ctrl+C - closes opened browsers and terminates the process.

Quarantine mode

Enable quarantine mode to eliminate false negatives and detect unstable tests. When a test fails, TestCafe quarantines it, and repeats it until the test yields conclusive results.

Quarantine mode options

The successThreshold option (default value: 3) — the number of successful attempts necessary to confirm the test’s success. The attemptLimit option (default value: 5) — the maximum number of test attempts.

Success conditions

Quarantined tests succeed when the number of successful attempts reaches the successThreshold value.

Failure conditions

Quarantined tests fail as soon as it becomes impossible for them to succeed within the attemptLimit.

For example, the test fails immediately if: a) The number of successes is 0. b) You need 3 successful attempts to reach the successThreshold. c) You have fewer than 3 attempts left (before you reach the attemptLimit).

Stability status

If consecutive runs of a quarantined test yield varying outcomes, TestCafe marks the test as unstable. This means that the test’s outcome is, in all likelihood, inconclusive.

• If test results do not change over time (the test succeeds or fails several times in a row), the test is stable, regardless of its success. You can trust the results.
• If a test alternates between successes and failures, TestCafe marks the test as unstable. There is probably something wrong with the test.

Example

Enable quarantine mode with the -q (--quarantine-mode) command line flag, or the quarantineMode runner option:

testcafe chrome ./tests/ -q
testcafe chrome ./tests/ -q attemptLimit=5,successThreshold=2

await runner
    .browsers('chrome')
    .src('./tests/')
    .run({ quarantineMode: true });

await runner
    .browsers('chrome')
    .src('./tests/')
    .run({ quarantineMode: { successThreshold: 1, attemptLimit: 3 } });

See Martin Fowler’s Eradicating Non-Determinism in Tests article for more information about non-deterministic tests.

Troubleshooting

Disable the ‘Close Multiple Tabs’ Popup in IE

Internet Explorer displays a dialog when you try to close a browser window with more than one opened tab. This dialog blocks the page and pauses tests. Disable this dialog before you run tests.

To disable this dialog, open multiple tabs in one Internet Explorer window and close the browser. Check Always close all tabs and click Close all tabs. IE saves this setting and the dialog does not appear again.

Repeat this action on any new machine where you run tests.