Run Tests
You can run TestCafe tests from the command line or JavaScript/TypeScript API.
testcafe safari ./tests/my-fixture.js
const createTestCafe = require('testcafe');
const testCafe = await createTestCafe('localhost', 1337, 1338);
try {
const runner = testCafe.createRunner();
await runner
.src('./tests/my-fixture.js')
.browsers('safari')
.run();
}
finally {
await testCafe.close();
}
TestCafe also allows you to create a configuration file where you can define test run settings. You can then omit these settings in the command line or API to use values from the configuration file.
- Specify Tests to Run
- Specify Target Browsers
- Specify the Report Format
- Customize Screenshot and Video Settings
- Run Tests Concurrently
- Stage the Tested App
- Specify a Proxy URL
- Live Mode
- Quarantine Mode
- Troubleshooting
Important
Front-end development tools (such as React DevTools or Vue DevTools) can interfere with TestCafe and cause errors. Do not open them while you run or debug TestCafe tests.
Specify Tests to Run
You should specify a path to a file or directory with tests you want to run in the second command line argument:
testcafe chrome ./tests/
In the API, use the runner.src method:
await runner
.browsers('chrome')
.src('./tests/')
.run();
Related configuration file property: src
Run Tests from Multiple Sources
You can specify multiple test files or directories:
testcafe safari ./js-tests/fixture.js ./studio-tests/fixture.testcafe
await runner
.browsers('safari')
.src(['./js-tests/fixture.js', './studio-tests/fixture.testcafe'])
.run();
Use Glob Patterns
TestCafe also supports glob patterns to run a set of files that match a specified pattern:
testcafe firefox ./tests/*mobile*
await runner
.browsers('firefox')
.src('./tests/*mobile*')
.run();
Filter Tests and Fixtures by Name
Use the -t (--test) command line argument or runner.filter method to run a test by name:
testcafe safari ./tests/sample-fixture.js -t "Click a label"
await runner
.browsers('safari')
.src('./tests/sample-fixture.js')
.filter(testName => testName === 'Click a label')
.run();
Related configuration file property: filter.test
You can also use the -T (--test-grep) argument to specify a grep
pattern for the test name:
testcafe chrome ./my-tests/ -T "Click.*"
Related configuration file property: filter.testGrep
To run a fixture by name, use the -f (--fixture) argument:
testcafe firefox ./my-tests/ -f "Sample fixture"
The runner.filter method’s predicate accepts the fixtureName
parameter:
await runner
.browsers('firefox')
.src('./my-tests/')
.filter((testName, fixtureName) => fixtureName === 'Sample fixture')
.run();
Related configuration file property: filter.fixture
To use grep
patterns for a fixture name, specify the -F (--fixture-grep) option:
testcafe safari ./my-tests/ -F "Page.*"
Related configuration file property: filter.fixtureGrep
Filter Tests and Fixtures by Metadata
You can also run tests whose metadata contains specific values. Use the --test-meta argument to do this:
testcafe chrome ./my-tests/ --test-meta device=mobile,env=production
The runner.filter method’s predicate accepts the testMeta
parameter:
await runner
.browsers('chrome')
.src('./my-tests/')
.filter((testName, fixtureName, fixturePath, testMeta) => {
return testMeta.device === 'mobile' && testMeta.env === 'production';
})
.run();
Related configuration file property: filter.testMeta
To filter fixtures with specific metadata, use the --fixture-meta argument:
testcafe firefox ./my-tests/ --fixture-meta device=mobile,env=production
In runner.filter, the fixture metadata is available in the fixtureMeta
parameter:
await runner
.browsers('firefox')
.src('./my-tests/')
.filter((testName, fixtureName, fixturePath, testMeta, fixtureMeta) => {
return fixtureMeta.device === 'mobile' && fixtureMeta.env === 'production';
})
.run();
Related configuration file property: filter.fixtureMeta
Specify Target Browsers
You should specify the browsers where you want to run tests in the first command line parameter.
TestCafe automatically detects supported browsers installed on the local machine. Use browser aliases to identify the target browser:
testcafe chrome ./tests/
In the API, use the runner.browsers method:
await runner
.browsers('chrome')
.src('./tests/')
.run();
Related configuration file property: browsers
Use Multiple Browsers
You can run tests in several browsers. In the command line, specify a comma-separated browser list:
testcafe safari,chrome ./tests/
In the API, pass an array of browser identifiers to runner.browsers:
await runner
.browsers(['safari', 'chrome'])
.src('./tests/')
.run();
Run Tests in All Installed Browsers
Use the all
alias to run tests in all locally installed browsers TestCafe can detect:
testcafe all ./tests/
await runner
.browsers('all')
.src('./tests/')
.run();
Test in Portable Browsers
You can also specify the path to a browser executable to launch portable browsers. Use the path:
prefix followed by the full path:
testcafe path:d:\firefoxportable\firefoxportable.exe ./tests/
await runner
.browsers('path:d:\firefoxportable\firefoxportable.exe')
.src('./tests/')
.run();
Note
If the path contains spaces, you should escape them in the command line as described in Command Line Interface.
Use Headless Mode
TestCafe can run tests in headless mode in browsers that support it. To run tests in headless mode, put the :headless
suffix after the browser name:
testcafe firefox:headless ./tests/
await runner
.browsers('firefox:headless')
.src('./tests/')
.run();
Enable Mobile Device Emulation
You can use Google Chrome mobile device emulation to test mobile layout and features on desktops. Specify the :emulation
postfix followed by emulation options:
testcafe "chrome:emulation:device=iphone X" ./tests/sample-fixture.js
await runner
.browsers('chrome:emulation:device=iphone X')
.src('./tests/sample-fixture.js')
.run();
Test in Cloud Testing Services
TestCafe can also run tests in cloud testing services such as BrowserStack or SauceLabs. Install the browser provider for your service and specify the browser alias as described in the browser provider documentation.
For instance, to use SauceLabs, install the testcafe-browser-provider-saucelabs module from npm
and run tests as follows:
testcafe "saucelabs:Chrome@52.0:Windows 8.1" ./tests/sample-fixture.js
await runner
.browsers('saucelabs:Chrome@52.0:Windows 8.1')
.src('./tests/sample-fixture.js')
.run();
Test on Remote and Mobile Devices
To run tests remotely on a mobile device or a computer with no TestCafe installation, specify remote
instead of the browser alias in the command line:
testcafe remote ./tests/sample-fixture.js
TestCafe generates a URL and displays it in the console. When you visit this URL from the remote device, TestCafe runs tests in this browser. To run tests in several remote browsers, specify their number after the remote
keyword: remote:2
or remote:4
.
In the API, create a remote browser connection with the testcafe.createBrowserConnection method, visit the generated URL and run tests once the connection is initialized:
const createTestCafe = require('testcafe');
const testCafe = await createTestCafe('localhost', 1337, 1338);
const runner = testCafe.createRunner();
const remoteConnection = testcafe.createBrowserConnection();
// Visit this URL from the remote device.
console.log(remoteConnection.url);
// Wait until the remote device's browser connects.
await new Promise(resolve => remoteConnection.once('ready', resolve));
await runner
.src('./tests/sample-fixture.js')
.browsers(remoteConnection)
.run();
Specify the Report Format
A reporter is a module that formats and outputs test run results. TestCafe ships with five basic reporters, including reporters for spec, JSON, and xUnit formats. You can install other reporters as plugins or create a custom reporter.
Use the -r (--reporter) flag in the command line and the runner.reporter method in the API to specify which reporter to use.
testcafe all ./tests/sample-fixture.js -r xunit
await runner
.browsers('all')
.src('./tests/sample-fixture.js')
.reporter('xunit')
.run();
testcafe all ./tests/sample-fixture.js -r my-reporter
await runner
.browsers('all')
.src('./tests/sample-fixture.js')
.reporter('my-reporter')
.run();
Related configuration file property: reporter
To define the output target, specify it after a semicolon in the command line or as the second parameter in runner.reporter.
testcafe all ./tests/sample-fixture.js -r json:report.json
await runner
.browsers('all')
.src('./tests/sample-fixture.js')
.reporter('json', 'report.json')
.run();
You can use multiple reporters, but only one reporter can write to stdout
:
testcafe all ./tests/sample-fixture.js -r spec,xunit:report.xml
await runner
.browsers('all')
.src('./tests/sample-fixture.js')
.reporter(['spec', {
name: 'xunit',
output: 'report.xml'
})
.run();
Customize Screenshot and Video Settings
TestCafe can take screenshots of the tested page automatically when a test fails. You can also capture screenshots at arbitrary moments with the t.takeScreenshot and t.takeElementScreenshot actions.
Use the -s (--screenshots) command line flag or the runner.screenshots API method.
testcafe all ./tests/sample-fixture.js -s path=artifacts/screenshots,takeOnFails=true
await runner
.browsers('all')
.src('./tests/sample-fixture.js')
.screenshots({
path: 'artifacts/screenshots',
takeOnFails: true
})
.run();
Related configuration file property: screenshots
To record videos of test runs, pass the --video command line flag or use the runner.video API method.
You can also specify video recording options in the --video-options command line argument or a runner.video parameter:
testcafe chrome ./test.js --video ./videos/ --video-options singleFile=true,failedOnly=true
await runner
.browsers('chrome')
.src('./test.js')
.video('./videos/', {
singleFile: true,
failedOnly: true
})
.run();
Related configuration file properties:
Run Tests Concurrently
To speed up testing, TestCafe allows you to execute tests concurrently. In concurrent mode, TestCafe invokes multiple instances of each browser. For example, the testcafe chrome, firefox test.js --concurrency 3
command invokes 3 Chrome instances and 3 Firefox instances (6 browser instances in total). These instances constitute the pool of browsers against which tests run concurrently, that is each test runs in the first available instance.
TestCafe invokes the specified number of browser instances despite of the number of tests in the tested fixture. Empty browser instances that run no tests remain on the screen. For example, if you specify the --concurrency 3
CLI flag for a fixture with 2 tests, TestCafe runs two tests in two browser instances and keeps the third instance in standby mode.
TestCafe executes fixture hooks before all tests run (the before
hook) and when all tests in the fixture are complete (the after
hook).
Hooks in a fixture are synchronization points between fixture executions. TestCafe executes the next fixture only when all tests and hooks in the previous fixture are complete.
To enable concurrency, use the -c (--concurrency) command line option or the runner.concurrency API method.
Important
Microsoft Edge Legacy does not support concurrent test execution, because it cannot be started in a new window, not can it open a specified URL.
The following CLI command invokes three Chrome instances and runs tests concurrently:
testcafe -c 3 chrome tests/test.js
To run tests with the same settings in code, use the following code sample:
var testRunPromise = runner
.src('tests/test.js')
.browsers('chrome')
.concurrency(3)
.run();
Related configuration file property: concurrency
Note that you can also use concurrency when testing against multiple browsers.
testcafe -c 4 safari,firefox tests/test.js
In this case, TestCafe distributes tests across four Safari instances and runs the same tests in four Firefox instances.
Note
If an uncaught error or unhandled promise rejection occurs on the server during test execution, all concurrently running tests fail.
Use Concurrency on Remote Devices
When you run tests on remote devices, create browser connections for each instance of each browser you test against. The total number of instances is c*n, where c
is the concurrency factor and n
is the number of browsers.
In the command line interface, specify the number of browser instances after the remote:
keyword. For example, use -c 2 remote:6
to run tests in Chrome, Firefox, and Edge, with two instances of each browser. If you use the API, create a browser connection for each instance in code.
Launch all the required browser instances manually on a remote device. The total number of instances should be divisible by the concurrency factor c
. Otherwise, an exception is thrown.
testcafe -c 2 remote:4 tests/test.js
If you test against multiple remote browsers, open and connect all instances of one browser before you connect the next browser.
Stage the Tested App
TestCafe can execute a specified shell command before it starts tests. For instance, you can run a command that starts a local web server and deploys the tested app. TestCafe automatically terminates the process when tests are finished.
Use the -a (--app) CLI flag or the runner.startApp API method to provide a command:
testcafe chrome ./my-tests/ --app "node server.js"
await runner
.browsers('chrome')
.src('./my-tests/')
.startApp('node server.js')
.run();
Related configuration file property: appCommand
TestCafe delays tests to allow the shell command to execute. The default timeout is 1000 milliseconds. Use the --app-init-delay CLI flag or a runner.startApp parameter to specify the timeout value.
testcafe chrome ./my-tests/ --app "node server.js" --app-init-delay 4000
await runner
.browsers('chrome')
.src('./my-tests/')
.startApp('node server.js', 4000)
.run();
Related configuration file property: appInitDelay
Specify a Proxy URL
If your network uses a proxy to access the internet, specify the proxy URL to TestCafe. Use the --proxy command line argument or the runner.useProxy API method:
testcafe chrome ./my-tests/ --proxy proxy.mycompany.com
await runner
.browsers('chrome')
.src('./my-tests/')
.useProxy('proxy.mycompany.com')
.run();
Related configuration file property: proxy
You can also specify URLs that should be accessed directly. Pass the list of URLs in the --proxy-bypass command line argument or a runner.useProxy parameter:
testcafe chrome ./my-tests/ --proxy proxy.corp.mycompany.com --proxy-bypass localhost:8080
await runner
.browsers('chrome')
.src('./my-tests/')
.useProxy('proxy.corp.mycompany.com', 'localhost:8080')
.run();
Related configuration file property: proxyBypass
Live Mode
Live mode ensures TestCafe and the browsers remain active while you work on tests. You can see test results instantly because the tests are restarted when you make changes.
Note
Live Mode replaces the deprecated testcafe-live
module.
How to Enable Live Mode
Use the -L (--live) flag to enable live mode from the command line interface.
testcafe chrome tests/test.js -L
In the API, create a live mode runner with the testcafe.createLiveModeRunner function and use it instead of a regular test runner.
const createTestCafe = require('testcafe');
const testcafe = await createTestCafe('localhost', 1337, 1338);
try {
const liveRunner = testcafe.createLiveModeRunner();
await liveRunner
.src('tests/test.js')
.browsers('chrome')
.run();
}
finally {
await testcafe.close();
}
How Live Mode Works
When you run tests with live mode enabled, TestCafe opens the browsers, runs the tests, shows the reports, and waits for further actions.
Then TestCafe starts watching for changes in the test files and all files referenced in them (like page objects or helper modules). Once you make changes in any of those files and save them, TestCafe immediately reruns the tests.
When the tests are done, the browsers stay on the last opened page so you can work with it and explore it with the browser’s developer tools.
You can use live mode with any browsers: local, remote, mobile or headless.
Important
Live mode is designed to work with tests locally. Do not use it in CI systems.
Tip
Use the only function to work with a single test.
Console Shortcuts in Live Mode
Ctrl+S
- stops the current test run;Ctrl+R
- restarts the current test run;Ctrl+W
- turns off/on the file watcher;Ctrl+C
- closes opened browsers and terminates the process.
Quarantine mode
Enable quarantine mode to eliminate false negatives and detect unstable tests. When a test fails, TestCafe quarantines it, and repeats it until the test yields conclusive results.
Note
Quarantine mode may increase the total run-time of your test suite.
Quarantine mode options
The successThreshold
option (default value: 3) — the number of successful attempts necessary to confirm the test’s success.
The attemptLimit
option (default value: 5) — the maximum number of test attempts.
Note
The attemptLimit
has to be greater than the successThreshold
.
Success conditions
Quarantined tests succeed when the number of successful attempts reaches the successThreshold
value.
Failure conditions
Quarantined tests fail as soon as it becomes impossible for them to succeed within the attemptLimit
.
For example, the test fails immediately if:
a) The number of successes is 0
.
b) You need 3
successful attempts to reach the successThreshold
.
c) You have fewer than 3
attempts left (before you reach the attemptLimit
).
Stability status
If consecutive runs of a quarantined test yield varying outcomes, TestCafe marks the test as unstable. This means that the test’s outcome is, in all likelihood, inconclusive.
- If test results do not change over time (the test succeeds or fails several times in a row), the test is stable, regardless of its success. You can trust the results.
- If a test alternates between successes and failures, TestCafe marks the test as unstable. There is probably something wrong with the test.
Example
Enable quarantine mode with the -q (--quarantine-mode) command line flag, or the quarantineMode
runner option:
testcafe chrome ./tests/ -q
testcafe chrome ./tests/ -q attemptLimit=5,successThreshold=2
await runner
.browsers('chrome')
.src('./tests/')
.run({ quarantineMode: true });
await runner
.browsers('chrome')
.src('./tests/')
.run({ quarantineMode: { successThreshold: 1, attemptLimit: 3 } });
See Martin Fowler’s Eradicating Non-Determinism in Tests article for more information about non-deterministic tests.
Troubleshooting
Disable the ‘Close Multiple Tabs’ Popup in IE
Internet Explorer displays a dialog when you try to close a browser window with more than one opened tab. This dialog blocks the page and pauses tests. Disable this dialog before you run tests.
To disable this dialog, open multiple tabs in one Internet Explorer window and close the browser. Check Always close all tabs
and click Close all tabs
. IE saves this setting and the dialog does not appear again.
Repeat this action on any new machine where you run tests.