TestCafe Goes Native

TestCafe v2.5.0 introduced an experimental mode that allows users to automate Chromium-based browsers, such as Google Chrome and Microsoft Edge, with the native CDP protocol. In TestCafe v3.0.0 and higher, native automation is the default setting.

Read this article to find out the reasons for this decision, and its impact on TestCafe users like you.

The TestCafe proxy: A solution that causes problems

DevExpress developed TestCafe to test the company’s commercial products across platforms, browsers, and devices. Back then, no tool on the market satisfied our needs, so we had to figure out our own way to implement cross-browser automation.

The TestCafe team solved this problem with testcafe-hammerhead — a URL-rewriting proxy. Hammerhead intercepts HTTP requests, and injects automation scripts into test pages. That way, TestCafe can control what happens in the browser. But the proxy solution has its faults.

• The URL rewriting submodule of the proxy is complex and requires a lot of maintainance. It has to identify, intercept, and modify every HTTP request. Request handling errors can lead to application malfunction, and unexpected test failure.

  Even if the URL rewriter works perfectly, it still slows down the framework. To keep this overhead to a minimum, the TestCafe team needs to expend considerable effort to optimize the submodule.

• The event emulation submodule is no less complex. To emulate user behavior the same way in every browser, it needs to account for each browser’s unique characteristics. Browser updates often upset this balance, and introduce event emulation issues.

The TestCafe proxy as a whole needs to account for a large amount of edge cases. It needs multiple safeguards to make sure that TestCafe does not interfere with your application. Due to these factors, the TestCafe proxy has been growing in size and complexity. And the bigger it gets, the harder it is to maintain.

The TestCafe team has long wanted to replace the proxy with a better tool. That’s where the CDP protocol comes in.

Benefits of Native Automation

Native Automation solves many of the issues that the TestCafe proxy created.

When TestCafe uses the native CDP protocol, the framework doesn’t need to rewrite URLs and disrupt the logic of the user application. If TestCafe needs to mock HTTP responses or perform HTTP authentication, it can use the native requestPaused hook. The browser can handle most other tasks on its own.

The CDP protocol allows TestCafe to natively execute common browser actions (click, type, pressKey). As a result, these actions are more stable and easier to troubleshoot.

The move to CDP allowed the TestCafe team to remove a lot of complex code that caused trouble. This, in turn, increased the framework’s stability and ease of debugging.

Perhaps, most importantly, CDP automation increases test speed. To illustrate this point, we benchmarked the TestCafe test suite of the DevExpress ASP.NET Grid Components team.

Each of the 98 tests ran a total of eight times — four times with Native Automation, and four times with the legacy TestCafe proxy. We calculated the average test runtime for each run configuration, and compared the two.

The average test run duration with Native Automation was 6 minutes 28 seconds. The average proxy test run lasted 17 minutes 50 seconds, a whole 275% longer. Individual tests were up to 85% faster with Native Automation.

Multiple tests that ran well with Native Automation yielded errors with the proxy. Three tests failed on one occasion, and one test failed 3 out of 4 times.

Challenges to overcome

Native automation is great, but not all-powerful. TestCafe includes a wide variety of capabilities, some of which just don’t work with CDP. Most of these limitations are minor, and won’t cause much discomfort. For example:

• The CDP protocol only works in local Chromium-based browsers. If your browser selection includes other browsers, such as Firefox or Safari, TestCafe runs your tests with the Hammerhead proxy. To take advantage of Native Automation and run tests in other browsers, create two separate test runs.
• TestCafe cannot run tests in multiple browser windows.
• You cannot set a custom page request timeout or AJAX request timeout.

As the framework’s development continues, the TestCafe team hopes to resolve these issues. Read the Native Automation FAQ for more information on the current limitations of native automation.

Try it yourself

We welcome you to try native automation and share your thoughts!

If you have any questions, consult the Native Automation FAQ, reach out to the TestCafe community on StackOverflow, or create a GitHub issue that outlines your concerns.