How Neos CMS Increased Reliability and Confidence with E2E testing and TestCafe

A Seamless Content Management Experience

Neos is an open-source web content management system that attaches great importance to usability and uses the website itself as an interface for editing wherever possible. Neos empowers its users and allows editors to concentrate on content rather than the actual processing technology. Neos can create content across multiple dimensions, including language, country, target group and more.

The Pursuit of Reliability

I was fed up with the fact that we had unit tests for certain parts of the UI, but we still could not merge pull requests reliably. We could not even merge an automatically generated dependency update. We basically had zero confidence in the system until we tried and tested it ourselves. Neos’ UI is quite complex – it includes numerous inter-connected parts. The parts are not complicated in and of themselves but when you join them all together it is impossible to maintain confidence without end-to-end tests.

“Crucial to us was the need to seamlessly switch between iframes, but most E2E testing tools couldn’t handle this critical requirement – well, that is until we found TestCafe.”

TestCafe to the Rescue

I always had this intuition that you should begin with end-to-end testing and use unit tests only in places with complicated logic. I was looking for a tool that – in addition to full iframe support – would be easy to set up and help me get off the ground quickly. In the past, I had negative experiences with Selenium’s complex setup requirements (also had negative experiences with other testing solutions) and I did not want to repeat the experience. Unfortunately, tools such as Selenium were created before the emergence of today’s JavaScript landscape. It is quite obvious to anyone who sets up Selenium that it is not something you can do in ten seconds.

TestCafe, on the other hand, is easy to setup, offers a robust JavaScript API, and supports all modern browsers. The importance of the latter capability cannot be overstated. We had some regression tests in Firefox wherein we used JavaScript API not implemented in the browser. Safari also does things differently from other major browsers. If you have to support Internet Explorer 11, it is even worse. I think it is quite important for end-to-end tests to have cross-browser testing capabilities because it gives development teams confidence. After all, confidence is why you create tests.

“It is quite obvious to anyone who sets up Selenium that it is not something you can do in ten seconds. TestCafe, on the other hand, is easy to setup, offers a robust JavaScript API, and supports all modern browsers.”

Another key TestCafe feature I have come to reply upon is its React selector library. It allows us to make assertions based on component states (for example, we were able to make assertions based on Redux state). Honestly, this is an amazing feature because in addition to basic DOM assertions, you can do anything needed inside the React ecosystem.

Building Confidence through TestCafe

The biggest benefit of TestCafe is its ability to build confidence within your team and in your product. In our case, if a TestCafe test passes, it means that at least half of our UI works correctly. If you compare the regressions and bugs which we had before and after we wrote our TestCafe test scripts, it is a whole lot different. Since we are open-source, we have a lot of pull requests coming in from external contributors. It is really stressful if you need to manually check everything. Now, if it is something simple, I just merge it without opening it locally. With TestCafe, we got confidence that not just a single module works in isolation but the whole system together would work after we merge a certain PR. That is the number one thing.

“With TestCafe, we got confidence that not just a single module works in isolation but the whole system together would work after we merge a certain PR.”

Oh and before using TestCafe, we reviewed the last hundred bug fixes, and found that 70% of these fixes could have been caught had we used TestCafe from the start. As it stands today, we use TestCafe for our end-to-end tests (to test basic functionality) and rely on unit tests for complex app logic.

“70% of hundred our bugs could have been caught had we used TestCafe from the start”.

Would You Recommend TestCafe?

TestCafe is intuitive, lightweight, easy to use, and setup. It includes a powerful selector library and a great support team. I would highly recommend TestCafe.

Results

• Increased reliability and confidence through TestCafe tests
• Caught 70% of regressions in advance
• Saved significant time by ending the need to check changes manually