How Neos CMS Increased Reliability and Confidence with E2E testing and TestCafe
A Seamless Content Management Experience
Neos is an open-source web content management system that attaches great importance to usability and uses the website itself as an interface for editing wherever possible. Neos empowers its users and allows editors to concentrate on content rather than the actual processing technology. Neos can create content across multiple dimensions, including language, country, target group and more.
The Pursuit of Reliability
I was fed up with the fact that we had unit tests for certain parts of the UI, but we still could not merge pull requests reliably. We could not even merge an automatically generated dependency update. We basically had zero confidence in the system until we tried and tested it ourselves. Neos’ UI is quite complex – it includes numerous inter-connected parts. The parts are not complicated in and of themselves but when you join them all together it is impossible to maintain confidence without end-to-end tests.
“Crucial to us was the need to seamlessly switch between iframes, but most E2E testing tools couldn’t handle this critical requirement – well, that is until we found TestCafe.”
TestCafe to the Rescue
Another key TestCafe feature I have come to reply upon is its React selector library. It allows us to make assertions based on component states (for example, we were able to make assertions based on Redux state). Honestly, this is an amazing feature because in addition to basic DOM assertions, you can do anything needed inside the React ecosystem.
Building Confidence through TestCafe
The biggest benefit of TestCafe is its ability to build confidence within your team and in your product. In our case, if a TestCafe test passes, it means that at least half of our UI works correctly. If you compare the regressions and bugs which we had before and after we wrote our TestCafe test scripts, it is a whole lot different. Since we are open-source, we have a lot of pull requests coming in from external contributors. It is really stressful if you need to manually check everything. Now, if it is something simple, I just merge it without opening it locally. With TestCafe, we got confidence that not just a single module works in isolation but the whole system together would work after we merge a certain PR. That is the number one thing.
“With TestCafe, we got confidence that not just a single module works in isolation but the whole system together would work after we merge a certain PR.”
Oh and before using TestCafe, we reviewed the last hundred bug fixes, and found that 70% of these fixes could have been caught had we used TestCafe from the start. As it stands today, we use TestCafe for our end-to-end tests (to test basic functionality) and rely on unit tests for complex app logic.
“70% of hundred our bugs could have been caught had we used TestCafe from the start”.
Would You Recommend TestCafe?
TestCafe is intuitive, lightweight, easy to use, and setup. It includes a powerful selector library and a great support team. I would highly recommend TestCafe.
- Increased reliability and confidence through TestCafe tests
- Caught 70% of regressions in advance
- Saved significant time by ending the need to check changes manually