In previous post we found out that there is reasons to invest money to automated tests. It helps people to develop their skills and automated test increases product quality. We also found out that automated testing suite gives ROI not in short term, but in long term.
But is it always giving ROI? Now we will take a look what are the reasons to not automate tests.
No competence to automate tests
If you want to have automated test suite for your product, make sure team has competence to do it. If there is no person who can take care of architecture of test suite, take care of test tools or to help others, you will end up to trouble.
It’s like working with team without competence to build user stories. Nobody really knows how they should code the thing, but they do it. At the end you find out that implementation cost a lot and readability and maintainability is low. Same with the automated test suite. It’s all the time broken and nearly impossible to develop further.
How you could avoid this trap? Make sure there is competence. Simply ask. Good signal is that, when you propose automated test suite for team, they will throw bunch of questions back. There could be asked questions like:
- Do you mean, unit tests, component tests or end-to-end tests?
- What kind of coverage product would need?
- Do you want stress and security tests as well?
- What is the value of tests? We are not delivering so many issues inside our stories?
- Would you like us to help you to find tests which gives most value for product?
Other thing which was pointed out for me, by colleague of mine, Darius Muncys, was worry about misunderstanding the usage of automated tests. He was thinking that people might use automated test suite against the idea of “Quality needs to be baked in”. We were talking about having some end-to-end tests to our product.
His point was that, even people would know how to do automated test suite, people might think that automated test suite is the safety net which would catch all bugs after coding. Good point. That is not the purpose. That is not efficient usage of automated test suite.
Team needs to have competence to know how test suite is used most efficiently. For example team should automate manual regression tests, which they find out repeating. Then they can do, for example better exploratory testing in new areas. If team doesn’t know how to use automated test suite efficiently, then it’s not worth to create such.
One way to think is that, automated testing suite is not separate testing department. Automated tests are like a team member, part of the team. Tests works with the team to build better product.
Dusty automated test suite
Even team has competence to create and use automated test suite, it’s easy to do with the way, that it doesn’t produce needed value. As a product owner I like to think it as a new feature. If you user needs new story only once, was it worth of doing it? Probably no.
If your automated test suite was meant to catch side-effects and you have only once changes to area where tests would be valuable, were test worth of doing? Probably not. There is just not enough ROI. Don’t do things without good ROI.
If you need to run tests only once, it’s better to consider to doing those manually.
What about the new product, which you don’t know if it will be success or not? I liked Google point of view. They start with the small team which knows how automated tests can be added later. Google is not adding test from the beginning, because if product is not success there wouldn’t be any ROI from automated test cases.
But they choose architecture so that it’s easy to add tests later. And later, if product is success, they will create automated test cases. For this they need hiring guys with high competence.
How to know if you will have ROI from automated test suite? Take your road map and issue backlog. Find out where you are going to need changes and then find out which test cases would have highest ROI in future.
For example team can know codebase side-effects which would be valuable to cover with test cases. And you might know what business domain relation changes will have. If you don’t use road map and analysis to decide needed automated test cases, you end up to guessing what is valuable and what’s not.
Team and product will gain benefit from automated tests but good will is not enough. For having value from automated tests you need to have
- Skills to do and use automated tests
- Integrating test automation strategy for your product strategy
If you don’t have those in place, don’t bother to create automated tests.