Why to not automate testing?

Robot, don't do it

Robot, don’t do it

In previous post we found out that there is reasons to invest money to automated tests. It helps people to develop their skills and automated test increases product quality. We also found out that automated testing suite gives ROI not in short term, but in long term.

But is it always giving ROI? Now we will take a look what are the reasons to not automate tests.

No competence to automate tests

If you want to have automated test suite for your product, make sure team has competence to do it. If there is no person who can take care of architecture of test suite, take care of test tools or to help others, you will end up to trouble.

It’s like working with team without competence to build user stories. Nobody really knows how they should code the thing, but they do it. At the end you find out that implementation cost a lot and readability and maintainability is low. Same with the automated test suite. It’s all the time broken and nearly impossible to develop further.

How you could avoid this trap? Make sure there is competence. Simply ask. Good signal is that, when you propose automated test suite for team, they will throw bunch of questions back. There could be asked questions like:

  • Do you mean, unit tests, component tests or end-to-end tests?
  • What kind of coverage product would need?
  • Do you want stress and security tests as well?
  • What is the value of tests? We are not delivering so many issues inside our stories?
  • Would you like us to help you to find tests which gives most value for product?

Other thing which was pointed out for me, by colleague of mine, Darius Muncys, was worry about misunderstanding the usage of automated tests. He was thinking that people might use automated test suite against the idea of “Quality needs to be baked in”. We were talking about having some end-to-end tests to our product.

His point was that, even people would know how to do automated test suite, people might think that automated test suite is the safety net which would catch all bugs after coding. Good point. That is not the purpose. That is not efficient usage of automated test suite.

Team needs to have competence to know how test suite is used most efficiently. For example team should automate manual regression tests, which they find out repeating. Then they can do, for example better exploratory testing in new areas. If team doesn’t know how to use automated test suite efficiently, then it’s not worth to create such.

One way to think is that, automated testing suite is not separate testing department. Automated tests are like a team member, part of the team. Tests works with the team to build better product.

Dusty automated test suite

Even team has competence to create and use automated test suite, it’s easy to do with the way, that it doesn’t produce needed value. As a product owner I like to think it as a new feature. If you user needs new story only once, was it worth of doing it? Probably no.

If your automated test suite was meant to catch side-effects and you have only once changes to area where tests would be valuable, were test worth of doing? Probably not. There is just not enough ROI. Don’t do things without good ROI.

If you need to run tests only once, it’s better to consider to doing those manually.

What about the new product, which you don’t know if it will be success or not? I liked Google  point of view. They start with the small team which knows how automated tests can be added later. Google is not adding test from the beginning, because if product is not success there wouldn’t be any ROI from automated test cases.

But they choose architecture so that it’s easy to add tests later. And later, if product is success, they will create automated test cases. For this they need hiring guys with high competence.

How to know if you will have ROI from automated test suite? Take your road map and issue backlog. Find out where you are going to need changes and then find out which test cases would have highest ROI in future.

For example team can know codebase side-effects which would be valuable to cover with test cases. And you might know what business domain relation changes will have. If you don’t use road map and analysis to decide needed automated test cases, you end up to guessing what is valuable and what’s not.

Conclusion

Team and product will gain benefit from automated tests but good will is not enough. For having value from automated tests you need to have

  • Skills to do and use automated tests
  • Integrating test automation strategy for your product strategy

If you don’t have those in place, don’t bother to create automated tests.

Advertisements

Why to automate testing

During last spring I have had lot of discussions and studies about automated testing. Question has been that should we start to automate regression tests to more than 10 year old code base and related functionality. Discussions were done with stakeholders including customers, shareholders, management, scrum masters, dev teams and testers. Studies were done by using web and also books from Lisa Crispin and from Google and I participated to testing days to listen how Spotify and F-Secure handle things. Should we start to automate in our context? Answer is both yes and no. In this article I will keep focus on the added value for our product.

How it helps people?

Let’s first check from the people point of view. Good team is more likely going to build product which will deliver needed value.

For the team which wants to improve, it’s important to have fast feedback. With having automated tests integrated to builds we have possibility to have feedback. It’s usually faster than having feedback from manual tests. And definitely faster, and cheaper, than having feedback from not working functionality from customer.

Other point is that team is changing. Some people goes out, some comes in. When product lives longer than the team, you need to manage knowledge how to build the product. One of the ways to store product correct behaviour is to store it to automated tests. This way new people can know, how things needs to work. It works as a document for developers.

For example: you end up coding of new rounding functionality for invoice payments. You tested your new code and it works nicely. It’s time to go home. At the morning you find out that nightly build throwed error. It shows that you broke invoice closing when it is targeted to credit note. It’s not exactly invoice payment, but it works with same code base. Your focus was too much in your story testing, not in side-effects testing.

But you got fast feedback, still remembered your yesterday coding, understood the problem, learned about it and had a clue where issue is. Also you saved money because issue didn’t reached end customers. Who added the case to test suite? It was added together with customer, because team didn’t understood business domain completely.

If issue would have been reported by end customer, it could be month or two later when you would know about it. Would it be easy to fix then? How many similar issues you have done during those months? If you would have known about issue would it have steered you to other direction with architecture?

One of the cons for humans is that we can’t rely discipline of them. I assume that everybody has heard sentences: “This small change won’t break anything. We don’t need to test it.” Computers don’t say that. They test and test if we have told them to do so. If test suite is good then computers will test even small change side effects.

Back to example case. Let’s say we would have found our payment issue in manual regression testing near the sprint end and we wouldn’t have automated tests. After small fixes would someone be running all regression tests manually again?

What about product?

People are changing and so does the environment. Either we understand reality differently or customer business is changing. When we do changes then we might have need for refactoring. When we have good testing suite for product we are able to do changes more easily and with good quality. We are more capable to change product to model real world and to serve customer needs.

When we automate tests we are also saving time from manual regression testing. This is important for big systems. Saved time can be used for testing more tricky case and scenarios which are more difficult to automate. For those cases there might not have been time before. By focusing to corner cases we are able to increase product quality.

In our example case we could assume that we have automated tests in place. Now we have new change request from customer. It doesn’t fit to our current architecture. We know that we will have more similar requests later so team decides to change class structure. That way it will model new understanding of the environment. Team wouldn’t do this change with reasonable cost without automated regression tests. After changes code base is easier to maintain, it’s easier to understand and it’s faster to do upcoming changes.

Conclusion

My point is that either you check automation from people or product point of view it has many good things. Its investment for product success in long run. Good test suite can:

  • give fast feedback and therefore increase learning possibility
  • work as a document
  • always watch your back even for small changes
  • give possibility to create better architectures with less technical debt
  • give possibility to catch more bugs before deployment

Some of the items are more important to other stakeholders than something else for other stakeholder. So, there is good reasons to automate, but there is good reasons to not automate. I will write about those in my next post.