During last spring I have had lot of discussions and studies about automated testing. Question has been that should we start to automate regression tests to more than 10 year old code base and related functionality. Discussions were done with stakeholders including customers, shareholders, management, scrum masters, dev teams and testers. Studies were done by using web and also books from Lisa Crispin and from Google and I participated to testing days to listen how Spotify and F-Secure handle things. Should we start to automate in our context? Answer is both yes and no. In this article I will keep focus on the added value for our product.
How it helps people?
Let’s first check from the people point of view. Good team is more likely going to build product which will deliver needed value.
For the team which wants to improve, it’s important to have fast feedback. With having automated tests integrated to builds we have possibility to have feedback. It’s usually faster than having feedback from manual tests. And definitely faster, and cheaper, than having feedback from not working functionality from customer.
Other point is that team is changing. Some people goes out, some comes in. When product lives longer than the team, you need to manage knowledge how to build the product. One of the ways to store product correct behaviour is to store it to automated tests. This way new people can know, how things needs to work. It works as a document for developers.
For example: you end up coding of new rounding functionality for invoice payments. You tested your new code and it works nicely. It’s time to go home. At the morning you find out that nightly build throwed error. It shows that you broke invoice closing when it is targeted to credit note. It’s not exactly invoice payment, but it works with same code base. Your focus was too much in your story testing, not in side-effects testing.
But you got fast feedback, still remembered your yesterday coding, understood the problem, learned about it and had a clue where issue is. Also you saved money because issue didn’t reached end customers. Who added the case to test suite? It was added together with customer, because team didn’t understood business domain completely.
If issue would have been reported by end customer, it could be month or two later when you would know about it. Would it be easy to fix then? How many similar issues you have done during those months? If you would have known about issue would it have steered you to other direction with architecture?
One of the cons for humans is that we can’t rely discipline of them. I assume that everybody has heard sentences: “This small change won’t break anything. We don’t need to test it.” Computers don’t say that. They test and test if we have told them to do so. If test suite is good then computers will test even small change side effects.
Back to example case. Let’s say we would have found our payment issue in manual regression testing near the sprint end and we wouldn’t have automated tests. After small fixes would someone be running all regression tests manually again?
What about product?
People are changing and so does the environment. Either we understand reality differently or customer business is changing. When we do changes then we might have need for refactoring. When we have good testing suite for product we are able to do changes more easily and with good quality. We are more capable to change product to model real world and to serve customer needs.
When we automate tests we are also saving time from manual regression testing. This is important for big systems. Saved time can be used for testing more tricky case and scenarios which are more difficult to automate. For those cases there might not have been time before. By focusing to corner cases we are able to increase product quality.
In our example case we could assume that we have automated tests in place. Now we have new change request from customer. It doesn’t fit to our current architecture. We know that we will have more similar requests later so team decides to change class structure. That way it will model new understanding of the environment. Team wouldn’t do this change with reasonable cost without automated regression tests. After changes code base is easier to maintain, it’s easier to understand and it’s faster to do upcoming changes.
My point is that either you check automation from people or product point of view it has many good things. Its investment for product success in long run. Good test suite can:
- give fast feedback and therefore increase learning possibility
- work as a document
- always watch your back even for small changes
- give possibility to create better architectures with less technical debt
- give possibility to catch more bugs before deployment
Some of the items are more important to other stakeholders than something else for other stakeholder. So, there is good reasons to automate, but there is good reasons to not automate. I will write about those in my next post.