Why to not automate testing?

Robot, don't do it

Robot, don’t do it

In previous post we found out that there is reasons to invest money to automated tests. It helps people to develop their skills and automated test increases product quality. We also found out that automated testing suite gives ROI not in short term, but in long term.

But is it always giving ROI? Now we will take a look what are the reasons to not automate tests.

No competence to automate tests

If you want to have automated test suite for your product, make sure team has competence to do it. If there is no person who can take care of architecture of test suite, take care of test tools or to help others, you will end up to trouble.

It’s like working with team without competence to build user stories. Nobody really knows how they should code the thing, but they do it. At the end you find out that implementation cost a lot and readability and maintainability is low. Same with the automated test suite. It’s all the time broken and nearly impossible to develop further.

How you could avoid this trap? Make sure there is competence. Simply ask. Good signal is that, when you propose automated test suite for team, they will throw bunch of questions back. There could be asked questions like:

  • Do you mean, unit tests, component tests or end-to-end tests?
  • What kind of coverage product would need?
  • Do you want stress and security tests as well?
  • What is the value of tests? We are not delivering so many issues inside our stories?
  • Would you like us to help you to find tests which gives most value for product?

Other thing which was pointed out for me, by colleague of mine, Darius Muncys, was worry about misunderstanding the usage of automated tests. He was thinking that people might use automated test suite against the idea of “Quality needs to be baked in”. We were talking about having some end-to-end tests to our product.

His point was that, even people would know how to do automated test suite, people might think that automated test suite is the safety net which would catch all bugs after coding. Good point. That is not the purpose. That is not efficient usage of automated test suite.

Team needs to have competence to know how test suite is used most efficiently. For example team should automate manual regression tests, which they find out repeating. Then they can do, for example better exploratory testing in new areas. If team doesn’t know how to use automated test suite efficiently, then it’s not worth to create such.

One way to think is that, automated testing suite is not separate testing department. Automated tests are like a team member, part of the team. Tests works with the team to build better product.

Dusty automated test suite

Even team has competence to create and use automated test suite, it’s easy to do with the way, that it doesn’t produce needed value. As a product owner I like to think it as a new feature. If you user needs new story only once, was it worth of doing it? Probably no.

If your automated test suite was meant to catch side-effects and you have only once changes to area where tests would be valuable, were test worth of doing? Probably not. There is just not enough ROI. Don’t do things without good ROI.

If you need to run tests only once, it’s better to consider to doing those manually.

What about the new product, which you don’t know if it will be success or not? I liked Google  point of view. They start with the small team which knows how automated tests can be added later. Google is not adding test from the beginning, because if product is not success there wouldn’t be any ROI from automated test cases.

But they choose architecture so that it’s easy to add tests later. And later, if product is success, they will create automated test cases. For this they need hiring guys with high competence.

How to know if you will have ROI from automated test suite? Take your road map and issue backlog. Find out where you are going to need changes and then find out which test cases would have highest ROI in future.

For example team can know codebase side-effects which would be valuable to cover with test cases. And you might know what business domain relation changes will have. If you don’t use road map and analysis to decide needed automated test cases, you end up to guessing what is valuable and what’s not.


Team and product will gain benefit from automated tests but good will is not enough. For having value from automated tests you need to have

  • Skills to do and use automated tests
  • Integrating test automation strategy for your product strategy

If you don’t have those in place, don’t bother to create automated tests.


Why to automate testing

During last spring I have had lot of discussions and studies about automated testing. Question has been that should we start to automate regression tests to more than 10 year old code base and related functionality. Discussions were done with stakeholders including customers, shareholders, management, scrum masters, dev teams and testers. Studies were done by using web and also books from Lisa Crispin and from Google and I participated to testing days to listen how Spotify and F-Secure handle things. Should we start to automate in our context? Answer is both yes and no. In this article I will keep focus on the added value for our product.

How it helps people?

Let’s first check from the people point of view. Good team is more likely going to build product which will deliver needed value.

For the team which wants to improve, it’s important to have fast feedback. With having automated tests integrated to builds we have possibility to have feedback. It’s usually faster than having feedback from manual tests. And definitely faster, and cheaper, than having feedback from not working functionality from customer.

Other point is that team is changing. Some people goes out, some comes in. When product lives longer than the team, you need to manage knowledge how to build the product. One of the ways to store product correct behaviour is to store it to automated tests. This way new people can know, how things needs to work. It works as a document for developers.

For example: you end up coding of new rounding functionality for invoice payments. You tested your new code and it works nicely. It’s time to go home. At the morning you find out that nightly build throwed error. It shows that you broke invoice closing when it is targeted to credit note. It’s not exactly invoice payment, but it works with same code base. Your focus was too much in your story testing, not in side-effects testing.

But you got fast feedback, still remembered your yesterday coding, understood the problem, learned about it and had a clue where issue is. Also you saved money because issue didn’t reached end customers. Who added the case to test suite? It was added together with customer, because team didn’t understood business domain completely.

If issue would have been reported by end customer, it could be month or two later when you would know about it. Would it be easy to fix then? How many similar issues you have done during those months? If you would have known about issue would it have steered you to other direction with architecture?

One of the cons for humans is that we can’t rely discipline of them. I assume that everybody has heard sentences: “This small change won’t break anything. We don’t need to test it.” Computers don’t say that. They test and test if we have told them to do so. If test suite is good then computers will test even small change side effects.

Back to example case. Let’s say we would have found our payment issue in manual regression testing near the sprint end and we wouldn’t have automated tests. After small fixes would someone be running all regression tests manually again?

What about product?

People are changing and so does the environment. Either we understand reality differently or customer business is changing. When we do changes then we might have need for refactoring. When we have good testing suite for product we are able to do changes more easily and with good quality. We are more capable to change product to model real world and to serve customer needs.

When we automate tests we are also saving time from manual regression testing. This is important for big systems. Saved time can be used for testing more tricky case and scenarios which are more difficult to automate. For those cases there might not have been time before. By focusing to corner cases we are able to increase product quality.

In our example case we could assume that we have automated tests in place. Now we have new change request from customer. It doesn’t fit to our current architecture. We know that we will have more similar requests later so team decides to change class structure. That way it will model new understanding of the environment. Team wouldn’t do this change with reasonable cost without automated regression tests. After changes code base is easier to maintain, it’s easier to understand and it’s faster to do upcoming changes.


My point is that either you check automation from people or product point of view it has many good things. Its investment for product success in long run. Good test suite can:

  • give fast feedback and therefore increase learning possibility
  • work as a document
  • always watch your back even for small changes
  • give possibility to create better architectures with less technical debt
  • give possibility to catch more bugs before deployment

Some of the items are more important to other stakeholders than something else for other stakeholder. So, there is good reasons to automate, but there is good reasons to not automate. I will write about those in my next post.

Why technical debt removing is painful?

Technical debt exists in code architecture and in code base. Debt appears when product requirements changes and new features are built in without taking care needed, small or big, architectural changes. Your code base starts to contain more and more shortcuts which will make it more difficult to develop system. Development speed will slow down. Finally you are so slow that developing the system does not make any sense. That sucks.

Why it’s difficult to remove technical debt? In this post I will introduce some common obstacles. I will show that usually debt comes because short term goals drives over long term goals and that without craftsmanship of the whole company or department product will have more and more technical debt. We need to start fight against debt.

Short term goals are driving

It’s easy to be blind for long term goals because of short term goals. You do what you were asked to do and deliver it fast. Everybody is happy. You have taken some shortcuts but who cares. Feedback is positive and mood is high. Some guys from the team are praised for finding clever shortcuts, like using database fields for totally different things than their name says, or just controlling same functionality with more and more input parameters and if -branches.  Short term speed tastes good.

I think this short term goal achieving is ok. As far as it’s understood that by doing these shortcuts you take debt which you need to pay back. If only short term goals are driving the development and short cuts are taken iteration after iteration and year after year then your product will be screwed up. There has to be balance between short and long term goals. Product owner has a major role in goal setting.

Technical debt is difficult to measure

If you don’t have any measurements for technical debt then it’s difficult to know that it exists. For example: As product owner, do you know how many shortcuts were taken in last sprint? Do you know their effect on upcoming sprints?  It might be that development team is the only one who knows current technical debt amount. When you don’t know about technical debt in your product then you are not ready to eliminate it. Most likely paying debt back  is not in your backlog, or is it? After all, reducing technical debt will only bring indirect value for customer.

What happens if we combine these two first problems? Only developers know about technical debt and you are rewarding them to take more debt.  You might see that velocity will drop and what you will do? You put more pressure to get to the deadlines. And again more short cuts are made. It’s like product has fallen to ground and you are still kicking it. Lucky us, product doesn’t feel pain (yet).

Legacy code

Ok, we know we need to know if technical debt exists and some way to measure it. Then we look those 1000 classes with 20 000 LOC each and without any automated tests. Critical business processes use the existing build binaries every second. What do we do? Maybe some sensitive guy starts to cry? Somebody proposes to refactor it all (product owner opens bottle of Jack). Well yes, this is serious situation. Then we need the leading developer to stand and say: “I have some ideas so let’s not make it any worse than this, let us know where it’s most valuable to start to remove technical debt and when we will start?”


Usually technical debt is compared to financial debt. I have an another example in my mind.  Technical debt is like a unhealthy food. Like candies and cakes. It tastes good but if you eat those candies all the time you will start to get fat, you will end up being in not a good shape. Same way as product architecture starts to be in bad shape. If you don’t continuously measure your weight you might not notice effect from candies. Fat comes slowly.  At the beginning you might not even notice it. At some point you will find out that you are too fat. You will notice that you can’t run anymore. Maybe then you will start to think differently about eating candies and start to do some walking exercises.

Be as serious with technical debt as you are with your own health. As a product owner try

  • Balancing short and long term goals for product development
  • Studying and understanding your product and technical debt in it
  • Small steps to get rid of debt

Otherwise principles of technical debt will kill your product.