Why technical debt removing is painful?

Technical debt exists in code architecture and in code base. Debt appears when product requirements changes and new features are built in without taking care needed, small or big, architectural changes. Your code base starts to contain more and more shortcuts which will make it more difficult to develop system. Development speed will slow down. Finally you are so slow that developing the system does not make any sense. That sucks.

Why it’s difficult to remove technical debt? In this post I will introduce some common obstacles. I will show that usually debt comes because short term goals drives over long term goals and that without craftsmanship of the whole company or department product will have more and more technical debt. We need to start fight against debt.

Short term goals are driving

It’s easy to be blind for long term goals because of short term goals. You do what you were asked to do and deliver it fast. Everybody is happy. You have taken some shortcuts but who cares. Feedback is positive and mood is high. Some guys from the team are praised for finding clever shortcuts, like using database fields for totally different things than their name says, or just controlling same functionality with more and more input parameters and if -branches.  Short term speed tastes good.

I think this short term goal achieving is ok. As far as it’s understood that by doing these shortcuts you take debt which you need to pay back. If only short term goals are driving the development and short cuts are taken iteration after iteration and year after year then your product will be screwed up. There has to be balance between short and long term goals. Product owner has a major role in goal setting.

Technical debt is difficult to measure

If you don’t have any measurements for technical debt then it’s difficult to know that it exists. For example: As product owner, do you know how many shortcuts were taken in last sprint? Do you know their effect on upcoming sprints?  It might be that development team is the only one who knows current technical debt amount. When you don’t know about technical debt in your product then you are not ready to eliminate it. Most likely paying debt back  is not in your backlog, or is it? After all, reducing technical debt will only bring indirect value for customer.

What happens if we combine these two first problems? Only developers know about technical debt and you are rewarding them to take more debt.  You might see that velocity will drop and what you will do? You put more pressure to get to the deadlines. And again more short cuts are made. It’s like product has fallen to ground and you are still kicking it. Lucky us, product doesn’t feel pain (yet).

Legacy code

Ok, we know we need to know if technical debt exists and some way to measure it. Then we look those 1000 classes with 20 000 LOC each and without any automated tests. Critical business processes use the existing build binaries every second. What do we do? Maybe some sensitive guy starts to cry? Somebody proposes to refactor it all (product owner opens bottle of Jack). Well yes, this is serious situation. Then we need the leading developer to stand and say: “I have some ideas so let’s not make it any worse than this, let us know where it’s most valuable to start to remove technical debt and when we will start?”

Conclusion

Usually technical debt is compared to financial debt. I have an another example in my mind.  Technical debt is like a unhealthy food. Like candies and cakes. It tastes good but if you eat those candies all the time you will start to get fat, you will end up being in not a good shape. Same way as product architecture starts to be in bad shape. If you don’t continuously measure your weight you might not notice effect from candies. Fat comes slowly.  At the beginning you might not even notice it. At some point you will find out that you are too fat. You will notice that you can’t run anymore. Maybe then you will start to think differently about eating candies and start to do some walking exercises.

Be as serious with technical debt as you are with your own health. As a product owner try

  • Balancing short and long term goals for product development
  • Studying and understanding your product and technical debt in it
  • Small steps to get rid of debt

Otherwise principles of technical debt will kill your product.

Advertisements

Case study: Planning communication for managing stakeholder expectations

How could stakeholder expectation management be done in practice? We already went it through in theory and now I’ll adopt theory to practice with imaginary case study.

Let’s assume we have product which has been built as a back office solution. Marketing and product owner sees opportunities in markets if frontend would be web UI instead of back office UI. Development team has checked that new UI can be implemented with reasonable cost and owners of the company have agreed to invest money for the product development. Marketing department have got two pilot customer signed in for beta testing.

Identifying stakeholders

There has been named one person from the owners to be responsible from the cash flow for this project. He has authority to cancel project. There are two customers who have agreed to be pilot customers. They have named 2 main users (1 from each) to be responsible about feedback. Pilot customers are able to purchase software later on with discounted price. Customer support will support product after it’s published. Marketing department is interest how they are able to sell product and company has one 6 people scrum team which will work with the project

Analyzing and grouping stakeholders

Customer support wants to have product which is easy to support. They want to know that there has been taken care of usability issues and that there is followed company’s standard for web UI development. Support is also interested that team will not build too complicated features for new UI. As customer support knows the team and what they can do it’s enough for them to monitor such things.

Pilot customers are more interested. They have some investments in as they have people inside project. Also they are interested that final product will give as much value for their business as possible. They also have some power as product goal is to fulfill needs of customer segment which selected pilot presents.

Marketing has almost similar interest for the product as owners of the pilot customers. Market’s contains lot of potential new customers but it requires that product features matches market expectations. Marketing provides lot of important information for guiding product development.

Dev team is interested to deliver valuable product (case company is proud to have such team). After all it’s their job to do project. Also they have some power as this team is only team which can make product with reasonable cost.

Main users are interested from the product as well. Product impact will be high for them. They will use product in their daily work after this project. Their power is not high as dev team but it is higher than customer owner (we assume that owners of the customer companies are listening these users).

Owner is providing cash flow. He has high power. He is also interest about product success. There are certain risks with dev team and web UI. Because of that owner is interest to follow how product development starts.

Following picture shows how stakeholders are placed to matrix based on those groups.

When product beta’s will be taken to use it has impact to customer support. Even customer support priority group is  small it should be managed closely in release planning. Based on this short analyze there can be created groups

  • Manage closely: Owner, Dev team, Main users (+ Customer support for release planning)
  • Keep informed: Marketing, customer owners
  • Monitor: Customer support

But these groups are only initial.  For example, after project goes on then stakeholder can be moved from group to another. Why? Because environment changes. After a while owner finds out that he can trust the team. Also he notices that feedback from main users is positive. Things have changed and he doesn’t want to use such much time for this project. He is then moved to group Keep satisfied.

Communication

Grouping is now done and communication can start. First there is created vision with people who needs to be managed closely and kept informed. In this case study vision is written by product owner but it’s based on co-operation with marketing, owners from customer, main users, owner and representatives from dev team.

After vision there is planned schedule with people who needs to be managed closely and who will have impact based on schedule. Owner of the company has told that he only wants to be informed about release plans. Product owner starts to work with developers and with main users to create rough product backlog. This backlog has first estimations from the size of product features. Release plan is created based on rough product backlog and for that also customer support and marketing are asked to contribute. Backlog has now first priorities.

Development starts and reviews takes their place. Demos are important for  the people who are in the group which needs to be managed closely.  Other groups are invited to demos but its optional for them to join. This means that there can be found at least dev team, main users and the owner from the demo session. Sometimes you can find people from marketing, customer owners and customer support as well.

Stakeholders decides to split releases to two sessions. First one introduces release  and second session works as a retrospective for the release. To both sessions there is invited all stakeholders. In the second session there is checked how project is going based on the vision and release plan. If needed then changes to expectations and groups are made. This goes on until project is finished.

Conclusion

In case study there was used 4 steps to manage stakeholder expectations:

  1. Identify stakeholders
  2. Analyze and group stakeholders
  3. Manage expectations with known tools
  4. Iterate

Case was pretty nice and in practice there comes many surprises along the project. Environment changes. It would be nice to hear what are best surprises what people have experienced along the way. If you share some of yours, I can share some of mine. We can together think how to handle stakeholder expectation management in those cases.

Stakeholder expectation management – introduction

Let’s go more deeply to managing stakeholder expectations as it was discussed in earlier post comments. Successful project and product needs successful stakeholder expectation management. When expectation management is kept effective it provides valuable source for the feedback. Even expectation management sounds relative simple to do I have found myself many times figuring how it would be good to done. Other question is that how much it’s needed.

In this post I’ll go through basics of stakeholder expectation management from product owner point of view and in next post I’ll adopt theory to practical example in my agile context.

Stakeholders

From the wiki we can find that project stakeholders

  1. sponsor a project, or
  2. have an interest or a gain upon a successful completion of a project;
  3. may have a positive or negative influence in the project completion

In practice it can be management, customer, main users, dev team and customer support. For finding how much expectations management is needed we could group these entities with these question.

How much communication is needed

First you need to identify the need. Identify and analyze your stakeholder to different groups based on their needs. After you have groups then communicate based on group needs. This way you are avoiding unnecessary overhead. After a while start to get feedback and re-arrange groups if needed.

For grouping there is several ways

  • Users, governance, influencers and providers (source)
  • Priority groups which are based on variables: power (high/low) and interest (high/low) (source)
  • Impact of product (ROI, change in future work load etc.)

Also there is needed to consider that how big trust there is for project/product success. If stakeholder trusts product success then need of the communication is not so high. From my experience there is variances like

  • Competence people vs. not competence people working with product
  • Small project vs. big project
  • Known markets vs. unknown markets

How communication can be done

Keeping expectations in already familiar tools will help you to avoid unnecessary overhead.

Product vision

  • Here you will capture your business value, customer main needs, critical attributes etc. So it is already telling you what are the stakeholder expectations
  • Checking this regularly will give to you and other stakeholders opportunity to know if product is still going to deliver value what was expected
  • This usually is document. For communicating more about critical attributes I propose tool from Mike Cohn’s blog.

Release plan

  • What expectation are going to be filled and when? What are milestones? When people with impact needs to be ready?
  • With this it’s also easy to visualize what happens when stakeholders come up with new ideas and requirements (yes it really happens when stakeholders understands that changes and feedback are welcome) or some key people are lost from project.
  • Visualization can be simple time line or gant chart or release burndown chart.

Product backlog

  • Which are the next things to be build.
  • More detailed than release plan and from my experience big picture can be hided to backlog. Use it for correct purposes for needed stakeholders (like dev teams and users).

Demo sessions

  • Useful session to make sure that expectations for this sprint are filled.
  • Is more detailed and might hide big picture so this is not ideal for all stakeholders.

 Conclusion

Identify and analyze your product stakeholders. Try to find how often and what kind of information they are interest to get. After this is done then include information to existing tools which you are using for your product. During implementation phase get back to these tools and check is product still meeting stakeholder expectations. If not then fix direction (or expectations).

One point which also good to know is mentioned in this post. There are realistic and unrealistic expectations. Don’t let unrealistic expectations be alive.

Memo from Scandinavian Agile Conference 2012

I participated Scandinavian Agile Conference 2012. Conference was organized well and I think many people enjoyed that day. I participated to Human Touch track but also had last lesson from Marc McNeill. Here are main points which I remember from sessions. Note: This is how I understood things and which I felt were important. There was introduced other things as well but I focused on the things which I write here.

Dave Snowden:The new 3 C’s: Complexity based approaches to project management

This was the first lecture of the day and somehow it was a bit difficult to follow (at least for me). Here are some notes.

As software development is usually complex thing to do it is not wise to have a lot of rules to make the process effective. You should only give a couple of basic constraints which team should follow and the rest they can figure out by themselves. This was illustrated with complex traffic places where you can either have many traffic lights or just some traffic circles with simple rules. It’s usually faster to use basic rules in the traffic circles than wait all needed traffic lights turning to green.

It makes sense. Similar way for example coding standards can be used in company level. Standards guides teams with simple rules. Otherwise team can decide what kind of implementation it will create for the product (naturally there is other constraints as well but I want to keep it simple with this example).

Other point was how to find next stories to be implemented. Dave proposed approach of diary. Users don’t need to write exact problems what they have but instead weekly they keep a diary of their work. This way it’s easier to analyze what kind of problems users have. And then software company can notice that they could do (or already have) solution to challenges that users currently have. It’s like mixing new customer needs to new technology.

Other good example was how to find new business things. He told example where ordinary people were asked to tell what kind of things they would like to have in their garden. Those requirements were added to list. Then technology companies were asked to write a list what kind of things they have.  Then there was analyzed those just created two lists and finally matched possible technical solutions against gardeners visions . This produced couple potential new usage of technologies to new things in garden.

Joseph Pelrine: “Cynefin – Making Sense of Agile”

Here I learned that not all things are complex. Some of them are simple, some complicated and some chaotic. And for each of these categories you should use different approach.

  • Simple: Sense, categorize, respond
  • Complicated: Sense, analyze, respond
  • Complex: Probe, sense, respond
  • Chaotic: Act, sense, respond

If I understood correctly complex problems are tried to cut to pieces and handled as complicated problems. Specially in the scrum. Interesting topic and you can find more from here Cynefind.

Kati Vilkki: Creating continuous improvement culture

This was quite much from retrospectives. One point which I found interesting was that usually you should use more time to plan new improvements instead trying different things quickly. Planning takes time as there is needed to listen many people opinions (because usually it is complex context and no one alone can know best solution) and you need to compare different options. Then after you have done research then  you have to implement improvement quickly. After that there comes checking how improvement was and then learning about it. This is a common practice and introduced also in Toytota approach. It’s also interesting to think how “fail fast” ideology is related to this.

Laura Snellman-Junna: Mechanics of Empathy

I learned that it is easier to emphasize people who are like you than those which are different than you.

Marc McNeill: Agile Experience Design

Mark was discussing about creating environment where users and developers can work together. This is the way how developers can really understand what users need. They can stand in customer shoes and emphasize their problem. It will not require more than for example 1 day from month. It’s very valuable. Also prototypes should be used as much as it can be to minimize cost of failures.

When discussing with some colleagues we somehow founded that in other tracks there were also discussed a lot about importance to work near the customer. I’m not sure if it was theme of the day but at least that was main message which we got.

Checklist to keep myself agile product owner

This post is related to earlier post: Reminder: Why I choose agile.

I experienced that even I found out reasons why to choose agile I don’t follow those in my daily work as much as I should. Because of that problem I decided to create check list for myself. I usually plan my upcoming work week at the last week Friday or at the beginning of the week. Following list will help me to plan my week.

  • Product
    • Will you get  feedback for the product?
    • Will you work with vision and backlog based on feedback?
    • Will you arrange possibility for development team to work with end users?
  • People
    • Will you discuss with team if they are having needed competences?
    • Will you let team to self-organize to make  best process for themselves to build the product?
    • Will you let team to find out best implementation for product?
    • Will you let team know to which direction it needs to grow to build the best product?

Additional note: Most likely you need to work with constraints like budget, time, quality, shared vision etc. If needed then communicate with all stakeholders. Don’t let constraints to be obstacle. Let it be possibility for new discussions.

Reminder: Why I choose agile

I’ll keep myself in most important things.

Most of the software projects are complex. This is stated in Appelo’s book and is based on research of many people. My experience is that one major reasons for complexity is need to do something which is difficult to explain for the users. And for users and customers it’s difficult to explain to you what they really need. Nothing can be sure  before you can see it and try it.

Other major thing is organization layers. Even if it is difficult to know what to build it is even harder when you have multiple people in communication chain. How to know that we are building right thing?

Also complexity exists in the product itself. There needs to be taken care of maintainability, security, scaling and so on. Also there might be 15 year old legacy code behind product core architecture. How to make sure that product is build right?

Agile methods for my product

So I can’t know 100% what user want’s. I need to get feedback for product and therefore I prefer to have check points where I can check if we are building correct things for the users. And if there is something going to wrong direction I need to be able to change my plans. As we know iterations works here nicely. Like I read from Toyota way book about continuous improvement with PDCA:

  • Plan
  • Do
  • Check
  • Act

Also it is important that there is reduced organization layers between development and customer. That is the way how people who actually are building the product can understand problem to be solved. Developers needs to be able to stand in customer shoes. Side-effect is that when people are able to empathize problem they are able for trying to seek the best way to solve it. Not just add feature after feature. That is the way to build best product.

Agile methods for my people

I don’t want to work with organization where can be stated that:  management want’s more out from the development and development thinks that not full of their capabilities are used. It’s one of the biggest waste. It makes me angry (this energy can be used to positive way like changing organization to be more effective). As we are working with complex environment, where I assume that nobody alone can’t understand all things, then we need to have teams. Teams which contains people with different competences. For teams we need to make environment where teams can use their full potential. For me it means things like:

  • Team can decide what is best way to work together
  • Listening team about solutions what could be used
  • Let the team build the software

Management needs to give direction where to go. For example: they need guide what problems team needs to solve and which quality level to achieve. It can be either customer problem or process problem.