Refactoring for Software Design Smells - A question

[Stuie Clarky]

Hi guys,

My question is how does the book approach addressing technical debt, in the sense of identifying the scope of the changes required, to testing before/after the changes and ensuring what you have done is both functionally and technically correct? Working with a legacy product there are bound to be years of tweaks and fixes for obscure corner cases that may not be obvious at first glance, but you know will blow up when it reaches production.

Best of luck with the book,
Stuie

[S G Ganesh]

Hi Stuie,

Thanks for a nice question. I'm answering by addressing each parts in your question - so its a detailed and long answer - please bear with me

1. In this fast paced world, software development teams have the responsibility to quickly deliver value to customers. In this pursuit, there are various occasions when the team has to opt for quick-and-dirty solutions. In these situations, it is important to adopt a diligent and pragmatic approach to handle technical debt issues. The first aspect to manage technical debt is to “prevent” technical debt from accumulating. It includes increasing awareness within the software development teams about technical debt and introducing relevant processes within the organization. Another aspect to manage technical debt is to “repay” technical debt. Conscious effort is needed to identify, prioritize, and refactor technical debt issues in real-world projects. Refactoring is the primary means to repay techinical debt.

2. A cohesive view of refactoring is required for identifying smells and scope of refactoring (see http://www.designsmells.com/article.php?id=11 for more details). Why? Because identifying architecture smells and performing architecture refactoring is a different ball game compared to performing code refactoring.

3. It is important to perform change impact analysis before performing refactoring. Of course unit tests are mandatory for safe refactoring, but we find that code and design reviews to be necessary as well (especially in critical code bases - banking or avionics software, for example).

4. For the legacy software, our book addresses technical debt from a pragmatic perspective: quantify technical debt using tools (http://bit.ly/1IFuHnm), get buy in from stakeholders for repaying debt, create and follow a plan to perform refactoring. There are many difficulties on the way - as you rightly mentioned about the obscure corner cases that may break the working software. The biggest challenge we have seen is lack of unit tests that can serve as a "safety net" for refactoring in legacy projects. In those cases, we need to perform refactoring to make the piece of code "testable", add tests, and then perform refactoring... so it is an iterative process. I strongly recommend Michael Feather's book Working Effectively with Legacy Code in this context: http://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052

[T. Sharma]

Two cents from my side as well (in addition to Ganesh's answer):

The focus of the book is on "design smells" - what are design smells, causes of the smells, implications and effects of the smells, and examples with corresponding refactoring solutions. The aim of the book is to help developers understand design smells so that they avoid introducing them in their product and if smells occurred in the project they could identify the smells and refactor them.

Of course, working with a legacy product is not easy. However, not touching the legacy product due to the fear of breaking code is not a sustainable and effective way. Identifying and removing design smells (while employing change impact analysis and dense test-net) can reduce the design debt of the product significantly and improve the maintainability of the product.

[Stuie Clarky]

My thanks to you both for your top answers.

As a theoretical question, assuming I now differentiate between architectural/code smells (something I had not really thought about before) to my stakeholders (whom we will say are a risk adverse bunch), how do you think would be a good way to approach us paying off some of the technical debt?

Granted this is probably moving away from the scope and intent of the book, but I would like to know your opinions/thoughts on how best to approach this. It seems silly to me that there are companies/ decision makers out there that would freak out if you told them the rent payment was late, but don't seem to care at all about technical debt.

Thanks again for your detailed answers, very much appreciated

Stuie

[Junilu Lacar]

It's kind of like deciding how to clean up a messy house, isn't it? If you have a very messy basement but you don't go down there very often, then your clean-up efforts should probably focus on the kitchen, bedroom, and bathroom instead because those are the areas where you will spend most of your time. However, if your heating/cooling equipment is in the basement area, you probably want to spend some effort keeping that particular area clean as well.

As to companies/decision makers not caring about technical debt, can you really blame them? That's supposed to be what technical people care about. Granted, when you're dealing with technical debt that you and your team didn't create, you need to negotiate with the business for time/effort to do refactoring. But I see this as something that developers really shouldn't ask permission to do. If you were to undertake a bathroom remodeling project, you're going to need to demolish parts of the current bathroom, right? You don't ask for permission to take down the current things. In the same way, taking on the maintenance and enhancement of a legacy project will almost always require some refactoring effort and you shouldn't have to ask permission to do your job as a developer.

I think we as developers need to be more assertive about dealing with technical debt. We shouldn't think that we need permission to deal with it. We can inform and educate our stakeholders about the existing debt, helping them better understand the ramifications and costs of not dealing with the debt. We can negotiate and make concessions and compromises on the schedule/budget so that their concerns and interests are addressed as well. Ultimately, however, we developers need to take responsibility for dealing with TD and own it.

[Stuie Clarky]

Brilliant, thanks.

Will need to get my lucky negotiating hat out at some point then, to get the time allocated

[S G Ganesh]

Stuie Clarky wrote:
How do you think would be a good way to approach us paying off some of the technical debt?

We have dedicated a whole chapter (8: Repaying technical debt in practice) for this topic. But here I am focusing on one specific model named as IMPACT for planned repayment of technical debt.

Refactoring is best performed as an integral part of the development cycle, i.e., continual refactoring performed whenever fixes or feature additions are made to the software. However, when repaying technical debt in a larger scale, especially in large projects, critical projects, or legacy projects, it is important to follow a structured approach while refactoring. In our book, we describe a process model called “IMPACT” that provides guidance for systematic refactoring in practice. It is a simple model with following four fundamental steps (executed iteratively):

1. Identify/Mark refactoring candidates. Use manual code/design/architecture reviews and use automated code/design/architecture analysis tools to find smells and determine candidates for refactoring. Especially in large projects it is more efficient to run sophisticated tools to identify refactoring candidates (for example, applying tools such as InFusion & Sotograph for design and architecture smells respectively).

2. Plan your refactoring activities. Once you identify smells using design analyzers, clone analyzers, metric threshold violations, or manual reviews, analyze their impact, prioritize them, and prepare a plan to address them.

3. ACt on the planned refactoring tasks. This is the focus of many books on refactoring and is the easiest part to automate as well (to a large extent, you can use automated refactoring support provided by IDEs such as Eclipse/VisualStudio/IntelliJ/.. to perform refactoring tasks.)

4. Test to ensure behavior preservation. Of course it is widely accepted that refactoring should be followed by automated unit tests to ensure that the behavior remains unchanged. But our hard-earned experience in large industrial projects suggests that in many situations it is not enough! For example, in one of the cases involving multithreading, all tests passed for the performed refactoring, but the software crashed in the customer site after a few months. Because multithreaded problems aren't deterministic, relying on testing also isn't sufficient. Using static analyzers, dynamic analyzers and code/design reviews can catch such problems. Hence, we recommend complementing regression tests with at least manual reviews after refactoring.

Stuie Clarky wrote:
It seems silly to me that there are companies/ decision makers out there that would freak out if you told them the rent payment was late, but don't seem to care at all about technical debt.
Stuie

It is interesting to note that the term "technical debt" itself was coined by Ward Cunningham as a metaphor to communicate to non-technical managers (on the cost of not repaying for the shortcuts we take in the development process.) We have also come across situations where decision makers have commented that "refactoring is needless rework!" and we wrote an article on that (see http://www.designsmells.com/article.php?id=7)... So, your scenario is not at all uncommon.

[Junilu Lacar]

It's also worth noting that taking on Technical Debt, as Ward defined it, was not a bad thing. Taking on technical debt was supposed to be a good thing because it allowed you to quickly get experience with the "working" software and create a righteous feedback loop. Not paying down the debt was the bad thing. There's a difference between taking shortcuts and taking on technical debt.

The metaphor is explained like this: When you take out a loan from the bank, you have access to things that you normally wouldn't had you not taken out the loan, such as getting a car or a house or a college education. You only get in trouble with the bank if you don't pay your debt back. If you miss a payment, it's more difficult to make the next payment. The more payments you miss, the more difficult it is to get back on track with your payment schedule. Interest and penalties accumulate and make it even harder to get back in good financial standing with the bank. After a while, you are in such a bind that you have to declare bankruptcy.

This is a great metaphor for development and refactoring. The lesson is to always pay back the debt that you take on with the knowledge and experience that you gain from being able to have working software sooner rather than later.

Somewhere along the line, the original meaning of Technical Debt went through what Martin Fowler calls "semantic diffusion" and Technical Debt became more commonly known/understood as having a negative connotation. Understandable but unfortunate.

[Junilu Lacar]

Delving a little bit more into the metaphor:

Technical debt is the "loan" you take out. TD is anything that is "not quite right" in the software that you write. Having it means that you can get the software running sooner rather than later. Software that runs is valuable because it can give you tangible feedback. You learn things about your system through running software that you simply cannot learn otherwise. Once you gain more knowledge, experience, and understanding about your system and the software, you need to go back to those things that were "not quite right" and make them right. That's like paying back the principal. The interest is the time you spend refactoring.

[Junilu Lacar]

The other distinction you should make, too, is that A Mess is not Technical Debt. Using an in-memory data store can be technical debt. It can help you get your software running much more quickly than if you had to write code to use a DBMS. Using an in-memory data store doesn't obviate the need to have proper abstractions to hide the detailed implementation(s) of the DAO. If you hard-code the use of an in-memory data store, that's either a mess or a good start of it. If you set up the proper abstraction layers so that you can plug in a different implementation later, that's prudent preparation for repaying the technical debt you take on by using a simpler, in-memory implementation of the DAO.

[Stuie Clarky]

My thanks to you all, you have given me plenty to think about and digest