How to address technical debt in an evolving IT landscape?

Can you wipe out technical debt from your enterprise landscape? Can you always remain one step ahead? These are tough questions that require a calibrated answer. Before answering, we need to understand what is meant by technical debt, why it occurs and ways to manage them.

The first realization should be that technical debt is bound to happen no matter your toll gates and quality check.  The second is an organized program should be initiated, at periodic intervals, to estimate the size and manage its growth. The third is that not all technical debt is bad.

Like a financial debt, if not managed or contained, it would grow to such an extent that it affects the time to market, sink the morale of employees and cease to make the IT shop attractive. Technical debt accrues interest in the sense that it can generate new debts and make it progressively difficult to manoeuvre the development. Tech debt will always be there.

What is TECHNICAL DEBT?

In simple terms, it is the marginal (incremental) work required to complete the software development in order to address the drawbacks. It doesn’t apply only to those projects that are in development stage. It follows even when it is in BAU (Business As Usual). Technical debt holds the organization back from introducing new functionality quickly. We can get an estimate of the debt as the time or money required for refactoring. This refactoring could result in change in design, cleaning up of bad code or porting to a new technology.

What causes technical debt?

The experts cite 4 major reasons for introduction of technical debt.

Cause #1 - Poor Conception: It is the rush or speed in delivery that causes poorly designed software.

Cause #2 – Poor Scheduling: Underestimating the time to develop a product or complete a project often is the culprit that introduces technical debt.

Cause #3 - Bad and inconsistent development practices: Various developers working across different modules tend to introduce their own practices that affect the design and possibly rebuild the logic independently.

Cause #4 - Outdated Technology: As technology evolves, software standards become higher every day. With each improvement, new technical debt can arise.

Types of TECHNICAL DEBT

Type #1: Intentional debt – The software engineers almost always know the right way to code something and the quicker to way to do it. In many cases, the quick way also turns out to be the right way and in others, it may not be. To quickly deliver a project or a product, the functionality will be achieved but not in the best possible manner.

Type #2: Design tech debt – Do we spend time thinking ahead? Or Do we future-proof our design with quicker delivery? As systems become richer with more functionality, the developers may find implementing a new feature very difficult. It may be easier to refactor the original design if it was constructed in a suitable manner. If not, what do you do? Be ready to bite the bullet and get into significant refactoring.

Type #3: Obsolete debt – As many people work on a system, it tends to evolve in a rotten manner. Some symptoms like copy-paste and cargo-cult programming can be easily seen to identify this type. This is directly incurred by the developers. This is one debt that we should avoid in a consistent manner.

Another way to look at this is using the following quadrant that classifies using two dimensions – deliberate or accidental and manner of introduction of such debt.

How to manage this?

Type #1 – Record the debt as backlog at the time it is incurred. This can be revisited later.

Type #2 – Whenever the system is in steady state BAU, allocate some time to look at this type of debt and see if any refactoring has to be done.

Type #3 – Continuous refactoring is the solution to address this. Very experienced and strong teams tend to take time to understand the design of the system before they work on it. When they work, the improve the design incrementally and clean up the bad code en route.

Measures / Signs of technical debt:

Source Code Formatting: It is a common measure. Insisting on the right tool as well as template during and before the SDLC can reduce this type.

Low Test Coverage: It is a measure of code quality. A very low level of the test coverage reduces the certainty of the accuracy of the software's behaviour and makes it difficult to solve problems when they occur.

Lack of Modularity: This generally results from a poor code design. Some code sometimes serves different business logic. The more codes developers write, the more lack of modularity can bottleneck. It is harder to manage software that has logic all over the codes and have parts of codes handling several logic.

Code Complexity: Complexity can be measured in several different ways, but it measures the dependence and path length to perform an operation. A long path leads to complex code.

Lack of the Documentation: Documentation is part of software development best practices. The software is often driven to evolve. It is important that the written code is always understood at all times by everyone who may be involved in the development process.

One approach is to perform a static analysis of code using tools that support the analysis. The following is a list of the most popular tools used: Coverity, SonarQube, Checkstyle, Closure Compiler.

There are two ways of measuring technical debt. The first one is to get a ratio of technical debt according to code volume, and the second one is to use directly the estimates given by the tools (like SonarQube), along with the list of technical debts and their references to the code, SonarQube gives an estimate in days or hours needed to fix this debt. For the ratio approach, we can use the initial estimates or even better, the overall time needed to develop the software so far and extrapolate the value according to the technical debt ratio. The time needed for the development is very accurate so measuring technical debt from ratio can give an accurate estimate of the work needed to fix the issues.

How to Reduce or Eliminate Technical Debt

Being agile is the best way of managing technical debt and reducing it when it appears. The sooner we address the issue, the less interest we'll have to pay over time. To address technical debt, software development teams can use the following approach.

The Quickest to Solve: Fixing debts that take little time to fix is an excellent way to eliminate technical debt gradually. Debts like code formatting can be solved in a little time, making up a template and apply these templates to all the codes that have been developed so far, then integrate these templates in the tools used by the developers.

Priority: It is also important to address issues by priority. All issues that can lead to more significant issues should be addressed quickly and should be prioritized to avoid accumulation.

Technology Update: When outdated technology leads to technical debt, it is important to update the software to the newest versions of the frameworks, application servers, databases etc. It is even important to include every stable evolution of a framework used for instance to always have the latest update and to bring small change without breaking the software.

Refactoring: Reviewing the software architecture and refactoring codes often can be useful when we don't want to end up with duplicate code or codes that lack modularity.

The quadrant, shown, can help to categorize the technical debt to identify which ones to fix first. There are many such approaches. Coming to agile processes, how to estimate / eliminate technical debt? It should be entered in the product backlog as a user story and should be prioritized like any user story. The prioritization should take into account the impact of not managing technical debt identified at the beginning of a new iteration. When deciding which stories to include in the next development iteration, we should analyze whether postponing technical debt correction for the next iteration is more or less advantageous in the long term. As a rule of thumb, we should address every critical issue as soon as it is identified.

Closing remarks

We have to learn to live with certain amount of technical debt. A good IT shop will always have a handle on the measure in terms of either the time or money needed to fix the accumulated debt to a manageable level. Understanding of different types of debt and the rate at which the organization accumulates will help in introducing specific measures to eliminate or minimize such debt.

References:

https://www.agilealliance.org/introduction-to-the-technical-debt-concept/

https://hackernoon.com/there-are-3-main-types-of-technical-debt-heres-how-to-manage-them-4a3328a4c50c

https://www.atlassian.com/agile/software-development/technical-debt

https://www.bmc.com/blogs/technical-debt-explained-the-complete-guide-to-understanding-and-dealing-with-technical-debt/

Skills relevant for the 21st Century and the role of Universities

What are the skills relevant for the coming decades? Are our institutions imparting them? Two important observations stand out especially in our Indian context:

1. There is a growing feeling that more and more focus, especially in India, is given to STEM at the cost of all other subjects. In fact, we can go on to say that those taking up humanities are frowned upon.
2. How relevant are the skills being imparted? Do they help the students fit into the society better and prepare them for solving the problems of tomorrow?

Assuming the key focus of education, amongst many others, is to prepare students for the job market, there is an inherent assumption that how the job market will look like and what skills would be necessary are well known. Sorry to throw a spanner in the works!  The rate of change is so fast that a research suggests that 35% of the skills necessary to thrive in the job market today will be different 5 years now. Former HCL Technologies CEO Vineet Nayar, once, remarked that 50% of revenues, 5 year thence, would come from services that did not exist in the present. Though the statement was made in the context of a specific industry, it applies equally well. Is this alarming? Yes and No.

Mr Bullrich, Education Minister of Argentina, succinctly summed it up:

Argentina is making rapid progress. In Finland, the “teachers” are the best paid and command a huge respect. Self-discovery, team play and soft skills dominate.

It is worth recalling the following comment:

“We prepare children to learn how to learn, not how to take a test,” said Pasi Sahlberg, a former math and physics teacher who is now in Finland’s Ministry of Education and Culture. “We are not much interested in PISA. It’s not what we are about.”

India has started opening up recently with a view to discovering the long lost “Liberal Arts and Sciences”. Universities like Ashoka, FLAME, Jindal, Shiv Nadar University and Krea are some examples where attempts are being made to help the students discover their true passion. It is true that data is going to be the oil of this century. That doesn’t mean that everyone should aim to become data scientist. A combination of analytical, technical, linguistic skills will be required. No problem of today can be understood and analysed within a single dimension. To solve the poverty problem, economic, data, sociological and technical skills will be required. It is all the more reason that team work will dominate.

Take a step back and go into our own old system. A student was expected to learn from a variety of areas starting from philosophy, logic, math etc. Then they were encouraged to discover the areas they would like to delve deep. Our very own interwoven system of learning was forced into oblivion and the western influence came. Instead of taking relevant aspects, we completely immersed ourselves into the new system and rewarded rote memory. In the process, we also encouraged people who take up science subjects look down on those who are passionate about history or literature!

Let’s take a look at the skills that are going to be required to survive and grow:

-          Complex problem solving
-          Critical thinking
-          Creativity
-          People management
-          Coordinating with others
-          Emotional intelligence
-          Judgement and decision making
-          Service orientation
-          Negotiation.

These skills also have a common characteristics i.e. they cannot be easily done by machines and would still be relevant in the artificial intelligence dominated world.

Coming back to the universities and institutions of learning, there is a rush to get higher and higher in the rankings. Common rankings include QS, ARWU etc. Singapore set up a target, pumped in resources, hired super star professors, increased throughput and systematically moved up higher. NUS (National University of Singapore) and NTU (Nanyang Technological University) are well in the top consistently. China’s Tsinghua, along with Korea, Japan and HK universities are in the top. We have our very own IITs/NITs and other eminent institutes. We bemoan the fact that we don’t have top ranking universities yet. There is a renewed thrust. Does this really mean the teaching is good and the students are churned to be future ready learners? We don’t know yet. One thing that these ranking may not take into account significantly is the effectiveness of teaching at the undergraduate levels. The ranking may not reflect this adequately. It may reward research or Post-graduate academics much more. This is where our institutions can make a big difference. This is not to say that we should stop aiming for high ranks!

How do you impart the quality of “Learning never stops”? Learning is not a reflection of what has happened so far. If that were the case, there would be only one way of doing things right. Today’s problems require not only analysis from multiple perspectives but require tools from different fields to solve. In Argentina, the new changes ensure the youth are provided opportunities to learn and practice in their community. We don’t see many Scandinavian universities in the top 50. I was speaking to one of my Swedish colleague. His words were to this effect – “There are no elite universities in Scandinavia and all are equal in terms of quality. Of course, some are known in specific areas of research. But students go to those universities not based on the rank but on a host of other factors including fellow students, teaching quality, opportunities for practice etc.” Their system of education has been ranked very high. Formal schooling starts at 6 or later. By that time, in India, we may have enrolled him/her in a coaching class!

Does this mean a student is allowed to roam around for 3 or 4 years of bachelor program tasting and sampling different courses just letting time fly? Not at all. If, at the end of the bachelor studies, a student cannot get up and articulate his or her basic understanding in a well-grounded context and mastery in a chosen area, the program may not have met its purpose! This is where I feel that the world is crying out for “specialists” who are grounded generalists. A biologist with love for poetry can only make the world richer. The same biologist can take up a lead role in a specific problem involving endangered species rehabilitation but prepare to take a secondary role in striking a balance between deforestation and controlled tourism perhaps allowing a environmental scientist take up a prime position. The world needs such people who can work in a team and take up varying positions that the problem demands and allow other perspectives to complement. 

Let’s allow our students to experiment, to make mistakes, to learn from others, to cultivate ability to live in the society and above all “have an appetite to continue learning”.

References:

1.       https://www.weforum.org/agenda/2016/10/the-most-important-skills-of-tomorrow-according-to-five-global-leaders/

2.       https://www.weforum.org/agenda/2016/01/the-10-skills-you-need-to-thrive-in-the-fourth-industrial-revolution

3.       https://www.nytimes.com/2013/06/23/opinion/sunday/the-decline-and-fall-of-the-english-major.html?smid=pl-share

4.       https://www.linkedin.com/pulse/20130625075243-2259773-why-learning-the-humanities-is-a-key-to-success

5.       http://www.teachercertification.org/a/role-of-education-in-the-21st-century.html

6.       https://www.insidehighered.com/blogs/call-action-marketing-and-communications-higher-education/it-time-say-goodbye-liberal-arts

7.       http://time.com/4920389/bowdoin-college-liberal-arts-education/