How to handle Technical Debt in Machine Learning projects?
Technical Debt is a key concept in IT. It's part of every innovation partnership, and it's an inalienable feature of most Machine Learning projects.
In fact, hardly any IT project can escape at least some form of Technical Debt in the long run. This is because sometimes it is planned and intentional, but there are some types where it's emergent and random. Obviously, if you can, you always want to have Technical Debt that is planned and exists because of conscious decisions.
While it is a quite common phenomenon, it often feels hard to describe. Many IT professionals use this expression in different ways and often apply it inconsistently.
We like to approach things with an engineering mindset. That also means we like to keep things simple, for the sake of clarity, productivity, and modularity.
So, as a working definition we could start with something like this:
Technical Debt is productive procrastination.
Essentially, the strategy involves setting aside tasks that aren't immediately critical, thereby allowing you to concentrate on the most pressing issues at hand. This approach necessitates revisiting deferred tasks at a later time. During this period, while Technical Debt is present, it may lead to a decline in productivity in specific operational areas. However, this trade-off is beneficial. By allowing minor issues to remain unaddressed for the time being, you effectively mitigate the largest risks or make significant progress in developing crucial, time-sensitive features. If you interested in exploring related topics, a wealth of information is available for further investigation.
Technical Debt definition
Technical debt is the result when the software developers expedite the delivery of functionality or project which later needs to be refactored. It's the result of prioritizing speed over perfect functionality.
The 'debt' is the result of this prioritization, and it could entail bugs, legacy code, missing documentation, or simply an underdeveloped feature that is currently capable of performing its task, but it places extra burdens on the whole system.
Technical debt represents extra development work that will need to be done later on, and which is a direct consequence of a decision that favored short-term gains over long-term goals.
Types of Technical Debt
Technical Debt can be different from project to project. One of the most important aspects of Technical Debt is how it came to be. How did it happen?
You can do it as an important part of a long-term process, it can happen due to technology upgrades, or simply because after a while, even small updates accumulate their own sets of small problems, and you end up with a slow, inefficient system.
The three kinds of Technical Debt, in terms of emergence:
- deliberate Technical Debt
- accidental/outdated design (new functionality becomes slow/difficult to implement)
- system complexity rises unintentionally due to many small, incremental changes over longer periods of time
How to manage Technical Debt?
You can't really avoid Technical Debt, but you can compound the risks.
Let's go through the types mentioned above because they all require a different approach:
In the case of deliberate Technical Debt, you need to make sure to keep track of your decisions. So, whenever someone says 'we are going to need to come back to this later" - it has to be tracked in a special backlog. Whoever makes these decisions, should be held accountable. It's not about the blame game, it's about making sure that the Debt is accounted for and it's not causing unexpected problems down the line.
The human element is key, you need to know who understands your code! Because people sometimes come and go, and a huge portion of the required knowledge is in their heads. When people change, Technical Debt can suddenly appear, because they take away that knowledge, so you need to identify this risk and make sure that the critical information is transferred.
In the case of outdated design, all you can do is lessen the probability of shifting business decisions and pivots. Because this kind of Technical Debt happens when a system is built in a certain way, but then comes a decision that alters the functionality, and now the old system needs to satisfy the new demands. This will never be optimal, of course. If this happens too many times, the complete redesign will become inevitable after a while.
Unintended complexity is similar to the previous type of Technical Debt, but changes are tiny and incremental. Every system needs some reorientation after a while because technology never just sits still. All you can do to lessen the negative impacts is to keep this in mind and plan ahead for regular checkups and cleanups to contain this process as much as possible.
Consider Lehmans' Laws:
1. A system must be continuously adapted or it becomes progressively less satisfactory
2. As a system evolves, its complexity increases unless work is done to maintain or reduce it
(There are 6 more of these laws, but the first two are enough to keep in mind if we want to understand that Technical Debt sometimes just 'happens', without even us doing anything).
Causes of Technical Debt
Technical Debt can be caused by:
- time pressure
- overly complex tech design
- poor alignment with standards
- lack of appropriate skills
- suboptimal code
- delayed refactoring
- insufficient testing
Consequences of Technical Debt
Different kinds of problems can arise from Technical Debt:
- reduced development pace
- volatile performance
- drained productivity
- testing strain
Technical Debt in Proof of Concept
Technical Debt is created by a conscious choice, at least in an optimal scenario. This is easiest to explain when we are talking about the Proof of Concept phase.
The PoC is a key stage of software development that is especially important in innovation partnerships and IT staff augmentation projects.
The PoC is a way to validate scalability, technical potential, and the key functionalities of your final product on a smaller scale.
This is the essential situation where you know that the methods you are using and the systems you are building are not final. They are far from optimized, you just try to prove a point here. (Read more about why the PoC is crucial and what is its role in a complex technical partnership, here. The article contains real examples from our case studies.)
How can you prevent Technical Debt?
Technical Debt is a natural phenomenon in IT, but it's risky. You can't fully prevent it, but you need to manage it.
One thing that often plagues IT projects is that coders spend way too much time fixing old code done by others. Education, communication, smooth transitions, and the human element, these things can save you a lot of trouble.
And, coming from engineers, this might sound a little self-centered, but developers need to be able to push back on managers and business leaders - the true costs and risks of Technical debt coming from arbitrary changes to the product need to be accounted for.
Lexunit will never tell you that something is impossible. We will adapt to any situation if that is the commitment you make, and you can expect to be made fully aware of all the repercussions and risks you are taking with that commitment.
Because we know that Technical Debt makes sense, and in a lot of cases, it's necessary! (like in the case of making it to a deadline, or getting the product out before the competition)
What you really want though is someone who tells you when you are going over the line. When your actions generate so much Technical Debt that the risk becomes mission critical. If too much Debt is accumulated, It can sink the whole business.
Managing Technical debt is a beautiful game of choices. It requires deep, layered thinking. It is always about the present, the " right now", but you need to have a good understanding of the long-term plans and challenges ahead of you.
Follow us for practical examples of choosing direction when you arrive at a software development crossroads, and other insights about utilizing Machine Learning in the startup world and beyond!