Estimating is a common practice in various industries, including software development. It involves predicting the time and effort required to complete a project or task. However, estimating can be challenging and becomes a futile exercise when it comes to complex software development projects, especially when trying to estimate absolute numbers.In this article, we will explore why estimating complex software development tasks is problematic and discuss alternative approaches that can lead to better outcomes.
Software development is typically categorized as a complex problem. While simple problems such as building a house have clear and defined steps that give rise to few surprises, complex problems like software development present different challenges. Unlike building a house, developing software often involves facing entirely new problems with no established solutions. Just like writing a book. You do not know the final product when you start; it evolves while writing.At borisgloger consulting, we use the Stacey Matrix to explain in which setup you are. The simple domain (house construction) has known requirements and known technology. On the other hand, the complex domain is characterized by uncertainty and the need for emergent solutions, as both requirements and technology are not fully known.
When estimating complex software development tasks, it's important to recognize that the solutions to these problems cannot be designed in advance. Instead, they emerge from system interactions through trial and error. Estimating such tasks is inherently difficult, as you are trying to predict the unknown.
The moment you understand that software development is a complex problem, your confidence in estimates drops sharply. Trying to estimate a complex problem is an act of funambulism.Imagine you go to a mechanic with a car problem and demand that they give you an estimate without opening the hood. No mechanic would do that. Or even worse, they would give you the highest possible quote to cover their own back.When we ask a software engineer to estimate a user story, we often act like we are visiting said mechanic. We want them to estimate without giving them time to look at the engine. And then, we want to pretend that the estimates will be met.When years of experience lead you to understand this aspect, the next revelation comes when you discover that the only way to estimate a complex task correctly is to do it. There are simply too many biases. So, to estimate exactly how long it will take to accomplish a task, you need to solve it.And then you realize the futility of estimating in most cases and that software development is more like creating a symphony than constructing a building. Software development is more art than science.Following that reasoning, in most cases, estimating is irrelevant to the success of a product. In many cases, it is nothing more than a self-deception tool that we use to pretend that we are in control so that we can sleep more soundly at night, due to the following reasons:
If you've gotten this far, you already know that my confidence in estimating is low, and therefore, I tend to rely on it less. However, for junior teams and scaled teams, the act of estimating has certain advantages, as long as you don't believe too much in the outcome of the estimates.The main advantage is that, as long as you don't waste too much time, the estimation ritual is a good moment to detect which stories or functionalities are more or less complex. All those low estimates can be assumed to be simple problems that the team already has some experience in solving. These estimates are credible, but at the same time, they will not be very relevant to the future of the project.When a high estimate appears, pay attention, because it is probably a complex problem that the team has not faced before and, therefore, lacks experience. And precisely because they have no experience, you can't believe the estimate. A week can be a day or a month. Until the engineers get on the task, until they open the hood and find out what’s beneath it, the reality is that they can't know. Paradoxically, these tasks, invaluable beforehand, are the ones that will actually define the success of the project.But back to the advantages. The advantage is not to get a number that you can believe and schedule. The advantage is to discover these high-uncertainty tasks during the estimation so that you can react. You may discover that a detail you thought was minor is a complex problem. This is the best-case scenario because you can eliminate it, like removing a mirror from a bathroom design.In the worst-case scenario, the most common when you make worthwhile software, you will discover that the greatest degree of uncertainty is concentrated in the product's kernel. The task is valuable precisely because no one has done it before, and that is where the uncertainty is born. And, where the reward lies. In these cases, you will need to spend time trying to simplify the achievement of your goal. If what you want to do is too complicated, are there ways to validate it in a simpler way? Generally, you will have to break the problem into smaller subproblems that the team can develop and validate iteratively.Another advantage of estimating is the conversation between team members during the process. This is important in junior teams because it helps everyone to get a feel for the system through these discussions.
While estimating complex software development projects has its limitations, there are alternative approaches that can lead to better outcomes. These approaches focus on embracing uncertainty and adopting iterative and adaptive strategies. Here are some alternatives to consider:
Estimating complex software development tasks is challenging, often leading to unreliable and unrealistic expectations. Recognizing the limitations of estimation and going for alternative approaches can lead to better outcomes. By embracing uncertainty, adopting agile methodologies, and prioritizing learning and experimentation, software development teams can navigate the complexities of their projects more effectively.Remember, software development is more art than science, and success lies in embracing the unknown and adapting along the way.Photo from amriphoto / istock