Stop Estimating Complex Problems – Do This Instead

Estimating is a common practice in various industries, including software development. It involves predicting the time and effort required to complete a project or task. However, estimating can be challenging and becomes a futile exercise when it comes to complex software development projects, especially when trying to estimate absolute numbers.In this article, we will explore why estimating complex software development tasks is problematic and discuss alternative approaches that can lead to better outcomes.

Understanding the nature of complex problems

Software development is typically categorized as a complex problem. While simple problems such as building a house have clear and defined steps that give rise to few surprises, complex problems like software development present different challenges. Unlike building a house, developing software often involves facing entirely new problems with no established solutions. Just like writing a book. You do not know the final product when you start; it evolves while writing.At borisgloger consulting, we use the Stacey Matrix to explain in which setup you are. The simple domain (house construction) has known requirements and known technology. On the other hand, the complex domain is characterized by uncertainty and the need for emergent solutions, as both requirements and technology are not fully known.

When estimating complex software development tasks, it's important to recognize that the solutions to these problems cannot be designed in advance. Instead, they emerge from system interactions through trial and error. Estimating such tasks is inherently difficult, as you are trying to predict the unknown.

The challenges of estimating

The moment you understand that software development is a complex problem, your confidence in estimates drops sharply. Trying to estimate a complex problem is an act of funambulism.Imagine you go to a mechanic with a car problem and demand that they give you an estimate without opening the hood. No mechanic would do that. Or even worse, they would give you the highest possible quote to cover their own back.When we ask a software engineer to estimate a user story, we often act like we are visiting said mechanic. We want them to estimate without giving them time to look at the engine. And then, we want to pretend that the estimates will be met.When years of experience lead you to understand this aspect, the next revelation comes when you discover that the only way to estimate a complex task correctly is to do it. There are simply too many biases. So, to estimate exactly how long it will take to accomplish a task, you need to solve it.And then you realize the futility of estimating in most cases and that software development is more like creating a symphony than constructing a building. Software development is more art than science.Following that reasoning, in most cases, estimating is irrelevant to the success of a product. In many cases, it is nothing more than a self-deception tool that we use to pretend that we are in control so that we can sleep more soundly at night, due to the following reasons:

• Lack of established knowledge: In many cases, complex software development tasks involve working on problems that have not been encountered before. There is no solid body of knowledge to rely on when estimating the time and effort required. Without historical data or established best practices, it becomes difficult to accurately estimate the task.
• Emergent solutions and unforeseen issues: In complex software development, solutions often emerge through trial and error. This means that the development team can only anticipate some of the issues they will encounter during the process. Unforeseen complexities can arise, leading to delays and challenges in meeting estimated timelines.
• Varied expertise and skillsets: Software development teams are typically made up of individuals with diverse backgrounds and skillsets. Estimating complex tasks requires a deep understanding of the problem domain and the specific technologies involved. It can be challenging to accurately assess the capabilities and expertise of each team member and factor that into the estimation process.
• Changing requirements and scope creep: Software development projects are prone to evolving requirements and scope creep. As the project progresses, new features may be added, priorities may shift, and client expectations may evolve. Estimating in such a dynamic environment becomes even more challenging, as the initial assumptions and estimates may no longer hold.
• Overconfidence and unrealistic expectations: Estimating complex software development tasks can create an illusion of control and certainty. Stakeholders may rely heavily on the estimates provided, expecting them to be met without considering the inherent uncertainty. This overconfidence can lead to unrealistic expectations and disappointment when the estimates are not met.

Are there any benefits of estimating?

If you've gotten this far, you already know that my confidence in estimating is low, and therefore, I tend to rely on it less. However, for junior teams and scaled teams, the act of estimating has certain advantages, as long as you don't believe too much in the outcome of the estimates.The main advantage is that, as long as you don't waste too much time, the estimation ritual is a good moment to detect which stories or functionalities are more or less complex. All those low estimates can be assumed to be simple problems that the team already has some experience in solving. These estimates are credible, but at the same time, they will not be very relevant to the future of the project.When a high estimate appears, pay attention, because it is probably a complex problem that the team has not faced before and, therefore, lacks experience. And precisely because they have no experience, you can't believe the estimate. A week can be a day or a month. Until the engineers get on the task, until they open the hood and find out what’s beneath it, the reality is that they can't know. Paradoxically, these tasks, invaluable beforehand, are the ones that will actually define the success of the project.But back to the advantages. The advantage is not to get a number that you can believe and schedule. The advantage is to discover these high-uncertainty tasks during the estimation so that you can react. You may discover that a detail you thought was minor is a complex problem. This is the best-case scenario because you can eliminate it, like removing a mirror from a bathroom design.In the worst-case scenario, the most common when you make worthwhile software, you will discover that the greatest degree of uncertainty is concentrated in the product's kernel. The task is valuable precisely because no one has done it before, and that is where the uncertainty is born. And, where the reward lies. In these cases, you will need to spend time trying to simplify the achievement of your goal. If what you want to do is too complicated, are there ways to validate it in a simpler way? Generally, you will have to break the problem into smaller subproblems that the team can develop and validate iteratively.Another advantage of estimating is the conversation between team members during the process. This is important in junior teams because it helps everyone to get a feel for the system through these discussions.

Promising alternative approaches to estimating

While estimating complex software development projects has its limitations, there are alternative approaches that can lead to better outcomes. These approaches focus on embracing uncertainty and adopting iterative and adaptive strategies. Here are some alternatives to consider:

• Embrace agile methodologies!Edison, Tesla, Curie, the Wright brothers... the greatest scientists and inventors in history have been craftsmen advancing experiment by experiment, making small adjustments and checking what changed from the previous iteration. To create successful products, you have to be like them. Accept uncertainty and adopt ways of working to reduce it, shortening development cycles and simplifying processes. Agile methodologies like Scrum and Kanban are well-suited for complex software development projects. They have development cycles as small as possible that allow you to iterate and course-correct step by step. By breaking down projects into smaller, manageable tasks, teams can deliver incremental value while continuously learning and adapting to changing requirements.
• Use story points instead of time estimates!Story points are a relative measure of complexity rather than a specific time estimate. This approach allows teams to compare the size and complexity of different tasks without committing to specific timelines. Story points provide a more flexible and less deterministic way of estimating in complex software development. You should always estimate relatively to a reference story to speed up the process. Magic estimation is a great tool for this. Check my last article and Miro board template to moderate such a session. You can estimate a whole backlog in only one hour.
• Prioritize learning and experimentation!Instead of focusing solely on estimation, prioritize learning and experimentation. Encourage teams to explore different approaches, prototype solutions, and gather feedback early and often. This iterative and experimental mindset reduces the reliance on upfront estimation. It allows for course correction based on real-world feedback.
• Emphasize collaboration and communication!Effective collaboration and communication are essential in complex software development. Encourage open dialogue among team members, stakeholders, and clients. Regularly review and refine project goals, requirements, and priorities. This collaborative approach helps align expectations and reduces the need for rigid estimations.
• Make complex tasks more manageable!To tackle complex tasks effectively, break them down into smaller subproblems. This approach allows teams to focus on solving specific challenges incrementally, gaining insights and reducing uncertainty along the way. By addressing smaller subproblems, teams can validate assumptions and make informed decisions.

Conclusion

Estimating complex software development tasks is challenging, often leading to unreliable and unrealistic expectations. Recognizing the limitations of estimation and going for alternative approaches can lead to better outcomes. By embracing uncertainty, adopting agile methodologies, and prioritizing learning and experimentation, software development teams can navigate the complexities of their projects more effectively.Remember, software development is more art than science, and success lies in embracing the unknown and adapting along the way.Photo from amriphoto / istock