Stop Estimating Complex Problems – Do This Instead

Estimating is a common practice in various industries, including software development. It involves predicting the time and effort required to complete a project or task. However, estimating can be challenging and becomes a futile exercise when it comes to complex software development projects, especially when trying to estimate absolute numbers.In this article, we will explore why estimating complex software development tasks is problematic and discuss alternative approaches that can lead to better outcomes.

Understanding the nature of complex problems

Software development is typically categorized as a complex problem. While simple problems such as building a house have clear and defined steps that give rise to few surprises, complex problems like software development present different challenges. Unlike building a house, developing software often involves facing entirely new problems with no established solutions. Just like writing a book. You do not know the final product when you start; it evolves while writing.At borisgloger consulting, we use the Stacey Matrix to explain in which setup you are. The simple domain (house construction) has known requirements and known technology. On the other hand, the complex domain is characterized by uncertainty and the need for emergent solutions, as both requirements and technology are not fully known.

Stacy Matrix Grafik

When estimating complex software development tasks, it's important to recognize that the solutions to these problems cannot be designed in advance. Instead, they emerge from system interactions through trial and error. Estimating such tasks is inherently difficult, as you are trying to predict the unknown.

The challenges of estimating

The moment you understand that software development is a complex problem, your confidence in estimates drops sharply. Trying to estimate a complex problem is an act of funambulism.Imagine you go to a mechanic with a car problem and demand that they give you an estimate without opening the hood. No mechanic would do that. Or even worse, they would give you the highest possible quote to cover their own back.When we ask a software engineer to estimate a user story, we often act like we are visiting said mechanic. We want them to estimate without giving them time to look at the engine. And then, we want to pretend that the estimates will be met.When years of experience lead you to understand this aspect, the next revelation comes when you discover that the only way to estimate a complex task correctly is to do it. There are simply too many biases. So, to estimate exactly how long it will take to accomplish a task, you need to solve it.And then you realize the futility of estimating in most cases and that software development is more like creating a symphony than constructing a building. Software development is more art than science.Following that reasoning, in most cases, estimating is irrelevant to the success of a product. In many cases, it is nothing more than a self-deception tool that we use to pretend that we are in control so that we can sleep more soundly at night, due to the following reasons:

  • Lack of established knowledge: In many cases, complex software development tasks involve working on problems that have not been encountered before. There is no solid body of knowledge to rely on when estimating the time and effort required. Without historical data or established best practices, it becomes difficult to accurately estimate the task.
  • Emergent solutions and unforeseen issues: In complex software development, solutions often emerge through trial and error. This means that the development team can only anticipate some of the issues they will encounter during the process. Unforeseen complexities can arise, leading to delays and challenges in meeting estimated timelines.
  • Varied expertise and skillsets: Software development teams are typically made up of individuals with diverse backgrounds and skillsets. Estimating complex tasks requires a deep understanding of the problem domain and the specific technologies involved. It can be challenging to accurately assess the capabilities and expertise of each team member and factor that into the estimation process.
  • Changing requirements and scope creep: Software development projects are prone to evolving requirements and scope creep. As the project progresses, new features may be added, priorities may shift, and client expectations may evolve. Estimating in such a dynamic environment becomes even more challenging, as the initial assumptions and estimates may no longer hold.
  • Overconfidence and unrealistic expectations: Estimating complex software development tasks can create an illusion of control and certainty. Stakeholders may rely heavily on the estimates provided, expecting them to be met without considering the inherent uncertainty. This overconfidence can lead to unrealistic expectations and disappointment when the estimates are not met.

Are there any benefits of estimating?

If you've gotten this far, you already know that my confidence in estimating is low, and therefore, I tend to rely on it less. However, for junior teams and scaled teams, the act of estimating has certain advantages, as long as you don't believe too much in the outcome of the estimates.The main advantage is that, as long as you don't waste too much time, the estimation ritual is a good moment to detect which stories or functionalities are more or less complex. All those low estimates can be assumed to be simple problems that the team already has some experience in solving. These estimates are credible, but at the same time, they will not be very relevant to the future of the project.When a high estimate appears, pay attention, because it is probably a complex problem that the team has not faced before and, therefore, lacks experience. And precisely because they have no experience, you can't believe the estimate. A week can be a day or a month. Until the engineers get on the task, until they open the hood and find out what’s beneath it, the reality is that they can't know. Paradoxically, these tasks, invaluable beforehand, are the ones that will actually define the success of the project.But back to the advantages. The advantage is not to get a number that you can believe and schedule. The advantage is to discover these high-uncertainty tasks during the estimation so that you can react. You may discover that a detail you thought was minor is a complex problem. This is the best-case scenario because you can eliminate it, like removing a mirror from a bathroom design.In the worst-case scenario, the most common when you make worthwhile software, you will discover that the greatest degree of uncertainty is concentrated in the product's kernel. The task is valuable precisely because no one has done it before, and that is where the uncertainty is born. And, where the reward lies. In these cases, you will need to spend time trying to simplify the achievement of your goal. If what you want to do is too complicated, are there ways to validate it in a simpler way? Generally, you will have to break the problem into smaller subproblems that the team can develop and validate iteratively.Another advantage of estimating is the conversation between team members during the process. This is important in junior teams because it helps everyone to get a feel for the system through these discussions.

Promising alternative approaches to estimating

While estimating complex software development projects has its limitations, there are alternative approaches that can lead to better outcomes. These approaches focus on embracing uncertainty and adopting iterative and adaptive strategies. Here are some alternatives to consider:

  • Embrace agile methodologies!Edison, Tesla, Curie, the Wright brothers... the greatest scientists and inventors in history have been craftsmen advancing experiment by experiment, making small adjustments and checking what changed from the previous iteration. To create successful products, you have to be like them. Accept uncertainty and adopt ways of working to reduce it, shortening development cycles and simplifying processes. Agile methodologies like Scrum and Kanban are well-suited for complex software development projects. They have development cycles as small as possible that allow you to iterate and course-correct step by step. By breaking down projects into smaller, manageable tasks, teams can deliver incremental value while continuously learning and adapting to changing requirements.
  • Use story points instead of time estimates!Story points are a relative measure of complexity rather than a specific time estimate. This approach allows teams to compare the size and complexity of different tasks without committing to specific timelines. Story points provide a more flexible and less deterministic way of estimating in complex software development. You should always estimate relatively to a reference story to speed up the process. Magic estimation is a great tool for this. Check my last article and Miro board template to moderate such a session. You can estimate a whole backlog in only one hour.
  • Prioritize learning and experimentation!Instead of focusing solely on estimation, prioritize learning and experimentation. Encourage teams to explore different approaches, prototype solutions, and gather feedback early and often. This iterative and experimental mindset reduces the reliance on upfront estimation. It allows for course correction based on real-world feedback.
  • Emphasize collaboration and communication!Effective collaboration and communication are essential in complex software development. Encourage open dialogue among team members, stakeholders, and clients. Regularly review and refine project goals, requirements, and priorities. This collaborative approach helps align expectations and reduces the need for rigid estimations.
  • Make complex tasks more manageable!To tackle complex tasks effectively, break them down into smaller subproblems. This approach allows teams to focus on solving specific challenges incrementally, gaining insights and reducing uncertainty along the way. By addressing smaller subproblems, teams can validate assumptions and make informed decisions.

Conclusion

Estimating complex software development tasks is challenging, often leading to unreliable and unrealistic expectations. Recognizing the limitations of estimation and going for alternative approaches can lead to better outcomes. By embracing uncertainty, adopting agile methodologies, and prioritizing learning and experimentation, software development teams can navigate the complexities of their projects more effectively.Remember, software development is more art than science, and success lies in embracing the unknown and adapting along the way.Photo from amriphoto / istock

Agile Toolbox
Scrum
Estimation
ScrumMaster
Steffen Bernd
November 14, 2023

Table of content

Diesen Beitrag teilen

Das könnte auch interessant sein:

Agile Coach
Agile Management
Agile Organization
Agile Prinzipien
Agile Toolbox

FRAGE: Warum macht ihr eigentlich kein SAFe?

Agile Management
Agile Organization
Agile Prinzipien
Agile Toolbox
Transformation

FRAGE: Was kostet eine agile Transformation?

Agile Management
Agile Organization
Agile Toolbox
Leadership
Agiles Lernen

FRAGE: Welche Rolle spielt Training?

Agile Coach
Agile Management
Agile Organization
Agile Prinzipien
Agile Toolbox

FRAGE: Wer sind die Top 10 agilen Unternehmensberatungen?

Agile Management
Agile Organization
Agile Tools
Agiles Management
Leadership

FRAGE: Wie viel bringt die Investition? Was ist der Business Case dahinter?

Agile Management
Agile Organization
Agile Prinzipien
Agile Toolbox
Führung

FRAGE: Welche sind häufige Herausforderungen, die ihr beim Kunden löst?

Agile Management
Agile Organization

FRAGE: Warum sollten wir mit borisgloger arbeiten?

Agile Management
Agile Organization
Agiles Management
Transformation

FRAGE: Wie viel kostet eine Beratung und ist es wirklich rentabel bei borisgloger?

Agile Prinzipien
Agile Toolbox
Projektmanagement

The Lie Behind the Parable of the Golf Balls and the Jar

Video
Change
Digitale Transformation
Hardware
Agile Organization

Agile in Industrial Automation: The Digital Transformation of Yokogawa

Versicherung
Neues Arbeiten
Führung
Agile Prinzipien
Kundenfokus

Kundenzentrierte Versicherung: Kann ein agiles Projekt die Organisation retten?

Versicherung
Change
Digitale Transformation
Agile Prinzipien
Kundenfokus

Agilität in den Vertrieb bringen – für Versicherer sinnvoll

Versicherung
Agile Prinzipien
Kundenfokus
Agile Toolbox
Produktentwicklung

BizDevOps in der Versicherungsbranche – Wie multidisziplinäre Teams wirklich besetzt sein sollten

Versicherung
Agile Prinzipien
Kundenfokus
Neues Arbeiten
Meetings

Undercover Agile für Versicherer: 5 agile Praktiken für Ihr klassisches IT-Projekt

Versicherung
Change
Digitale Transformation
Agile Prinzipien
Kundenfokus

IT-Projekte in der Versicherungsbranche – Das Rennen um die Time-to-Market

Team
Neues Arbeiten
Agile Prinzipien
Selbstorganisation
Social Skills

Umgang mit Fehlern & Diversität – Erfolgreiche agile Teams #2

Team
Neues Arbeiten
Agile Toolbox
Produktentwicklung

Das Geheimrezept von High-Performance-Teams

Team
Arbeiten bei borisgloger consulting
Agile Prinzipien
Freiwilligkeit
Selbstorganisation

Konsent und offene Wahl: 2 Prinzipien aus der Soziokratie, die jedes agile Team gebrauchen kann

Team
Neues Arbeiten
Meetings
Social Skills

Der agile Adventkalender

Team
Agile Toolbox
Scrum
ScrumMaster-Praxistipps
Agile Prinzipien

Selbstorganisation der Teams fördern: Ask the team!

Team
Agile Toolbox
Design Thinking

Who Recognizes the Truly Good Ideas?

Team
Agile Organization
Transformation

Pizza Is Not Dead, and Neither Is Agility

Scrum4Schools
Neues Arbeiten
Führung
Life
Social Skills

Trauen wir unseren Kindern mehr zu – auch in der Schule!

Scrum4Schools
Change
Agiles Lernen
Neues Arbeiten
Remote Arbeiten

Eine Scrum4Schools-Projekt-Rückschau mit Physiklehrer Ivan Topic

Scrum4Schools
Mehr Formate
Interview
Nachhaltigkeit

Mit Scrum4Schools dem Weltraum auf der Spur

Scrum4Schools
Change
Agiles Lernen

Scrum4Schools - ein Projekt nimmt Fahrt auf

Scrum4Schools
Agile Schulentwicklung
Agile Toolbox

Technik im Alltag - Scrum4Schools zu Gast in Langenzersdorf

Projektmanagement
Agile Toolbox
Scrum
Scrum-Begriffe
ScrumMaster-Praxistipps

Sprechen Sie Agile? Den klassischen Projektplan in die agile Welt überführen

Projektmanagement
Agiles Management
Agile Toolbox
Scrum
Enterprise Scrum

Das Management in Scrum

Projektmanagement
Change
Digitale Transformation

Agilität in der Logistik oder: Liefern wie Amazon

Projektmanagement
Agile Toolbox
Scrum

Meilensteine und Scrum

Portfoliomanagement
Project management

Too many projects? Portfolio management simplified

Neues Arbeiten
Mehr Formate
Agile Toolbox
Scrum
Scrum Values

Wie agiles Arbeiten die Kommunikation aus der Selbstverständlichkeit holt

Neues Arbeiten
Change
Agiles Lernen
Mehr Formate
Audio

New Learning heute für das New Work von morgen – mit Angelika Weis

Neues Arbeiten
Change
Soziale Innovation

New Work Experience 2019 – ein Erfahrungsbericht

Neues Arbeiten
Audit
Change

Agil im Audit: das Starter-Kit

Neues Arbeiten
Agile Toolbox
Scrum
Scrum4Schools
Agile Prinzipien

Scrum4Schools: Lernen für die Zukunft

Neues Arbeiten
Agile Toolbox
Scrum
Scrum Meetings
Retrospektive

Arbeiten wir uns gesund!

Neues Arbeiten
Agile Toolbox
Scrum
ScrumMaster-Praxistipps

Who should be in (agile) HR?

Neues Arbeiten
Agile Toolbox
Scrum
Scrum Values

Glauben Sie an die Seele Ihrer Firma?

Neues Arbeiten
Agile Toolbox
Scrum
Product Owner
ScrumMaster-Praxistipps

Produktivität auf Irrwegen: "Führen wir schnell mal Scrum ein!"

Neues Arbeiten
Agile Prinzipien
Selbstorganisation
Social Skills
Team

Freiwilliges Teilen von Wissen – Erfolgreiche agile Teams #5

Neues Arbeiten
Agile Prinzipien
Selbstorganisation
Social Skills
Team

Doing vs. Being Agile – Erfolgreiche agile Teams #1

Neues Arbeiten
Agile Prinzipien
Selbstorganisation
Social Skills
Team

Freude bei der Arbeit & Sustainable Pace – Erfolgreiche agile Teams #3

Neues Arbeiten
Agile Prinzipien
Selbstorganisation
Social Skills
Team

Anpassungsfähigkeit & schonungslose Offenheit – Erfolgreiche agile Teams #4