Necessary vs. Accidental Complexity in Software Engineering

Software engineering is a complex process that involves creating, maintaining, and updating code. Over time, the complexity of software can grow to the point where it becomes difficult and expensive to maintain, especially for teams of dozens or hundreds of developers. However there are two types of complexity, necessary and accidental, and understanding the difference between them is an important distinction.

Necessary complexity

Necessary complexity refers to the complexity that is inherent to the problem being solved by the software. For example, if a software system needs to manage data from multiple sources, there will be inherent complexity involved in handling that data. For example your company needs to aggregate information from multiple APIs and do calculations before handing off to a third-party, this type of complexity cannot be avoided, and is necessary to solve the problem.

This type of complexity stems from the actual business logic of the application being built and cannot really be simplified, at least on the engineering side. It is worth asking questions about whether the business logic is necessary or overly complex, but once those decisions have been made the engineering logic is pretty much set in stone.

Accidental complexity

Accidental complexity, on the other hand, refers to complexity that is introduced by the software development process itself. This can include things like complicated code structures, overly complex algorithms, or an overabundance of features that are not essential to the core functionality of the software.

One common cause of accidental complexity is overly complex code structures, which can make it difficult to understand how different parts of the system work together. This can include things like nested conditionals, deep object hierarchies with unnecesary dependencies, or complex data structures. Abstracting complexity into smaller methods and modules can help alleviate some of these issues. Others can be nipped in the bud by discussions around system design best practices.

In some cases, developers may implement algorithms that are much more complex than necessary, which can lead to code that is more difficult to maintain. This might stem from attempting to optimize too early for performance when it may not be necessary, or potentially from handling edge cases that may be so rare they would better be left for exceptions. I've even seen this come from a very smart engineer attempting to show off by shoehorning obscure parts of a language or framework and makes the code harder to understand for a less experienced engineer.

Accidental complexity can also result from the use of an overabundance of features and libraries that are not essential to the core functionality of the software. For example, there are hundreds of front-end frameworks and CSS libraries that are now available but does your MVP really need React or Tailwind? It might be simpler and more maintainable to stick with vanilla JavaScript and CSS, using best practices or common patterns.

Benefits of reducing accidental complexity

For the engineering team this means less time spent on technical debt and bug fixing, allowing for more meaningful changes to be introduced to the codebase at a higher rate with increased confidence. This leads to higher satisfaction with the work being done and fewer PagerDuty calls. Time spent onboarding new engineers will be reduced and developers joining the team will be able to understand the codebase and make meaningful contributions that much quicker.

Most importantly for stakeholders, reducing complexity means saving on cost. Fewer engineers are required to make changes, including building new features. Granted these engineers can and should be towards the higher end of the pay scale individually, but if they are able to build sustainable software then the savings overall are still exponential. When engineers are happy and productive, so are the engineering managers and product managers. Happy teams make successful products for stakeholders.

Conclusion

Software complexity is an often quasi-invisible metric that means the difference between a buggy, inferior product with disatisfied engineers and a successful, high-performing engineering team that is ready to grow.

It should be considered of utmost importance to focus on keeping complexity as small as is necessary by the business requirements from the very beginning if you are interested in developing the best product possible.