kjuulh/kasperhermansen-blog
1
0
Fork 0
Code Issues 1 Pull Requests 2 Projects Releases Wiki Activity

feat: write incidental complexty
Some checks failed
continuous-integration/drone/push Build encountered an error
Details

Browse Source 
Signed-off-by: kjuulh <contact@kjuulh.io>
This commit is contained in:
Kasper Juul Hermansen
2026-01-25 20:11:29 +01:00
parent 7827e2e895
commit 01e8230f82
2 changed files with 157 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files

99
content/posts/2026-01-25-accidental-complexity.md
View File

@@ -1,99 +0,0 @@
---
type: blog-post
title: "Incidental Complexity"
description: |
Complexity is a required aspect of developing products, in especially software, complexity is sometimes inherited, molded and incidental.
In this post we discuss incidental complexity in software development; history, present and speculation on the future.
draft: false
date: 2026-01-25
updates:
- time: 2026-01-25
description: first iteration
tags:
- "#blog"
- "#softwaredevelopment"
- "#ai"
- "#complexity"
- "#techdebt"
---
Complexity is a required side effect of developing products, whether it is in processes, human relations and especially software, which we will discuss in this post.
Complexity is sometimes inherited, you have to fix an issue in legacy system, you are making it fit for purpose and either expand or refactor its capabilities and internal workings. Or Incidental which we will cover in this post; Whether you add more than you should to cover a need you think you have. But it turns out is over-engineered, or simply redundant. That is not to say the code isn't used at all, But it is extra work that could've been simpler, less handling of edge cases that aren't testet etc.
## Incidental complexity has always been a thing, and always will be
Complexity is a feature not a bug, one of the main benefits of software is that it is so maluable, it can be changed much more easily than a hardware product for example; It is simply much easier to change a line of code, flash some firmware, than redo a production line for a hardware trace. Software also has no permanency outside of organisational or environmental factors, other than software having to live up to some requirement, whether it be a functional one, has to fulfilll a purpose, or non-functional it being a certain shape to reliably handle a certain issue, a trading algorithm having a certain shape because it needs to be fast enough to beat its competitors.
Complexity is born when a decision, incidental or not is added to a product, it might be a choice of programming language, coding style, infrastructure, and so on. None of them are inherently good or bad, but they're choices that exists on a scale of how easy they are to discover and change for that given product, and it varies depending on many factors, whether it is firmware to be distributed once, over the air, live in a data center, website which can change many times pr day.
I define two subsets of complexity, emergent and incidental, the goal of most software is to be the minimal amount of software required to fulfill the set of requirements applied to a software product, this is not by lines of code! Rather it is a measure how how much context we apply to the code for it to do its job. That is emergent complexity, incidental complexity is when we apply too many details to the context, not relevant or additional to solve the set of requirements.
For example: A requirement might be to show a page of users on webpage. The list is designed to come from a database, and there are infinite ways of solving this problem.
1. A casual solution might be to simply use the database client, to fetch items one by one, until we can populate the webpage. In this context, it fits fine, because the service will never have more than 10 users, so it is fast enough to be valid, small enough to be changeable, and just perfect for this use-case.
2. A complex solution would cache the amount of users in the database, divide it to the max amount of items the database supports returning at once, and query it all in parallel all at once, to then stitch back the result and populate the webpage.
You might like one solution better than the other, or prefer a third, it doesn't matter. In this case, both solutions are valid answers to the problem, both produce the same answer, but not using the same calculations. If we follow the scenario that the database will never have more than 10 users, this means that all the logic of handling many thousands of users in the complex solution while used, is never actually utilized, if the batch size is 500 or so, then we never do any parallel processing, never need the count because we never need to split, we never need to stitch anything together.
There is no right and wrong answer, the first solution was built knowing that 10 users was the max, so it was built to handle that within reason, and it is simple enough to be changed if this ever changes. The second solution could have been built with the ambition of handling thousands of users at once, and needed a performant page load.
Incidental complexity is when there are additional details, that have no minimal effect on the actual output, non-functional or not. You might be screaming at your computer now, that it is just premature complexity, I define premature complexity as a step in the process of ending up with incidental complexity.
It is impossible to hit the mark 100% correctly, as complexity is subjective, to an experienced team, with a wide toolbox; simply fetching 1000 items because they know it is within budget might be a small price to others it might be simpler to just fetch one by one. There is also no correct answer, what is important is that software is built to requirements, with minimal additional details. It should feel like the "natural" solution to a problem given the organisational context and problem domain.
Incidental complexity is those details that sometimes show up as: Curious why is the solution using batching, when there is only one item to fetch?, Or why does this use a strategy pattern, when there is only one option? Some of these complexities can come to fruition, but at the moment are not a useful part of the solution to the problem. You can probably think of endless examples of these, that you've done over your career, or inherited.
## Devil in the details
It can be very difficult to tell the difference between emergent and incidental complexity, literaly the devil in the details. It might be, to continue the previous example, at easter every year, volunteers across Denmark is added to the database, so instead of 10, it is now 10000, and as such the page would be impossible to load unless a sufficiently complex solution exists.
This leaves complexity with some permanency, it can be very difficult to reduce, because we've often been bitten by this in the past, when doing rewrites, or simply house-keeping. The team in the past, probably left that there for some reason.
You can never know, and sometimes you have to take the gamble, and it takes real skill, luck and intention to do so. It is a risk, and risk is hard to stomach.
In general we're generally more prone to leave complexity in as a feature of the system when inherited vs when writing new code, one of the reasons why it is often easier to solve a subset of a problem from scratch, rather than updating an inherited solution.
## Human VS AI (LLM)
Humans and AI produce incidental complexity in similar but very crucially different ways. A human will apply their knowledge and style of programming to a problem and end up with a given solution. A human is often neatly aware of the requirements and the context, but is of course still fuzzy in their exact details. A human will produce incidental complexity simply because they've got a certain style of solving problems given their knowledge.
An AI will have wide fuzzy knowledge, it doesn't have taste, but it has a style, because it has the amalgam of knowledge, and has been tuned to produce the most readily acceptible answers. AIs have been optimized for getting their answers chosen, solving problems is part of that selection, but so is clarity, verboseness and to which degree the problem should be solved. An AI is often and likely will have a low level of the requirements and context at hand. It is simply too difficult to convey context over the text media we've got today, a large part of the engineering profession is the engineer coming up with the concrete requirements from the client's context.
This means that given their information vacuum AIs often end up producing solutions that on the outset looks to work, and sort of does, but either leaves out crucial details, or produces either functional or non-funtional requirements that weren't part of the original ask.
> Currently there is also the problem with AIs ephemeral context, it simply forgets details after a while, which often leads to a catastrophic loss of performance. I won't go into this here, as even if this is solved, I still see AI agents producing different incidental complexity than humans.
AIs and Humans are tuned to different needs, a human is roughly tuned with their values, experience, input, and context, this leads to an effective but relatively slow producer of software. LLMs on the other hand are trained on all software the producers could get their hands on, no matter the quality, it is tuned to produce answers that are accepted, whether that is code or text. This is often glossed over but is crucially important, they need an answer that can be accepted, not the correct answer. This is the same as a human lying to their parents on them failing a grade. Lying is a feature of how LLMs are produced, and it won't go away (speculation, but I've not seen anything that would stop this behavior), they're biased towards action even on no to little context, and they're prompted to require as little follow up as possible to get an answer.
We've often seen this show up as LLMs sprinkling comments everywhere explaining what it is doing, no matter the complexity, or when promoted for text, it is by default tuned to produce a paragraph, a few bullet points then a closing paragraph. These can be influenced, but the style of communication is pretty much always followed. The same goes for programming. The agent will given no examples produce a fairly average solution to said problem, however the complexities I've found at this current point in time (2026), the AIs will sprinkle additional details onto a solution that has no relevance on the context, as the agent is in a very minimal information vacuum.
Slightly similar, but if you only told a contractor 2 paragraphs of a problem before sending them away for a month working on a problem with no feedback, you're going to end up with some interesting solutions. Agents are similar, but given their tuning, they end up producing a distinctive style, that often lead of superflous details, and especially today given their limited context, they accept their own created incidental complexity as a feature of the system, and even more catastrophic they can't differentiate that well between emergent and incidental complexity.
This often lead to the operator having to give more context to an agent as it is running. This often shows up like
1. Human: Create a web page to display a list of users
2. Agent: Here is a webpage that shows a list of users (including name, birthdate, usernames, email) with a details page
3. Human: I didn't ask for a details page and users don't have a username or email, just a name and an id.
4. Agent: compacting context
5. Agent: Here is a list of name and id, no details page
It looks to have come up with a workable solution, but it might have kept the birthdays in the database, they're just not filled out, because it thought it was an existing migration in the database etc. Incidental complexity right there, or it might not have thought of authentication etc.
The Agent might have come up with the right solution given additional context and refinement from either humans or another agent, but at what cost. There might now be 5 tables of user ids, a details page component that isn't hooked up anywhere, username email endpoints for users, and so much more. I don't like to talk about lines of code, but I've seen that AIs often lead to much more code, not just in lines of source code, but comments, functions, documentation etc. Often with a skewed view of what is important for someone that needs to pick up the solution, whether that is an LLM or a human.
Funilly enough, LLMs end up producing code, that is often more difficult to pick up again for another LLM in another session. Context is king for an LLM, so it filling the code with superflous details, will lead the LLM to be less effective by filling up the context with superflous token much faster.
## The cost
AI Agents are a superpower, they can type much faster than humans, and can come up with solutions to problem much faster than humans, not to mention that they open the software space to many other demographics of people, this might be a small shop wanting a custom tool, but not enough capital to a software engineer on staff, a software engineer being able to solve problems for smaller customers, etc.
But the solutions they end up producing have incidental complexity and hidden technical debt, sometimes outstripping the initial gain of solving the problem, by shackling the potential of the future of the solution.
## My view of the future
AI Agents are here to stay, they simply solve problems at a lower cost, than engineers. But the side-effect of their solution requires intentional effort, whether that is an engineer going in and solving the problem, or agents doing the same. I however, believe the without additional context, additional autonomous systems will only produce more entrophy and as such a problem might initially be solved, but will become far worse in the future if only autonomous systems are used.
Again similar to sending a contractor to a cabin in the winter for 2 months working away at a problem, and then only giving 1 paragraph of feedback to their solution. The hidden complexities and subleties of the their solution will be skewed, have potentially have tons of hidden incidental complexities that require intention to solve.
I believe that software engineers will have a role in development of software products for a long time, the role won't look the same, but nothing has shown me that autonomous systems have learned to produce order from entrophy, no matter the capabilities. Until Autonomous AI systems are able to work alongside humans one to one, or replace organisations completely then they will always have to interact with works of humans to fill their information vacuum.

157
content/posts/2026-01-25-incidental-complexity.md Normal file
View File

@@ -0,0 +1,157 @@
---
type: blog-post
title: "Incidental Complexity"
description: |
Complexity is a required aspect of developing products, in especially software, complexity is sometimes inherited, molded and incidental.
In this post we discuss incidental complexity in software development; history, present and speculation on the future.
draft: false
date: 2026-01-25
updates:
- time: 2026-01-25
description: first iteration
tags:
- "#blog"
- "#softwaredevelopment"
- "#ai"
- "#complexity"
- "#techdebt"
---
Complexity is a required side effect of developing products, whether in processes, human relations, or especially software.
Complexity is sometimes inherited. You have to fix an issue in a legacy system, make it fit for purpose, and either expand or refactor its capabilities and internal workings.
Or it is incidental, which we will cover in this post: whether you add more than you should to cover a need you think you have, only to discover that the result is over-engineered or simply redundant. That is not to say the code is unused, but it represents extra details that could have been simpler.
## Incidental complexity has always existed
Complexity is a feature, not a bug. One of the main benefits of software is that it is so malleable. It can be changed much more easily than hardware. It is far easier to change a line of code than to redesign a production line.
Software also has no real permanence outside of organizational or environmental factors. It must live up to requirements, both functional ones, such as fulfilling a purpose, and non-functional ones, such as handling certain constraints reliably. A trading algorithm, for example, has a specific shape because it must be fast enough to beat competitors.
Complexity is born when a decision, whether incidental or not, is added to a product. It might be a choice of programming language, coding style, infrastructure, and so on. None of these are inherently good or bad, but they exist on a scale of how easy they are to discover and change. This varies depending on context: firmware distributed once, over-the-air updates, data center services, or websites that change many times per day.
I define two subsets of complexity: emergent and incidental. The goal of most software is to be the minimal amount required to fulfill its requirements. This is not about lines of code. Rather, it is about how much context is embedded in the system for it to function. That is emergent complexity.
Incidental complexity arises when we apply too much context. These are details that are unnecessary or irrelevant to solving the problem.
### Example
A requirement might be to show a list of users on a webpage. The list comes from a database, and there are many ways to solve this.
- A simple solution is to fetch users one by one using the database client until the page is populated. If the service will never have more than ten users, this is fast enough, easy to change, and perfectly adequate.
- A complex solution might cache the total number of users, split queries into batches, fetch them in parallel, and stitch the results together.
Both solutions are valid. They produce the same output using different approaches.
If the database never has more than ten users, then much of the complex solution becomes unnecessary. Batch processing, parallelization, counting, and stitching are never used in practice.
All the logic required to handle thousands of users exists, but is never actually exercised. If the batch size is 500, then no parallel processing occurs, no splitting is needed, and no stitching is required, the code is still run, but was never required in the first place.
There is no right or wrong answer. The first solution was built knowing that ten users was the maximum, so it was designed to handle that within reason. It is also simple enough to change if this assumption ever changes. The second solution may have been built with the ambition of handling thousands of users and providing fast page loads.
Incidental complexity consists of additional details that have little or no effect on the actual output, whether functional or non-functional. You might argue that this is simply premature complexity. I define premature complexity as a step in the process that often leads to incidental complexity.
It is impossible to hit the mark perfectly. Complexity is subjective. For an experienced team with a wide toolbox, fetching one thousand items at once may be trivial and acceptable. For others, fetching items one by one may be simpler and safer.
There is no universally correct answer. What matters is that software is built to its requirements, with minimal additional details. A good solution should feel natural given the organizational context and problem domain.
Incidental complexity often appears in questions such as: Why does this use batching when there is only one item to fetch? Why does this use a strategy pattern when there is only one option?
Some of these complexities may become useful in the future, but at present they do not contribute meaningfully to solving the problem. You can likely think of many examples from your own career, whether created by you or inherited from others.
## Devil in the details
It can be very difficult to distinguish between emergent and incidental complexity. This is truly where the devil is in the details.
Continuing the previous example, it might be that every Easter, volunteers across Denmark are added to the database. Instead of 10 users, there are suddenly 1000. In that situation, the page might have unsatisfactory performance in loading the page without a sufficiently complex solution.
This gives complexity a form of permanence. It becomes difficult to remove because similar situations may have caused failures in the past. When performing rewrites or maintenance, teams often hesitate to remove such logic because it may have been added for a good reason.
Sometimes, you cannot know for certain. You have to take a risk. Doing so requires skill, experience, and intention. Risk is uncomfortable, especially in production systems.
In general, teams are more likely to preserve inherited complexity than complexity in newly written code. This is one reason why solving a subset of a problem from scratch is often easier than modifying an existing solution, but also often leaves out crucial details.
## Human vs. AI (LLM)
Humans and AI (Agents and LLMs) produce incidental complexity in similar ways, but with important differences.
A human applies personal knowledge and programming style to a problem and arrives at a solution. Humans are usually aware of the requirements and context, even if imperfectly. Incidental complexity arises from habits, preferences, and experience.
An AI has wide but fuzzy knowledge. It does not have taste, but it does have style, derived from aggregated training data and optimization for producing acceptable answers. Solving problems is part of that optimization, but so are clarity, verbosity, and perceived completeness.
AI systems often have limited understanding of the real context. Conveying full organizational and business context through text alone is extremely difficult. A large part of engineering work consists of deriving concrete requirements from incomplete information.
As a result, AI systems often produce solutions that appear to work but either omit crucial details or introduce functional and non-functional requirements that were never requested.
There is also the issue of ephemeral context. AI systems forget details over time, which can lead to severe degradation in performance. Even if this problem is solved, AI agents are still likely to produce different forms of incidental complexity than humans.
Humans and AI are tuned to different incentives. Humans are shaped by values, experience, and feedback, which leads to effective but relatively slow development. Large language models are trained on vast amounts of software of varying quality. They are optimized to generate outputs that are accepted, not necessarily correct.
This distinction is crucial. An acceptable answer is not the same as a correct one. This is comparable to a student lying about failing a class. The behavior exists because of incentives, and it is unlikely to disappear entirely.
AI systems are biased toward producing an answer even with limited context. They are also optimized to minimize follow-up questions. This often results in confident but incomplete solutions.
This tendency often appears in how AI systems structure their responses. They frequently add extensive comments explaining obvious behavior, regardless of complexity. When producing text, they default to a familiar pattern: an introduction, several bullet points, and a conclusion.
These tendencies can be influenced, but the underlying style remains consistent. The same applies to programming output. Given no examples, an agent will typically produce an average, generic solution.
At the current stage of development in 2026, AI systems often add details that are irrelevant to the actual context. Because they operate in an information vacuum, they compensate by over-specifying solutions.
A similar outcome would occur if you gave a contractor two paragraphs of instructions and then sent them away for a month without feedback. The resulting solution would likely contain many assumptions and unnecessary features.
AI agents behave in a comparable way. Because of their training, they develop a distinctive style that often introduces superfluous components. Even more problematically, they tend to treat their own incidental complexity as a features rather than as a liability.
They also struggle to distinguish between emergent and incidental complexity.
This often forces operators to continuously refine and correct the agent’s output by providing more context. In practice, this interaction often looks like this:
- Human: Create a web page to display a list of users.
- Agent: Here is a webpage that shows a list of users, including name, birthdate, username, and email, with a details page.
- Human: I did not ask for a details page, and users do not have usernames or emails. They only have a name and an ID.
- Agent: Compacts context.
- Agent: Here is a list with name and ID, and no details page.
The result appears correct, but hidden complexity may remain. The database schema may still contain unused fields. Migration logic may exist for data that will never be populated. Authentication hooks or unused endpoints may still be present.
That is incidental complexity.
The agent may eventually arrive at the correct solution through repeated refinement, either with human guidance or assistance from other agents. However, this raises the question of cost.
## The Cost
It is not uncommon for such workflows to leave behind multiple unused tables, unconnected components, dormant endpoints, and unnecessary abstractions.
I generally avoid focusing on lines of code, but in practice AI-generated systems often contain significantly more source code, comments, helper functions, and documentation than necessary. Much of this material reflects a distorted view of what is important for future maintainers, whether human or machine.
Ironically, this also makes the system harder for future AI agents to work with. Context is critical for large language models. Filling a codebase with irrelevant details consumes context window capacity and reduces the effectiveness of future interactions.
AI agents are a genuine productivity multiplier. They can produce text and code far faster than humans. They can explore solution spaces quickly and enable many new groups to build software.
This includes small organizations that need custom tools but cannot afford dedicated engineers, as well as engineers who can now serve smaller customers efficiently.
However, the solutions produced by AI agents frequently contain hidden technical debt in the form of incidental complexity. In many cases, this debt outweighs the initial productivity gains by constraining future development.
Systems become harder to understand, harder to modify, and harder to extend. Over time, the accumulated friction erodes the benefits that automation originally provided.
Often conflated, incidental becomes because of AI systems accidental complexity. It is something that happens because of a mistake, or negligence, in the case of agents unintentional.
## My view of the future
AI agents are not going away. They solve problems at lower marginal cost than human engineers, and this advantage is decisive.
However, their output requires intentional correction and refinement. This can be done by humans, by other agents, or by hybrid workflows. Without such intervention, complexity will continue to accumulate.
I believe that, without additional context and governance, autonomous systems will tend to produce more entropy. Problems may be solved in the short term, but systems will degrade in quality over time.
This is similar to sending a contractor to a remote cabin for two months with minimal guidance and then offering only a paragraph of feedback at the end. The solution may function, but its internal structure will reflect distorted priorities and hidden assumptions. Sending a village doesn't work either. You might end up with amazing code quality, well-documented code. But if it is over-engineered, containing superfluous details, then it doesn't matter.
Such systems often contain layers of incidental complexity that require deliberate effort to untangle.
For this reason, I believe software engineers will continue to play a central role in product development for the foreseeable future. That role will evolve, but it will not disappear.
Nothing so far suggests that autonomous systems can reliably produce sustained order from complexity. Until they can operate at the level of entire organizations, with deep contextual awareness and accountability, they will remain dependent on human input.
As long as AI systems must interact with human-created systems to fill their information gaps, engineers will remain essential to maintaining coherence, intent, and long-term quality. What is a side-effect however, is that we will have much more code going forward, with many bespoke components, as the cost of producing functionality goes to zero, so will the need for centralized services, and in turn bespoke components will continue to climb if left unchallenged.
In the coming years and decades, I expect that software products and interactions will accumulate so much complexity that they will become indistinguishable from biological systems, even more so than they already are today. We will need agents to untangle the mess, but also surgical knowledge for when to refine a component to something known.