Antipatterns of Software Engineering with AI

I am a big believer in AI coding agents, but I have seen people fall into some terrible traps: comprehension debt, building too much in one go (yes, really).

I have been building software with AI since I got my hands on the first private previews of Github Copilot. I’ve switched tools back and forth between whatever has been the cutting edge for the last several years, and I’ve never shirked paying for more than one “Pro Max”/top tier subscription from the frontier labs.

I have also gone down some rabbit holes of over-enthusiasm in doing so, learned the lessons and gotten the scars of things going wrong in trusting AI too much. Although, the definition of how much and how far you can trust AI is a shifting line on a monthly basis, so any allegation of “AI did X wrong, you can’t trust it to do X!” is more likely to be a point in time snapshot, than a permanent fact.

However, I think there are some antipatterns cropping up, which are durable, permanent, not because AI is weak, but because they occur at the boundary where software meets reality. There is much AI can do, but it cannot predict the feedback of the real world.

“Spec-driven development”: return of the big bang

On LinkedIn, Spec-driven development has been all the rage, wannabe thought leaders have flocked to proclaim how a really detailed specification handed to the AI agent is now the way to produce software. To an extent, they are correct, but the way most go about it, is fundamentally wrong. I know, because I too have done it wrong.

The main issue is how teams are now drifting towards larger and larger scopes of work, just because it is possible to produce more scope in one go. This means larger product specifications, rather than focused, single feature/functionality specifications.

This has a couple of issues:

• Large scope produced in a vacuum of no feedback.

• Large specification introduces room for ambiguity, even if double- and triple-checked and acknowledged by your AI assistant.

The first issue, large scope produced with no feedback is the underlying root cause: if you do not test what you have built with real users, or in the real domain for which it is intended, no amount of specification detail is going to answer whether you are building the right thing in the first place! The fact that we can build quicker, doesn’t solve the bottleneck of real-world feedback. And if you keep building past sensible points of soliciting that feedback, you might be piling more and more code onto a foundation that is all wrong. All you are achieving is a big pile of technical debt, faster. Fundamentally, all code is debt, the only question is whether that debt is invested productively, and if you don’t know, you don’t know, the likelihood is that you just have a large debt, based on conjecture and guesswork.

The second issue is large specification introduce room for more ambiguity. If you want to write an entirely unambiguous specification, it would have to be written in.. code. English, or whatever your native language you speak is just not as unambiguous as code can be. And even if you ask an LLM if something is unambiguous, and it says “yes”, chances are, just like a human would, it has simply made implicit assumptions it thought sensible in the areas that hid real, but not obvious ambiguity. You let an LLM, or a human, loose on a large specification, with lots of tiny ambiguities, those tiny ambiguities compound to a real and large drift from the intended purpose.

Multiply scope with no feedback and numerous micro-ambiguities filled in with implicit assumptions, and the drift from your intended purpose can grow meaningfully large.

Spec-driven development isn’t all that. By all means, write a spec, document the intended outcome and hypothesis tested, it will be useful both for development and posterity, but keep the scope as small as possible, and be clear about how you prove or falsify that it was the right thing to do with feedback.

Architectural over-engineering

AI is trained on a corpus of all the code it can get its hands on. This means it will converge towards the average of that corpus. The average of our industry has some fundamental issues: over-engineering, complex, yet leaky abstractions, anemic data models, layering of design pattern upon design pattern because cargo-culting, and much much more.

If you let an AI define your entire architecture, chances are you’ll sit upon a brittle pile of 15 layers and 100 abstractions that make no sense and which fit badly together. It means adding new- or changing existing functionality becomes risky and/or difficult. You will have built your MVP quickly, but as months and years pass, change becomes increasingly difficult, outages and accidental bugs grow.

This issue is not one I expect more sophisticated models to be able to address: the issue is in the quality of the training data. Even labeling “good” and “bad” requires the right people would be labeling the data (how do you find the above average people willing to do it?). Unlike mathematics and other hard sciences, or even human language grammar, good software architecture is hard to objectively prove without the benefit of years of hindsight of what has been produced. The feedback cycle of architecture runs in the years, and to some extent the quality is also dependent on how it interacts with the people driving and using it. But even in this case, leaving it rudderless, with no human direction is bound to cause issues down the line.

AI grading its own homework

AI writing the code, and writing the tests unsupervised is becoming the norm. But just like the issue with specification, is it writing the right tests? Are the tests reflective of what needs to be achieved? Are they comprehensive, do they test failure scenarios? Or are they simply written to match the code?

You wouldn’t let a school-child grade their own homework, but we are ok with AI doing this. Applying disciplines like property-based testing and mutation-testing may go some way of improving the quality, but fundamentally it is still AI grading its own homework.

Another approach might be using skills, different context/processes and/or models to grade the work of another AI agent, in adversarial grading. But it still doesn’t solve the top-level issue of: has the right thing been produced, and are the tests reflective of what the code should do, rather than what it does?

How do we solve this issue? I am not sure at this time, maybe we will find automated solutions eventually. But for now, I am relying on a couple of dials: mutation testing is a useful tool, but a human in the loop at least reviewing the tests and architecture might be the sensible solution until we figure it out. The level to which we need to do this will obviously depend on how critical the code is. Throwaway prototypes in non-critical domains? Fine, trust the output. Real money or health on the line? Perhaps spend the human effort to review that the tests reflect the desired outcome of the change.

After all, tests are effectively the specification written in code, so you should care if it is important.

Comprehension debt

The problem that creeps up on you if you have had a break from actively working on a specific codebase, or a corner of it for some time, and it was generated: you forget a lot of the context and decisions made along the way. Of course, you could try to document those along the way, but this is still inferior to the mental recollection you build up by doing something yourself. Both in coding, and in note-taking, there is a tactile memory footprint created by the physical act of writing something yourself. In a world of generated code, we are slowly losing this, and as a consequence, we are creating comprehension debt for ourselves.

This is compounded if we take our hands of the wheel with appropriate sized increments, directing the architecture, and knowing what the output does (checking the tests).

The solution is probably not to go back to hand-writing every line of code, even if it would be beneficial to reducing comprehension debt. But it might be to be more disciplined around how we deal with the prior three antipatterns discussed in this post, as they will individually compound the debt.

What falls out of comprehension debt and poor generated architecture, is also increased incidental coupling, which in turn makes it difficult to fully estimate the full blast radius of changes. At the moment, even models like Fable5 (trust me, I had an issue recently), where on larger code-bases, obvious breakages will be missed, even if prompted to thoroughly investigate the blast radius. It might change in the future, but for the time being, the combination of human comprehension of a full system combined with AI is the best solution to avoid this.

Finally, someone might argue comprehension debt is not a problem if AI can understand it for you. It is partially true, but it falls apart at the point when there is not only a history of decisions made, but also underlying implicit reasons why those decisions were made and what other things were discarded in the process. Without human comprehension of the context and history of a system, and how it works, it is easy to retread past mistakes every so often, wasting human time and effort, and machine tokens.

Conclusion

The common thread here is, while AI accelerates the production of software, the failure modes of the produced software are still the same as in pre-AI software:

• Knowing what to build in the first place.

• Aligning what you built with reality, based on feedback (most likely the “Knowing what to build” was not entirely correct, but hopefully in the right ballpark).

• Evaluating what was built was what we intended to build.

• Steering AI away from building mediocre architectures based on the average.

Someone will likely argue that one day AI will be able to decide all of this for us. I doubt it will go much further than radical acceleration (which in itself is really valuable, but incomplete). Why? Just by the nature of how contemporary AI algorithms work and are trained: they will revert to mean, approach the average. For objective facts, this is good, we are converging on truth. But for subjective matters, such as human intent and value, it is not so easy: provide ten AI models with instructions to come up with a business plan in a domain, and they will likely converge on ten similar ideas with little differentiation.

Along the boundaries of where reality meets code, we still need human judgement. Without it, we are just accelerating creating larger and larger technical debt and sunk costs.

The new discipline of Software Engineering is in how to make this process rigorous, data-driven and repeatable.