Why This Piece Exists
In Piece 1, I argued that AI code generation didn't shift the bottleneck from generation to specification; the bottleneck was always specification, and AI collapsed the pretense that generation was where the difficulty lived. If you buy that argument, the natural follow-up is: okay, what specifically is hard about the world we're now exposed to? What does the difficulty actually look like when you stop pretending the hard part is typing?
I think it resolves into three distinct structural constraints, plus an amplifier that makes all of them worse. This piece maps those constraints, traces their origins, and shows how they cascade into the downstream problems the rest of this series will address. It's the structural backbone of the argument, the root cause map that everything else hangs from.
Constraint 1: The Specification Bottleneck
Fred Brooks drew the line in 1986: the essential difficulty of software is deciding what to build, the accidental difficulty is expressing that decision as code. I covered this in Piece 1 and I won't re-litigate it, but I want to push deeper into why specification is so hard, because I think the industry has a shallow read on this.
The shallow read goes something like: "we need better prompts." The slightly less shallow read is that prompt engineering is a real skill that deserves investment. Both of these are true as far as they go, but they miss the actual structure of the problem.
What we call "prompt engineering" is specification engineering in disguise. When a developer writes a prompt for an AI code assistant, they are specifying intent: what the code should do, how it should behave, what constraints it should respect. The quality of the output is bounded by the quality of that specification. This is not a prompting problem, it is the same specification problem Brooks identified forty years ago, wearing a new interface.
And the reason it feels harder now is that the feedback loop has changed. When a developer wrote code by hand, the act of writing forced a confrontation with ambiguity. You can't type if (user.isAuthenticated) without deciding what authentication means in your system, how it's checked, what happens when it fails. The specification was embedded in the generation, clause by clause, decision by decision. It was slow, but it was also a forcing function for precision.
AI generation removes that forcing function. A developer can now write "add authentication to this endpoint" and receive a complete implementation that makes dozens of decisions they never explicitly considered. Connection pooling strategy, retry behavior, error granularity, token expiration policy, all resolved by the model's statistical defaults rather than the developer's intentional design. The specification gap, the distance between what the developer meant and what the code actually does, used to be constrained by the effort of typing. Now it's unconstrained, and most of the time, nobody notices until something breaks.
Daniel Jackson's work on conceptual design captures this precisely. In The Essence of Software, Jackson argues that the real problems in software are conceptual, not implementational; that the difficulty lives in the design of concepts and their relationships, not in the code that realizes them. The specification bottleneck isn't about writing better prompts, it's about thinking more clearly about what you're building: what concepts exist in the system, how they relate, where the boundaries are, what invariants must hold. This is design work, and we have almost no infrastructure for it.
Leslie Lamport has been making a version of this argument for decades, insisting that programmers should specify before they code, that the act of formal specification reveals errors that coding never will. The industry mostly smiled and nodded and kept shipping code. Because when generation was slow and expensive, the cost of not specifying formally was partially masked by the forcing function of manual implementation. You couldn't write code fast enough to outrun your understanding, at least not by much.
Now you can. And the specification gap compounds with every model improvement, because faster generation means more unexamined decisions per hour.
Here's what makes this structural rather than transitional: we have almost no training, tooling, or process infrastructure for specification as a first-class activity. We don't teach specification in computer science programs, at least not as a practical skill for working engineers. Our tools are optimized for code editing, not intent editing. Our processes gate on code review, not specification review. Our metrics track code output, not specification quality. The entire ecosystem assumes that the hard part is downstream of the decisions, not the decisions themselves.
I think "prompt engineering" will eventually be recognized for what it actually is: the beginning of a specification discipline that the industry should have built forty years ago. But right now, we're trying to practice that discipline with no tools, no training, and no organizational support, while the cost of poor specification compounds faster than ever.
Constraint 2: The Local-to-Global Coherence Gap
The second constraint is the quiet one. Specification failures announce themselves when features break. Coherence failures hide in the spaces between components, invisible until the system is under real load.
AI code generation is fundamentally local. It operates on what's in the context window: the current file, the prompt, whatever retrieval mechanism pulled in. Within that window, it's remarkably capable. It can generate clean, well-structured code that handles edge cases, follows patterns, and passes tests. The quality of local generation is genuinely impressive and continuing to improve.
But software systems are not defined by their local properties. They're defined by how their pieces interact: how the authentication service talks to the session manager, how the retry logic in service A interacts with the timeout configuration in service B, how the database connection pool behaves under load when three services are competing for connections. Architecture, security, performance, and correctness are emergent properties of whole systems, and they cannot be evaluated by looking at any single component in isolation.
This is not a context window size problem. The "Lost in the Middle" paper by Liu et al. demonstrated in 2023 that language model performance degrades significantly for information placed in the middle of long contexts, even when that information is technically within the window. Attention doesn't scale linearly with context length; it degrades. Bigger windows help at the margins, but they don't solve the fundamental issue, which is that coherence across a system requires a kind of reasoning that transformers are not architecturally designed to perform.
RAG, retrieval-augmented generation, is the industry's current response to this problem, and it's worth being precise about what it does and doesn't solve. RAG shifts the problem from "what fits in the context" to "what gets retrieved into the context." This is a real improvement for some use cases, but it's a different failure mode, not a solution to the coherence problem. The retrieval system has to know what's relevant before the generation happens, which means it needs exactly the kind of whole-system understanding that we're trying to use it to provide. It's a dependency loop.
Grady Booch has observed that architecture involves multi-objective trade-off reasoning: balancing performance against maintainability, security against usability, consistency against autonomy. These trade-offs are not local decisions, they're system-level choices that ripple through every component. LLMs can simulate this reasoning, they can generate text that reads like architectural analysis, but simulating trade-off reasoning and actually performing it are different things. The simulation breaks down at exactly the point where it matters most: when the trade-offs interact in ways that aren't represented in the training data.
Chelsea Troy has made a related point about runtime behavior: AI has no model of how code actually executes. It can generate code that looks correct, that follows patterns, that even passes unit tests, but it has no understanding of what happens when that code runs in a real environment under real load with real concurrency. Runtime behavior is an emergent property, and emergence is exactly what local reasoning can't capture.
This creates what I'd call the emergent architecture anti-pattern. When AI generates code per-prompt, with each prompt producing locally correct output, the architecture of the resulting system is never designed. It emerges from the accumulated history of prompts and their statistical resolutions. Nobody decided that the system should use three different connection pooling strategies, or that retry logic should be implemented inconsistently across services, or that the authentication model should have subtle semantic differences between the API layer and the queue processor. These things just happened, one reasonable-looking prompt response at a time.
Kent Beck's concept of "tidying," the practice of making continuous small structural improvements to maintain system coherence, points toward part of the solution. But tidying assumes someone understands the system well enough to recognize when coherence is degrading. As AI generates more code faster, the rate of coherence degradation outpaces the human capacity to detect and correct it. The tidying can't keep up, not because the practice is wrong, but because the generation has been decoupled from the understanding.
I want to be clear about what I think will and won't improve here. Context windows will get larger, retrieval will get better, and AI will get better at reasoning about multi-file interactions. Some of what I'm describing as a structural constraint may turn out to be a transitional gap. But the core issue, that software systems have emergent properties that cannot be fully captured by local analysis, is not a technology limitation that scales away. It's a property of the systems themselves. Even with perfect context retrieval and unlimited context windows, someone still has to design the system-level properties. That's human judgment work, and delegating it to statistical pattern matching doesn't make it go away, it makes it invisible.
Constraint 3: The Trust Calibration Impossibility
The third constraint isn't technical, it's about whether we can even tell when something is wrong, and I think it's the most unsettling of the three because there's no engineering trick that fixes it.
The problem is straightforward to state: there is no reliable signal for when AI-generated code is wrong. The failure modes are unpredictable and inconsistent. The same model, given the same prompt, might produce correct code nine times and subtly incorrect code the tenth, with no external indicator of which is which. This means that engineers must evaluate every piece of generated output, which is exactly the cognitive work that automation was supposed to reduce.
This isn't a new problem in human-automation interaction. Parasuraman and Manzey published a comprehensive review in 2010 documenting automation complacency across domains: aviation, medicine, process control. The consistent finding is that as automation becomes more reliable, human operators become less vigilant in monitoring its output. This is not a character flaw or a training gap, it is a documented cognitive phenomenon. Humans are not equipped to maintain sustained vigilance over a system that is almost always right.
Lisanne Bainbridge identified the deeper irony in 1983, in a paper called "Ironies of Automation" that reads as if it were written about AI code generation. Her central argument: the more reliable automation becomes, the harder it is for human operators to detect and correct its failures. The skills required to monitor automation, the ability to maintain a mental model of what the system is doing, to recognize anomalies, to intervene effectively, are exactly the skills that automation erodes through disuse. The better the automation works, the worse the human gets at catching when it doesn't.
Microsoft Research found something consistent with this pattern: developers using Copilot accepted suggestions faster over time, regardless of whether suggestion quality had changed. The acceptance rate increased not because the suggestions got better, but because the cognitive cost of evaluation felt increasingly disproportionate to the perceived risk. This is rational behavior in the moment and catastrophic behavior in aggregate.
The feedback loop problem makes calibration structurally impossible in the way we normally think about it. Trust calibration requires timely, attributable feedback: you need to know when a decision was wrong and which decision it was. But code correctness feedback is slow, often arriving weeks or months later when a bug surfaces in production, and noisy, because it's genuinely hard to attribute a production issue to a specific AI-generated suggestion versus a human design choice versus an environmental factor. Without clear feedback, there's no mechanism for calibration. The signal doesn't exist.
And this is where the "almost right" problem becomes critical. AI-generated code is rarely obviously wrong. It doesn't usually produce syntax errors or blatant logic bugs, those are easy to catch. What it produces is code that is subtly wrong: correct in most cases but incorrect at the boundaries, functionally adequate under normal load but fragile under stress, locally sound but globally incoherent. Subtle wrongness is categorically harder to detect than obvious wrongness, because it requires exactly the deep system understanding that the generation process bypassed.
I want to resist the temptation to frame this as a temporary technology problem that better models will solve. I think that framing is wrong, and precisely wrong in a way that matters. This is not primarily a problem of model accuracy, it's an epistemological problem about the fundamental impossibility of calibrating trust in a system whose failure modes are unpredictable. Even a model that is 99.9% accurate presents a trust calibration problem if you can't tell which 0.1% is wrong. And the better the model gets, the harder the remaining failures are to detect, because the easy failures get resolved first and what's left are the subtle, context-dependent, boundary-condition failures that require deep understanding to catch.
The structural nature of this constraint becomes clear when you think about what would actually solve it. Not a better model, but a verifiable model: one whose outputs come with formal guarantees about their correctness. That's a fundamentally different thing from a more capable model, and it's not on any current research trajectory's near-term roadmap. Until we have that, we're operating in a regime where every efficiency gain from code generation has to be weighed against an unquantifiable risk of undetected errors, and where the humans responsible for managing that risk are being progressively deskilled by the very automation they're supposed to supervise.
The Amplifier: Jevons Paradox
There's a fourth factor that isn't a root cause in itself, but amplifies all three. It's worth naming briefly here, because the full treatment comes later in the series (Piece 5).
In 1865, William Stanley Jevons observed that making coal consumption more efficient didn't reduce total coal usage, it increased it. The same dynamic applies to code generation. As producing code becomes cheaper, we don't produce the same amount of code more efficiently; we produce dramatically more code. GitClear's 2024 data already shows this: code churn, the rate of recently-written code being rewritten or deleted, is increasing in the AI era. We're generating more, keeping less, and the net effect is a larger codebase that's harder to understand, with more surface area for all three constraints to operate on.
More code means more specification decisions being made implicitly rather than explicitly (amplifying Constraint 1). More code means more system interactions that nobody designed (amplifying Constraint 2). More code means more output to evaluate with the same limited human attention (amplifying Constraint 3). Jevons Paradox doesn't create the problems, but it determines the rate at which they compound. And right now, that rate is accelerating.
The Causal Chains
These three constraints don't operate in isolation. They cascade into the specific downstream problems that the rest of this series will address. I think it's worth being explicit about the causal structure, because understanding which root cause drives which problem changes what interventions make sense.
From the Specification Bottleneck:
Comprehension Debt. When code is generated from underspecified intent, the resulting system encodes decisions that nobody explicitly made. Those implicit decisions accumulate as comprehension debt: code that works but that nobody fully understands. This is different from traditional technical debt, which is usually about known shortcuts. Comprehension debt is about unknown design choices, and it compounds every time someone generates more code on top of decisions they never examined.
Tautological Testing. When the same AI generates both the code and its tests from the same underspecified prompt, the tests tend to verify that the code does what the code does, rather than what the system should do. The oracle problem, knowing what correct behavior looks like independently of the implementation, is a specification problem. Without a clear specification of intent, tests become circular.
SDLC Mismatch. Our software development lifecycle was designed around generation as the time-consuming, gating activity. When generation is instant, the lifecycle's assumptions break: code review becomes a bottleneck rather than a quality gate, sprint planning becomes meaningless when features can be generated faster than they can be specified, and the entire cadence of plan-build-test-ship stops mapping to how work actually flows.
From the Coherence Gap:
Comprehension Debt (at the integration level). The coherence gap produces a specific kind of comprehension debt: not just code nobody understands, but interactions nobody designed. A single service might be perfectly comprehensible; the system formed by twenty services, each generated independently, is comprehensible to no one, because its architecture was never intentionally created.
Solution Monoculture. AI models generate from training distributions, which means they converge on popular patterns, frameworks, and approaches. This creates a homogenization effect where different problems get the same solutions, different teams produce architecturally similar systems, and the diversity of approaches that normally drives innovation narrows. When local generation dominates and system-level design atrophies, every codebase starts looking the same.
From the Trust Calibration Impossibility:
Tautological Testing (again, from a different angle). Trust calibration requires independent verification, but if you can't reliably evaluate AI-generated code, you also can't reliably evaluate AI-generated tests. The trust problem in code and the trust problem in testing are the same problem, which means testing can't serve as the independent check that trust calibration requires.
Measurement Void. If you can't calibrate trust in AI output, you also can't measure the quality of AI-assisted development. Are we shipping better software or just more software? Traditional metrics, velocity, code coverage, defect rates, don't answer this question, because they were designed to measure a different kind of development. The trust calibration impossibility creates a measurement void: we genuinely don't know how we're doing.
Expertise Pipeline Collapse. Junior engineers develop expertise by writing code, struggling with problems, building mental models through direct confrontation with complexity. When AI generates the code and the junior engineer reviews it, the learning pathway is inverted: they're asked to evaluate before they can create, to exercise judgment before they've built the foundation that judgment requires. And the trust calibration impossibility means they have no reliable signal for whether their evaluations are correct. The expertise pipeline doesn't break because AI replaces juniors; it breaks because AI removes the mechanism by which juniors become seniors.
And Jevons Paradox amplifies all of it. More code, more unspecified decisions, more undesigned interactions, more output to evaluate, less time per evaluation. The amplifier doesn't pick favorites; it scales everything.
What This Map Means
If this root cause analysis is right, then several things follow.
First, these are different problems requiring different interventions. The specification bottleneck is a tooling and process problem. The coherence gap is an architectural reasoning problem. The trust impossibility is an epistemological and organizational design problem. A single solution, whether it's "better models" or "better prompts" or "more code review," can only address one root cause at best. The industry needs a portfolio of interventions, not a silver bullet, and that should sound familiar if you've read Brooks.
Second, two of the three constraints are structural, not transitional. Better models will partially address the coherence gap and will not meaningfully address the trust calibration impossibility. The specification bottleneck is structural in a different way: not because AI can't improve at understanding intent, but because the organizational infrastructure for specification doesn't exist and won't build itself. These problems require deliberate human intervention to solve, and waiting for better AI is not a strategy.
Third, and this is the part that I think matters most: naming these root causes is the prerequisite to building better infrastructure. As long as the industry treats AI-assisted development as a monolithic phenomenon, something that is either "working" or "not working," we can't make targeted improvements. But once you can point to a specific failure and say "that's a specification problem, not a generation problem" or "that's a coherence problem, not a code quality problem," you can start building the right tools, the right processes, the right organizational structures.
That's what the rest of this series does. Each downstream problem gets its own piece, with a diagnosis that traces back to this root cause map and an intervention that addresses the specific root cause driving it. It's not enough to know that things are breaking; you need to know why, specifically, so you can fix the right thing.
The constraints are real. The map is legible. What comes next is building infrastructure for the world as it actually is, rather than the world we organized for.