I've been building software for 25 years and I've spent the last decade of that watching agriculture slowly and sometimes reluctantly, "go digital". So I've had a front row seat to the explosion of software that large language models have set off, and lately I've been trying to work out what kind of explosion it actually is. After thinking about it I think, for now, that it is mostly an explosion in quantity rather than moving things on. The same approaches, written faster, by more people, for more niche problems. Not necessarily better approaches, faster execution, or genuinely novel designs.

The reason I think is that models are trained to predict what's common in their training data. That makes them very good at reproducing the median solution to a problem and structurally weak at anything off distribution. A novel algorithm is rare in the training data by definition, so "give me an approach nobody has tried" is exactly the kind of request a language model struggles with. Which makes you ask the question, how is this intelligent?

The incentives don't help either. Shipping something quickly is visible and gives a dopamine hit, as proven through all these "look at what I built" posts. Redesigning for efficiency or novelty is slow and the payoff uncertain, sometimes years away. It's also much easier to check that code passes the obvious tests than to check it's the best possible design, so both the training signal behind these models and the habits of the people using them push toward same, same but fast, rather than better.

That said, agentic coding tools are making a long tail of software possible that just did not exist before. Personal tools, one off scripts, automations that were never going to be economical at professional engineering rates. But I think it's worth being realistic about what this actually achieves. It's the same (or worse) quality of software as before, the same (or worse) architecture, the same (or worse) assumptions, just now available to a lot more people. It lowers the barriers for who can code. It doesn't raise the code itself.

When a coding model is asked to write something it converges on the statistically common pattern for that problem, which is mediocre by construction, since "common" includes every ordinary solution along with the good ones. Quality only improves when there's a real, non gameable check in the loop forcing selection toward something better than typical. Google DeepMind's FunSearch and AlphaEvolve are examples of this. They pair a language model with an evolutionary search and evaluation loop, and they've found new algorithms this way, faster matrix multiplication and better bin packing heuristics among them, that weren't sitting in any training set. The novelty comes from the search and evaluation loop, not from the model producing a brilliant answer on the first try. The model proposes and then the loop selects. This isn't the way Claude Code or Codex currently approach coding.

I think this thinking is important right now because as chip prices climb, and will only get worse because of shortages, there's an opportunity to use this same kind of loop to squeeze more performance out of hardware we already own, given software is usually the most inefficient part of the stack. Early work using models to search for faster kernels and better compiler output, measured against real hardware counters instead of human review, is looking like an opportunity. In a chip constrained environment, it certainly feels like a more compelling opportunity than handing over your life savings for the next Apple M7 chip.

But as with everything, it's complicated. Mitchell Hashimoto, the Ghostty and HashiCorp founder, ran an experiment recently where he deliberately wrote a naive, slow renderer in Go, 88 milliseconds a frame with 150,000 allocations, then set an AI agent loose on it for four hours and roughly $350 in API costs, with the instruction: make it faster without touching the public API or the tests. The agent got frame time down to 1.5 milliseconds. Genuinely impressive, by any normal measure, and a real, hard, non gameable metric was doing the selecting the whole time. Then Mitchell published his own hand written port of the same renderer, which ran in about 20 microseconds with zero allocations on the hot path. Seventy five times faster than the agent's result. The agent had optimised within the bad design it was handed, caching and batching, and never considered ripping out the architecture for something like an arena allocator with dirty rectangle tracking, because that requires understanding what the system is actually for, not just what the profiler is complaining about this iteration. It's the local optimum trap, and it is a good illustration of the limit of search and verification on its own. A hard metric will reliably pull a design toward better than typical. It will not, by itself, pull a design toward a different architecture that was never on the table to begin with. That still takes someone who understands the system well enough to throw the current one away. It takes intelligence!

None of this happens either by just asking a model to think outside the box. That just samples differently. What produces something better than average is building infrastructure around the model that measures things humans don't normally optimise for explicitly rather than "compiles and passes the obvious tests", that searches at scale across many structurally different candidates instead of one plausible attempt reviewed by a tired human on a deadline, and that verifies against hard, real world metrics rather than human comfort with the result, so a correct but weird design doesn't get rejected simply for looking strange. The bottleneck isn't model capability any more. It's whether we bother building the evaluation infrastructure that lets unfamiliar but better outcompete familiar but safe.

Here's a thought. Hallucination and a genuine creative leap come from the same place, mechanically. Both are the model sampling a low probability region of its distribution instead of the safe median. FunSearch's mutation step is doing something similar when it generates a batch of candidate programs at higher temperature, most of it garbage, every now and then a few are novel and better. The difference isn't the generation step, it's what happens next. FunSearch can run each candidate, time it, check it's still correct, and throw away the garbage in milliseconds while keeping the gem. A hallucinated fact has no equivalent. Nothing to execute, nothing to check against cheaply, so the same off distribution sampling that occasionally finds something real just sits there wrong, indistinguishable from the correct low probability claim that happened to also be true. Which suggests hallucination is where the glitch in the matrix could come from, in any domain that had a verifier for it. In domains that already have one, code, algorithms, anything checkable by execution, that glitch is already being harvested, we just don't call it hallucination any more, we call it search. Training regimes that punish hallucination hard are probably also flattening the exploratory tail a verifier equipped system would want to keep. The fix isn't allowing more hallucination. It's building more domains a verifier can actually check, so the same underlying tendency gets to be search instead of error.

If a model's output is, in expectation, the statistical centre of its training distribution, that raises a question about how much "thinking" is happening versus sophisticated recombination of what's already been written. Search and verification don't really resolve this, they just bolt an external selection process onto an interpolator. The selection is doing the work of finding something better. The model is still only proposing. But I suppose this isn't just what machines do. Most human written software is also the median of what that particular engineer has already seen. Genuine breakthroughs are rare among people too. So the fact that a model mostly reproduces familiar patterns doesn't really separate it from human cognition, which does much the same thing most of the time. But it also doesn't mean it's anywhere near AGI.

Whatever produces something genuinely new, in a person or a model, I don't think "large scale statistical regularity in past output" explains it on its own. I think the interesting question was never whether the model is thinking. It's what, if anything, distinguishes the rare cases where something genuinely new appears from the much more common cases where it doesn't. Search and verification are maybe one answer, and as far as I can tell they apply equally whether the thing doing the proposing is a person or a machine and at the end of the day, I still haven't seen a new approach to software that makes my 5 year old M1 Mac Studio run any faster.