Do LLMs know how to make software? (No.)

An LLM knows nothing and writes nothing (this isn't an epistemological opinion, that's just the fact of what the algorithms do that LLMs are comprised of).

It's a deterministic statistical model trained on likelihoods of combinations of tokens (for text, that's words and parts of words) occurring together and then generating (based on input tokens in the prompt, previously generated tokens and added randomness) a string of tokens that looks like continuation of the prompt.

There is no knowledge or thinking or awareness of the actual topics discussed in the training data, the prompt or the generated output.

LLMs don't make or write, they generate.

So, how come LLMs are used successfully for programming?

While software is written in languages that arguably can be read and written by humans too, these programming languages aren't constructed like human language and communication is—they concern themselves essentially only with one thing: deterministic mathematical algorithms.

LLMs are never used directly for programming (despite being deterministic algorithms themselves). You categorically cannot use statistical text generation to make complex, working applications (or even parts of it). But, you can often generate text that is statistically close to working.

They can do this because of a few properties of LLMs and of almost any software itself; among those the triviality and self-similarity of most applications (showing data in lists and allowing "create, read, update and delete" operations) and the enormous amount of training data from source code repositories that comes not only with working code but also often literally decades of detailed editing history and documented ways how to fix all sorts of bugs.

Now even though it seems like it's set up as the perfect use case for LLMs, it still can't make working code out of the box (and will never be able to do so).

This is where harnesses come in, a type of application often referred to as "AI agent" (harness is in my opinion a better description of it).

The crux is this: because computer code is written in mathematically provably non-ambiguous language* with a vocabulary of a few dozen words and ultimately expresses deterministic mathematical algorithms it can be verified for accuracy and veracity.

Notably, to verify code, tools other than LLMs are used.

This includes syntax checkers, code coverage tools, benchmarking tools, automated tests, code introspection and so on—all these tools have been around since long before LLMs and are used by humans too when programming.

Harnesses usually comes in form of a chatbot interface, can call programs on behalf of the LLM (such as "read this source file", "compile this code", "execute this app and add the output to the prompt", "search for a file containing a string", "search the web for the documentation of this method", etc.) and force-feed it to the LLM. It relentlessly applies iterative verification loops, adding output from these tools and output from the running code to the prompt for the LLM.

It's literally brute-force driven development; orchestrated by the harness.

In other words, LLMs work to make code because of iterations over generating the code over and over again until it functions as intended (or the harness gives up). These iterations happen both internally while the LLM generates and externally together with the user adding to the growing prompt (often when the iterative loop gets stuck on a problem the statistical generation can't fix).

A harness is essentially saving the LLM from itself (happily generating nonsense) by making it iterate and verify until finally a (more or less) working product emerges.

Programmer brain

Now if you ever seriously tried LLMs for coding you should conclude that they are inherently useless for generating text that isn’t a verifiable algorithm.

But many programmers assume the opposite: because programming is "super hard" it must mean that LLMs must be very capable of generating any sort of text.

But this is simply not the case with any other type of text generation. It’s laughable to use it for emails, legal opinions, doctor visit transcriptions, scientific papers and so on because there's no way to verify the generated text; both for the veracity of what it says and for syntax and semantics.

"Programming systems can, of course, be built without plan and without knowledge, let alone understanding, of the deep structural issues involved, just as houses, cities, systems of dams, and national economic policies can be similarly hacked together. As a system so constructed begins to get large, however, it also becomes increasingly unstable. When one of its subfunctions fails in an unanticipated way, it may be patched until the manifest trouble disappears. But since there is no general theory of the whole system, the system itself can be only a more or less chaotic aggregate of subsystems whose influence on one another's behavior is discoverable only piecemeal and by experiment. The hacker spends part of his time at the console piling new subsystems onto the structure he has already built—he calls them 'new features'—and the rest of his time in attempts to account for the way in which substructures already in place misbehave. That is what he and the computer converse about."
—Joseph Weizenbaum, Computer Power and Human Reason, 1976

And even for programming there's a limit to what can be tested for veracity. For there is no way to automatically test if the larger system constructed by the programmer with the help of LLMs is a good solution to the whatever the problem at hand is.

This is of course also a problem for when no LLMs are used for making software—however, "agentic engineering" as it's also called delegates many decisions a programmer or software designer normally makes to the LLM (the "agent"); which literally calculates these decisions with applied statistics based on strings of words from elsewhere without even a trace to the source.

[S]cience promised [humans] power. But, as so often happens when people are seduced by promises of power, the price exacted in advance and all along the path, and the price actually paid, is servitude and impotence[… But] power is nothing if it is not the power to choose. Instrumental reason can make decisions, but there is all the difference between deciding and choosing”
—Joseph Weizenbaum, Computer Power and Human Reason, 1976

Programmers seem to confuse the imagined power of making computers do their bidding with the very real power of choosing.

In conclusion, do LLMs know how to make software?

Do LLMs know how to write code? Yes, with the help of brute-forcing and user input they can be relatively successful at doing so.

Do LLMs know how to make software? No, they can't choose between alternatives, they can only decide on them.

Don't let the agents take away your agency.

*some say except for Perl code