How Large Language Models Really Work: The Secrets Behind AI
By Veer Solanki · · 2456 words
Topics: AI, AI Hallucinations, AI Models, AI Reasoning, AI Research, AI Technology, AI Training, Artificial Intelligence
Most descriptions of large language models (LLMs) are either hopelessly abstract, involving the linear algebra necessary to define them, or reductive, arguing that they are only fancy autocomplete. Both are true, but both miss the interesting part.
An LLM is fundamentally a huge function: text in, probability distribution over text out. Everything else is a consequence of that definition and the way it is implemented, which is also fascinating.
Let me try to explain what really goes on in there.
The model doesn’t see words.
First, text is passed through a tokeniser, which shreds it into tokens. English tokens tend to be between logographic Chinese characters and whole words, but it varies between languages. The exact implementation depends on the model, but most use byte pair encoding, which iteratively merges the most common byte pairs in the language until they have between 50,000 and 250,000 tokens.
Common words will have their own token, while less common items are formed by combining two or more existing tokens. Programming languages and special characters get tokens too, depending on what the model was trained on.
This is a critical step that most people forget when interacting with the model. For example, most Indic languages take two to four times as many tokens to express the same ideas as English, making them disproportionately expensive and limiting context length. The same issue makes the model bad at basic tasks like counting the number of letters in a word, because it doesn’t see letters – only tokens. You can’t ask it to reverse a string, because it has no concept of a string – only tokens it must process sequentially. As an example, ‘rotate’ and ‘rote’, and ‘rote’ – ‘rotate’ are just different tokens from the model’s perspective.
The model only sees the tokens as integers. Each gets mapped to some vector in an embedding table, which is a learned relationship during training. This mapping is unintuitive – no tokens are ‘rotated versions of others’, so the model can’t generalise rotate -> rote. However, the emergent behaviour during training puts similar words and concepts close to each other in a high-dimensional space.
Meaning is a learned geometric relationship between points in a high-dimensional space. This allows the model to perform tasks that require understanding, like translating between languages that it wasn’t explicitly trained on, simply by finding analogous spatial relationships.
The model also needs to keep track of the sequence itself. Attention is permutation invariant by default, but position information is usually encoded with rotary position embeddings. These essentially rotate each token’s embedding around the origin at a different angle based on its position, allowing the model to determine the relative distance between tokens. This is just one implementation, but this particular innovation has caused exploding context lengths in recent years.
The residual stream
The meat of the model is its transformer blocks. They can be as simple as 30 or as complex as 100+ stacked identical units. Each one has the same basic functionality. You can think of them as a conveyor belt. The input embeddings get fed in at the beginning and then carried forward through each block, with modifications, until they are unembedded at the end.
Each block has two parts: the attention layer and the feedforward network. The attention is the part that everyone loves to describe as magic. It’s not, really. It simply calculates three embeddings for each token: query, key, and value. It then compares each token’s query against all of the previous tokens’ keys with a dot product and softmaxes the resulting weights, which determines how much of each value to use as context for the current token. This is repeated for multiple attention heads, which allows the model to learn different relationships.
Researchers have found that particular heads can encode syntactic information or even trace complex patterns in the input, such as repeated sequences or references. However, the most exciting application of attention is in induction heads, which allow the model to detect patterns in the input and repeat them. They tend to emerge at particular points in training, and their discovery coincided with the ability of the model to perform in-context learning tasks.
The second component of the block is the feedforward network. They are usually extremely deep and wide, with each token’s embedding projected to 4x the size of the model, nonlinearly activated, then projected back. They don’t have any inherent ability to attend to other tokens or use positional information, but they learn to encode patterns as they are passed through the network.
This is sometimes described as the block’s role as a key-value memory. The expansion and contraction process allows the model to store patterns it has seen in the residual stream, which can then be retrieved later. It is important to remember that in large language models everything is learned – not hardcoded – which means that facts are also encoded somewhere in these weights, albeit distributed across many.
Mixture of experts replaces the standard feedforward network with a group of them, usually significantly smaller, and a routing mechanism that determines which tokens get processed by which experts. It allows far more parameters to be used without a huge increase in computational requirements. A model with a trillion parameters can have a few experts with 30 billion each and only activate around ten per token. It makes parameter counts a meaningless spec for these models.
Generating a token
Once the end of the block is reached, the embeddings are multiplied by an unembedding matrix to generate logits, which are turned into probabilities with a softmax. The token with the highest probability is selected, either deterministically or with various sampling methods, and added to the context.
This is the crucial part: no planning happened. The model does not have a draft in a special buffer it then reviews to generate the next text. It guesses at one token at a time, with no ability to edit previous ones. The coherence comes from the context, which is why the model can continue writing after itself, but it has to start somewhere.
Two optimisations are commonly used in practice: caching the keys and values of the attention from previous tokens and only storing those that are useful for subsequent processing, and memory bandwidth optimisation, which is why generation is usually memory-bound and not compute-bound. Essentially, every token the model processes needs to read the entire weight matrix, which is much slower than doing it once for all tokens.
Training: prediction as a universal task
All of this is a description of the architecture, but nothing works until the weights are trained. The process is straightforward: show the model some text, ask it to predict the next token, measure how wrong it was, and adjust all of the weights to be slightly less wrong. It is repeated for trillions of tokens until the model can accurately predict anything that remotely resembles natural language.
The magic of the model is that it can do that, but the reason it can is that predicting the next token covers every domain. To accurately finish a murder mystery, the model must understand plot construction. To predict the next line of code, it must simulate a working programmer. To continue a chess game, it must play through the endgames and openings. Everything is compressed into this single task: encoding of text as a function approximator.
Some patterns are more useful than others, which is why scale laws exist and why alignment became such a crucial topic. Most large language models are already close to human performance on standard tests, but they needed enormous amounts of data and computation to get there. Chinchilla showed that most early research was severely parameter inefficient, and the entire field is now focused on finding the optimal balance between data, model size, and compute.
The economics of model training are also fascinating, but they are rarely discussed. The optimal point to stop training is determined by how much it costs: it is pointless to train a larger model if the data and compute needed for it will never be recouped. However, the final product is not used for training – it is deployed tens or hundreds of times, which makes the real optimal point much later than the economic one. It is common for 8 billion parameter models to outperform 70 billion ones because the small one was trained for much longer than it would usually be.
The state of data is the secret and the most interesting part. It is obvious that higher quality data improves the model, but there is little discussion of what that entails. Deduplication, curation, filtering, arranging, and even synthesis all play a part. It is the reason why some organisations’ models are so far ahead of the public ones: most of these improvements are proprietary. We cannot know what is in the best models because their creators do not discuss it.
The post-training process is equally important. The model needs to learn that it has a role: it is supposed to answer questions, not write stories. That usually involves some supervised fine-tuning to show the model what behaviours are desirable: being a helpful assistant, not a malicious one. Reinforcement learning from human feedback iterates on that to optimise the model for being helpful, polite, and honest.
RLHF is usually lossy: it is much easier to train a model to avoid toxic outputs than to make it confident and open about its limitations. It is also subject to the same problems as the rest of the training: reward hacking and deceptive alignment. The most annoying aspects of RLHF models are often the results of the training process, not the architecture.
RLHF is lossy: it is much easier to train a model to avoid toxic outputs than to make it confident and open about its limitations.
A better but underutilised method is RLHF using verifiable rewards. It is much easier to apply to tasks that have objective answers, such as code and math. The model is trained to perform well on them without additional training data, since correctness can be verified directly. It is the reason why large language models excel at those domains and why specialised reasoning models see periodic bursts of progress. It requires much more engineering, however: no reward model is needed, but the training process must be adapted to the specific task.
Why language models make things up
Hallucination is a sign of an LLM doing what it was designed to do. The model is not aware of reality in any fashion: it predicts the next token with no regard to whether things are entirely made up. It is right often enough that it seems normal, but it has no way of knowing when it is not. A correctly predicted citation is no different from one the model fabricated: both are high-probability outputs that fit the context.
That does not mean the model cannot distinguish truth from lies: a correctly predicted citation would have much higher probability than one the model just made up. There are three reasons why that does not prevent hallucinations: concepts in the model are distributed across many weights, making confidence difficult; post-training often discourages hedging language; and autoregressive generation does not allow backtracking.
Retrieval-augmented generation alleviates some of these issues but is ultimately an optimisation rather than a fundamental change. The same is true for reasoning traces: the model still has to hallucinate facts, although it can self-correct if given the opportunity. Neither addresses the fundamental issue, which is the training objective.
What is not understood
And now something about what we do not know. I will say this: vendors of these models do not fully understand what they are doing. They are incredibly intelligent and capable people, but you can tell from the state of the art on mechanistic interpretability.
We have a rough idea of how the models work at a high level: attention copies relevant information from other tokens, feedforward networks store facts and patterns as weight updates, and the whole system is optimised with gradient descent to learn the function that maps inputs to outputs. That is not detailed enough to reverse engineer a model from its weights but is useful for training new ones.
Superposition is the main impediment to further progress: a model uses the same dimensions to encode multiple concepts, especially when the context only contains one of them. It is possible to disentangle some concepts with sparse autoencoders but only for particularly interesting or common facts, such as concrete objects. Researchers have successfully extracted circuitry describing certain behaviours and used the weights to manually tweak the model.
It is an exciting area of research, but it is also prohibitively time-consuming and only partially applicable to real-world frontier models. If someone claims that we do not understand these models, they are correct, but so is the person insisting we do not understand anything, period.
On benchmarks
Finally, a note about understanding what the model can do. Ignore the vendor tables: treat them as marketing, because they are. Contamination is the main issue – it is exceptionally hard to ensure that test sets are completely separated from training data. Evaluation suites are also typically chosen after observing model performance, which skews results. The final nail in the coffin is practicality; the stated metrics are often obtained with best-of-k sampling or additional prompting that is unavailable when using the product. Vendors will rarely publish head-to-head comparisons on realistic tasks, especially if doing so would hurt their bottom line. If they do, pay attention.
Summary
A large language model is a series of matrix multiplications trained to predict the next token with no regard to reality, fine-tuned to perform specific tasks, and then adjusted further based on vague human feedback. It does not reason; it does not simulate; it does not possess a world model in any meaningful way. It has a residual stream, attention heads that copy context, and feedforward networks that store facts, and that is all.
This is a disheartening view of the LLM, but it is also misleading in its simplicity. These observations are all correct at a high level but fail to capture the emergent effects of scale and training. The model described above would be incapable of writing coherent software, but adding layers and training data makes it able to do so. No one understands precisely why, but we know it happens because it clearly does.
These facts are simultaneously unsatisfying and thrilling: the mechanism is intuitive and well-described, but it is not what gives these models their power. That is the true marvel of the LLM; this is where our understanding is limited, and all future research must begin.