How Large Language Models Actually Work: Tokens, Attention and Transformers Explained With Code

Where this guide comes from

This post is built around Piyush Garg's video How LLMs Works? - Overview (channel), a roughly one-hour walkthrough of what happens between your prompt and the model's answer. I've kept its order (GPT, transformer, tokens, embeddings, positions, attention, training, sampling) and added diagrams, runnable code and the original research paper. If you prefer watching, jump to the chapter that matches each section:

Other videos worth lining up next to it:

• 3Blue1Brown, "Transformers, the tech behind LLMs" (video, companion lesson): the best visual introduction to embeddings and attention.
• Andrej Karpathy, "Intro to Large Language Models" (video): a one-hour talk on how LLMs are trained and where the field is heading.
• Andrej Karpathy, "Let's build GPT: from scratch, in code, spelled out" (video): the hands-on follow-up, roughly two hours of building a GPT-style model line by line.

And the paper that started the modern era: Attention Is All You Need (Vaswani et al., Google, 2017) on arXiv (PDF).

What is a large language model?

A large language model (LLM) is a neural network trained to predict the next token in a sequence of text. "Large" refers to the number of learned parameters (billions to trillions) and the amount of training text. "Language model" refers to the job: assign probabilities to what comes next.

Chat assistants like ChatGPT, Claude and Gemini are LLMs that were additionally trained to follow instructions and be helpful. Underneath, they still do one thing per step: score every token in the vocabulary, pick one, repeat. Karpathy's intro talk describes it the same way, a model that predicts the next word, then gets shaped into an assistant through further training.

What GPT actually stands for

The video opens with the acronym, and it's a good compass for the rest of this guide. GPT = Generative Pre-trained Transformer.

• Generative: it produces new text, code or ideas rather than just classifying input.
• Pre-trained: it's first trained on a massive text corpus, then fine-tuned for specific behavior (more on that in the training section).
• Transformer: the neural network architecture underneath, introduced in the 2017 paper Attention Is All You Need. It replaced older recurrent networks (RNNs) because it processes all tokens in parallel.

Here is the entire generation loop in plain Python. The rest of the article explains what hides inside model(...):

def generate(model, tokenizer, prompt, max_new_tokens=50):
    ids = tokenizer.encode(prompt)            # text -> list of integers
    for _ in range(max_new_tokens):
        logits = model(ids)                   # scores for every token in the vocab
        next_id = sample(logits[-1])          # pick one token from the distribution
        ids.append(next_id)                   # feed it back in
        if next_id == tokenizer.eos_id:       # model says "I'm done"
            break
    return tokenizer.decode(ids)

That for loop is also why LLM output streams word by word: each token genuinely doesn't exist until the previous one has been produced. It's also the starting point for what people now call the harness around a model; the agent harness explainer covers how loops, tools and memory wrap this core.

Step 1: Tokenization, how text becomes numbers

Neural networks do math on numbers, so text has to be converted first. LLMs don't read characters or whole words. They read tokens: subword chunks from a fixed vocabulary, typically 30,000 to 200,000 entries.

Why subwords? Whole-word vocabularies can't handle new words, typos, names or code. Single characters make sequences very long and expensive. Subword tokenization is the compromise: common words become one token, rare words are split into familiar pieces, and anything can be spelled out as a fallback.

That has practical consequences you've probably already bumped into:

• Pricing and limits are in tokens, not words. English text averages somewhere around 3/4 of a word per token, and other languages and code often cost more.
• Letter-level tasks are hard. The model sees ["Un", "believ", "ably"], not individual letters, which is part of why counting letters or reversing strings trips models up.
• Whitespace and casing matter. " fast" and "fast" are different tokens.

The most common algorithm is Byte Pair Encoding (BPE): start with single characters, repeatedly merge the most frequent adjacent pair into a new token, stop at the target vocabulary size. Here's a tiny working version:

from collections import Counter

def learn_bpe(words, num_merges):
    # each word becomes a tuple of characters plus an end-of-word marker
    vocab = Counter(tuple(w) + ("</w>",) for w in words)
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for word, freq in vocab.items():
            for a, b in zip(word, word[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = max(pairs, key=pairs.get)       # most frequent adjacent pair
        merges.append(best)
        new_vocab = Counter()
        for word, freq in vocab.items():
            out, i = [], 0
            while i < len(word):
                if i < len(word) - 1 and (word[i], word[i + 1]) == best:
                    out.append(word[i] + word[i + 1]); i += 2
                else:
                    out.append(word[i]); i += 1
            new_vocab[tuple(out)] += freq
        vocab = new_vocab
    return merges, vocab

corpus = ["low"] * 5 + ["lower"] * 2 + ["newest"] * 6 + ["widest"] * 3
merges, vocab = learn_bpe(corpus, 8)
print(merges)
for word, freq in vocab.items():
    print(freq, word)

Running it prints:

[('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w'), ('n', 'e'), ('ne', 'w'), ('new', 'est</w>')]
5 ('low', '</w>')
2 ('low', 'e', 'r', '</w>')
6 ('newest</w>',)
3 ('w', 'i', 'd', 'est</w>')

Notice how est</w> (a common word ending) became a single token that is shared between "newest" and "widest". That's exactly how real tokenizers learn suffixes like -ing or -tion without anyone programming them in. To see a production tokenizer, try OpenAI's tiktoken or the Hugging Face tokenizers and print the pieces for your own prompts.

Step 2: Embeddings, how numbers become meaning

A token ID like 3118 is just a label. It carries no meaning, and 3118 isn't "close" to 3119 in any useful way. So the first layer of the model is an embedding table: a big matrix with one row per vocabulary token. Looking up row 3118 gives a vector of, say, 768 to 12,288 numbers.

These vectors are learned during training. Tokens used in similar contexts end up pointing in similar directions, which is why the famous "king − man + woman ≈ queen" demos work: directions in the space encode concepts. (It's an illustration, not a law, and real embeddings have thousands of dimensions that don't map neatly to human concepts.) The vector search guide shows how the same idea powers semantic search and RAG.

Positional information

Here's a catch: attention (next section) treats its input as an unordered set. Without extra information, "The cat chased the mouse" and "The mouse chased the cat" would look identical, even though they mean opposite things. So the model adds position information to each embedding. The original paper used fixed sine and cosine waves of different frequencies; many modern models learn positions or use rotary embeddings (RoPE) instead.

import numpy as np

def positional_encoding(n_positions, d_model):
    pos = np.arange(n_positions)[:, None]            # (n, 1)
    i = np.arange(d_model)[None, :]                  # (1, d)
    angle = pos / np.power(10000, (2 * (i // 2)) / d_model)
    return np.where(i % 2 == 0, np.sin(angle), np.cos(angle))

pe = positional_encoding(4, 8)
print(pe.shape)  # (4, 8): one unique pattern per position, added to the token embeddings

After this step, every token is a vector that says both what it is and where it sits.

Step 3: Self-attention, how tokens read each other

This is the heart of the transformer, and the reason for the paper title. Before 2017, the dominant language models were recurrent networks that processed text one word at a time, squeezing everything seen so far into a single hidden state. That was slow to train (no parallelism) and forgetful over long distances.

The Attention Is All You Need paper proposed a network built solely on attention, with no recurrence and no convolutions. Its abstract reports 28.4 BLEU on WMT 2014 English-to-German and 41.8 BLEU on English-to-French translation, beating the previous best results while being far more parallelizable to train.

The intuition: queries, keys and values

A quick example from the video: in "the bank of the river", self-attention lets "bank" pull context from "river" instead of leaning on its financial meaning. Same word, different meaning, decided by the words around it.

Take the sentence "The animal didn't cross the street because it was too tired." To understand "it", the model has to look back and figure out that "it" means "animal". Attention is how it does that look-back, using three vectors per token:

• Query (Q): what this token is looking for.
• Key (K): what this token offers, a label other tokens can match against.
• Value (V): the information this token passes along if it's selected.

Each token's Q is compared against every other token's K. A high match means "pay attention to that one." The scores are turned into weights with softmax, and the output is a weighted blend of the Values. In the paper's notation:

Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V

The division by √d_k keeps the scores from getting huge when vectors are long, which would push softmax into saturated regions where learning stalls.

Self-attention in 25 lines of NumPy

import numpy as np
np.random.seed(0)

def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)   # subtract max for numerical stability
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

def self_attention(X, Wq, Wk, Wv, causal=True):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv          # project each token three ways
    scores = Q @ K.T / np.sqrt(K.shape[-1])   # how much does each token match every other?
    if causal:                                # GPT-style: a token can't peek at the future
        mask = np.tril(np.ones_like(scores)) == 1
        scores = np.where(mask, scores, -1e9)
    weights = softmax(scores)                 # each row sums to 1
    return weights @ V, weights

words = ["the", "cat", "sat", "down"]
d_model, d_k = 8, 4
X = np.random.randn(len(words), d_model)      # pretend these are embeddings + positions
Wq, Wk, Wv = (0.5 * np.random.randn(d_model, d_k) for _ in range(3))

out, w = self_attention(X, Wq, Wk, Wv)
np.set_printoptions(precision=2, suppress=True)
print(w)
print(out.shape)   # (4, 4): one new vector per token

The weight matrix printed here is lower-triangular, and every row sums to 1:

[[1.   0.   0.   0.  ]
 [0.92 0.08 0.   0.  ]
 [0.02 0.07 0.9  0.  ]
 [1.   0.   0.   0.  ]]

Row 1 ("cat") is mostly looking at itself and "the"; it can't look at "sat" or "down" because of the causal mask. That mask is what makes a decoder-only model (GPT, Claude, Llama and friends) a next-token predictor: during training, every position predicts its next token without being able to cheat by reading it. The weights here come from random matrices, so the pattern means nothing yet. Training is what makes these weights meaningful.

Multi-head attention

One attention pass can only learn one kind of relationship. Multi-head attention runs several smaller attention operations in parallel, each with its own Q/K/V projections, then concatenates and mixes the results. One head might track grammar, another coreference ("it" → "animal"), another nearby words. The base model in the paper used 8 heads with a model dimension of 512.

def multi_head(X, heads, Wo):
    # heads: list of (Wq, Wk, Wv); each head works in a smaller subspace
    outs = [self_attention(X, *h)[0] for h in heads]
    return np.concatenate(outs, axis=-1) @ Wo   # mix the heads back to d_model

heads = [tuple(0.5 * np.random.randn(d_model, d_k) for _ in range(3)) for _ in range(2)]
Wo = np.random.randn(2 * d_k, d_model)
print(multi_head(X, heads, Wo).shape)   # (4, 8): same shape as the input

Because the output has the same shape as the input, you can stack these operations again and again. Which brings us to the next piece.

Step 4: The transformer block, and why we stack it

Attention lets tokens exchange information. But the model also needs to process what each token has gathered. So each transformer block pairs attention with a small feed-forward network (MLP) applied to every token independently. Two details make deep stacks trainable:

• Residual connections: each sub-layer's output is added to its input, so information and gradients can flow through dozens of layers.
• Layer normalization: keeps activations in a stable range.

A useful way to think about it: attention is communication between tokens, and the MLP is computation inside each token. Repeat that N times and early layers tend to capture local, surface patterns while later layers capture more abstract ones. (This is a common intuition from interpretability research, not a clean rule.)

The original paper's base model used an encoder-decoder layout with 6 encoder layers and 6 decoder layers, because it was built for translation (the encoder reads the source sentence, the decoder writes the translation token by token). Modern chat LLMs are typically decoder-only and much deeper: GPT-3, for example, had 96 layers and 175 billion parameters. The final step projects each token's vector to a score for every vocabulary entry, those scores are called logits.

Putting it together: a tiny causal transformer block

Here's what the block looks like in PyTorch. It's the same math as the NumPy above, wrapped in layers you can train. Karpathy's Let's build GPT video builds a full working model from this same set of pieces.

import torch
import torch.nn as nn
import torch.nn.functional as F

class CausalSelfAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        assert d_model % n_heads == 0
        self.n_heads = n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model)   # Q, K, V in one matrix multiply
        self.proj = nn.Linear(d_model, d_model)

    def forward(self, x):                              # x: (batch, time, d_model)
        B, T, C = x.shape
        q, k, v = self.qkv(x).split(C, dim=-1)
        # reshape to (batch, heads, time, head_dim)
        q, k, v = (t.view(B, T, self.n_heads, C // self.n_heads).transpose(1, 2) for t in (q, k, v))
        # scaled dot-product attention with the causal mask built in
        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        y = y.transpose(1, 2).contiguous().view(B, T, C)
        return self.proj(y)

class Block(nn.Module):
    def __init__(self, d_model=256, n_heads=4):
        super().__init__()
        self.ln1 = nn.LayerNorm(d_model)
        self.attn = CausalSelfAttention(d_model, n_heads)
        self.ln2 = nn.LayerNorm(d_model)
        self.mlp = nn.Sequential(
            nn.Linear(d_model, 4 * d_model), nn.GELU(), nn.Linear(4 * d_model, d_model)
        )

    def forward(self, x):
        x = x + self.attn(self.ln1(x))   # communication between tokens, with a residual
        x = x + self.mlp(self.ln2(x))    # computation inside each token, with a residual
        return x

class TinyGPT(nn.Module):
    def __init__(self, vocab_size, d_model=256, n_heads=4, n_layers=4, max_len=128):
        super().__init__()
        self.tok = nn.Embedding(vocab_size, d_model)
        self.pos = nn.Embedding(max_len, d_model)
        self.blocks = nn.Sequential(*[Block(d_model, n_heads) for _ in range(n_layers)])
        self.ln = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab_size)

    def forward(self, ids):                              # ids: (batch, time)
        x = self.tok(ids) + self.pos(torch.arange(ids.shape[1], device=ids.device))
        return self.head(self.ln(self.blocks(x)))        # logits: (batch, time, vocab)

Count the parts: embeddings, N blocks, one output layer. That's the whole architecture of a GPT-style model. Scale d_model, n_layers and the training data by several orders of magnitude and you're in frontier-model territory. The architecture is surprisingly small; the data, compute and tuning are what make it expensive.

Step 5: From scores to a word, softmax and sampling

The last layer outputs one logit per vocabulary token. Softmax converts logits into probabilities that sum to 1. Then the model has to pick one. How it picks is controlled by a handful of knobs you'll see in every API.

• Temperature: divides the logits before softmax. Low values sharpen the distribution (almost always the top token); high values flatten it (more surprising picks).
• Top-k / top-p (nucleus): restrict sampling to the k most likely tokens, or the smallest set whose probabilities add up to p. This cuts off the long tail of nonsense tokens.
• Greedy decoding: always pick the top token. Deterministic, but can get repetitive.

def sample(logits, temperature=1.0, top_k=None):
    logits = np.array(logits, dtype=float) / max(temperature, 1e-6)
    if top_k:
        cutoff = np.sort(logits)[-top_k]
        logits = np.where(logits < cutoff, -np.inf, logits)   # drop everything below top-k
    probs = softmax(logits)
    return np.random.choice(len(probs), p=probs), probs

# "The cat sat on the ___"   vocab: mat, sofa, moon, pizza
for t in (0.2, 1.0, 2.0):
    print(t, sample([4.0, 3.0, 1.0, 0.5], temperature=t)[1])

Output:

0.2 [0.99 0.01 0.   0.  ]
1.0 [0.69 0.25 0.03 0.02]
2.0 [0.5  0.3  0.11 0.09]

This is also the real answer to "why does the same prompt give different answers?": the model outputs a distribution, and sampling draws from it. Set temperature to 0 and you get (nearly) the same answer each time. If you're building on top of model APIs, the LLM API guide shows where these parameters live in practice.

Why generation is fast: the KV cache and the context window

Naively, producing token 500 would require recomputing attention over all 499 previous tokens from scratch. Real systems avoid that with a KV cache: the Keys and Values for earlier tokens never change (thanks to the causal mask), so the model stores them and only computes Q, K and V for the newest token.

This also explains the context window, the maximum number of tokens (prompt plus output) the model can attend to at once. Attention compares every token with every other token, so cost grows quickly with length, and the KV cache consumes GPU memory in proportion to it. That's why long prompts cost more, why very long contexts are an engineering challenge, and why ideas like cache-augmented generation are popular for reusing a large, fixed knowledge base.

A practical consequence: the model has no memory between requests. Anything it "remembers" in a chat is simply earlier messages re-sent inside the context window.

How the model learns: pre-training, fine-tuning and preference tuning

Everything above describes how a trained model runs. Training is how its billions of weights get their values.

1. Pre-training. The model reads an enormous amount of text and, at every position, predicts the next token. The loss is −log P(correct next token): if the model gave the right token low probability, the loss is big, and backpropagation nudges the weights to do better. Repeat across trillions of tokens and the model absorbs grammar, facts, code patterns and reasoning habits as a side effect of getting good at prediction. The result is a base model: a powerful autocomplete, not yet an assistant.
2. Supervised fine-tuning (SFT). Train on curated examples of instructions paired with good responses, so the model learns the format of being an assistant.
3. Preference tuning. Humans (or other models) rank candidate answers, and the model is optimized toward the preferred ones using methods such as RLHF or DPO. Newer reasoning models also use reinforcement learning on tasks with checkable answers, like math and code.

A minimal training step in code looks like this, using the TinyGPT from earlier:

model = TinyGPT(vocab_size=50_000)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)

def train_step(batch):                         # batch: (B, T+1) token IDs
    inputs, targets = batch[:, :-1], batch[:, 1:]   # target = input shifted by one token
    logits = model(inputs)                           # (B, T, vocab)
    loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1))
    opt.zero_grad()
    loss.backward()                            # backpropagation
    opt.step()                                 # nudge weights
    return loss.item()

The "labels" are just the text itself shifted by one position, no human annotation is needed, which is why pre-training can scale to internet-sized data.

Try a real model in 10 lines

You don't need to train anything to poke at a real model. This uses the small open GPT-2 weights through Hugging Face transformers to print the model's top guesses for the next token:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

ids = tok("The capital of France is", return_tensors="pt")
with torch.no_grad():
    logits = model(**ids).logits[0, -1]          # scores for the next token only
probs = torch.softmax(logits, dim=-1)
top = torch.topk(probs, 5)
for p, i in zip(top.values, top.indices):
    print(f"{tok.decode(i)!r:>10}  {p.item():.1%}")

You'll see the exact thing this article described: a probability for each candidate token, no "understanding" step in sight. Try changing the prompt and watching how the distribution moves.

Common misconceptions

• "It looks answers up in a database." No. Knowledge is encoded in the weights, which is why models can state wrong things confidently. Retrieval (RAG, search tools) is bolted on separately.
• "It thinks in words." It operates on vectors; words only appear at the input and output ends.
• "Bigger context means better memory." A larger window lets it see more text; it doesn't make attention over that text equally reliable at every position.
• "Temperature 0 means it's always correct." It means it's more deterministic, not more truthful.
• "The paper is only about translation." It was evaluated on translation, but the architecture (especially the decoder with a causal mask) became the basis for GPT-style language models.

What to build or read next

Once the mental model is in place, the practical questions get easier: why a prompt gets truncated, why outputs vary, why tool-calling needs a loop, why costs scale with tokens.

• Wrap the model in tools, memory and a loop: what an agent harness is and the smallest AI agent walkthrough.
• Run a model on your own machine and see all of this locally: running DeepSeek locally with Ollama and LangChain.
• Follow a structured path from fundamentals to production agents with the AI Agent Developer roadmap.
• Browse more practical AI write-ups on the iHateReading blog.

Resources

Paper

• Attention Is All You Need, Vaswani et al., 2017 (arXiv:1706.03762), PDF

Videos

• How LLMs Works? - Overview, Piyush Garg
• 3Blue1Brown, Transformers, the tech behind LLMs
• Andrej Karpathy, Intro to Large Language Models
• Andrej Karpathy, Let's build GPT from scratch

Articles and courses

• 3Blue1Brown lesson: Transformers and GPT
• The Illustrated Transformer, Jay Alammar
• How transformer models work for language processing, freeCodeCamp
• How Transformer LLMs Work, DeepLearning.AI
• nanoGPT by Andrej Karpathy, a minimal GPT training codebase

Key takeaways

1. An LLM is a function from a token sequence to a probability distribution over the next token, run in a loop.
2. Text becomes tokens (subwords), tokens become embedding vectors, and position information is added.
3. Self-attention lets every token weigh every earlier token using queries, keys and values: softmax(QKᵀ/√d_k)·V.
4. A transformer block is attention (communication) plus an MLP (computation), with residuals and normalization; stack it N times.
5. Logits become probabilities via softmax, and sampling settings like temperature and top-p control how one token is chosen.
6. Training is next-token prediction at huge scale, followed by instruction tuning and preference tuning to make an assistant.

The next time a model gives you a surprising answer, you'll have a better guess about where in this pipeline it came from.