You now have every component. This post assembles them. If you have read attention and embeddings, the transformer will look like something you already half know.
The architecture, top to bottom
Text enters, is tokenised, becomes embeddings, passes through N identical blocks, and emerges as a probability for every possible next token.
Positional encoding: why order needs adding back
Attention has a property that is easy to miss: it is order-blind. It computes a weighted average, and averages do not care about sequence. Shuffle the input and attention produces the same result.
Inside one transformer block
Two sub-layers, and they do genuinely different jobs.
| Sub-layer | What it does | Tokens interact? |
|---|---|---|
| Self-attention | Each token gathers information from the others | Yes — this is the only place they do |
| Feed-forward | Processes each token’s vector independently | No — same small network applied to each |
A useful way to hold it: attention moves information sideways between tokens; the feed-forward layer thinks about each token on its own. Alternating the two, dozens of times, is the whole design.
The feed-forward part is an ordinary two-layer neural network — the exact structure covered earlier on this site, with a non-linear activation in the middle. It also holds most of the model’s parameters.
The two details that make deep stacks trainable
Residual connections. Each sub-layer’s output is added to its input rather than replacing it. This gives gradients a direct path back through the network, which is what allows 96 layers to train at all — without it you hit vanishing gradients long before that depth.
Layer normalisation. Each token’s vector is renormalised to a consistent scale between sub-layers, keeping values in a stable range. Same motivation as feature scaling, applied continuously inside the network.
Neither is glamorous, and both are load-bearing. Remove them and deep transformers do not train.
Encoder, decoder, or both
Why transformers replaced everything else
The predecessor architectures — RNNs and LSTMs — processed tokens strictly one at a time, because each step needed the previous step’s output. That made them impossible to parallelise across a sequence, and it made long-range dependencies fade.
🧪 The insight that unlocked scale
That is the real breakthrough. Attention is elegant, but parallelism is what made it possible to train on trillions of tokens — and everything since has followed from being able to scale.
Note the trade: transformers gained parallelism and paid for it with quadratic cost in sequence length. On the hardware available, that was overwhelmingly worth it.
🔑 Key Takeaways
- A transformer block is attention plus a feed-forward layer, stacked N times.
- Attention moves information between tokens; the feed-forward layer processes each alone.
- Attention is order-blind, so positional encoding is added to the embeddings.
- Residual connections and layer normalisation are what make deep stacks trainable.
- Parallel processing of the whole sequence — not attention itself — is what enabled the scale.
Further reading
Jay Alammar’s Illustrated Transformer remains the clearest visual walkthrough, and Andrej Karpathy’s nanoGPT is a complete, readable implementation in a few hundred lines of Python.
Where to go next
- Fine-tuning vs prompting vs RAG — the next post: putting models to work
- Attention, explained simply — the mechanism at the core
- The vanishing gradient problem — what residual connections solve
- Start Here — the full learning path