Tokenisation gave us integer IDs, and those IDs mean nothing — token 5771 is no closer to 5772 than to 90000. Embeddings fix that by replacing each ID with a vector positioned so that closeness in space means closeness in meaning.
This is where your linear algebra stops being theory. Everything below is vectors, distances and dot products.
What an embedding looks like
An embedding is simply a vector — typically 384, 768 or 1,536 numbers. The word “dog” might become [0.21, −0.44, 0.87, …]. On its own that list is meaningless. What matters is where it sits relative to every other vector.
The famous demonstration is that king − man + woman lands near queen. The gender relationship is a consistent direction in the space, so you can add and subtract it like any other vector.
⚠️ The analogy is a real effect, but it is often overstated
Measuring similarity
Two vectors are compared by the angle between them, using cosine similarity:
$$\text{similarity} = \frac{A \cdot B}{\|A\| \, \|B\|}$$
That is the dot product divided by both magnitudes — exactly the formula from your linear algebra pages, now doing real work.
Why embeddings beat keyword search
Search for “how do I fix a puncture” and keyword matching finds documents containing those exact words. A document titled “repairing a flat tyre” contains none of them, and is exactly what you wanted.
Embedding both and comparing vectors solves this — the two phrases land close together because they mean the same thing. That is semantic search, and it is the mechanism behind retrieval-augmented generation, recommendation systems, duplicate detection and clustering of documents.
Word embeddings versus sentence embeddings
| Type | One vector per | Limitation |
|---|---|---|
| Static (word2vec, GloVe) | Word, fixed forever | “Bank” gets one vector for both the river and the money sense |
| Contextual (BERT-style) | Word in its sentence | More compute; needs the surrounding text |
| Sentence / document | Whole passage | The practical choice for search and RAG |
Contextual embeddings are what attention produces inside a modern model: the vector for “bank” is shaped by the words around it, so the two senses separate properly.
Where embeddings come from
They are learned, not designed. Nobody decided which dimension means “royalty” — and no individual dimension cleanly does. The vectors are parameters, adjusted by backpropagation like any others, under a training objective that rewards putting words used in similar contexts near each other.
The underlying assumption is old and linguistic: words appearing in similar contexts tend to mean similar things. Embeddings are that idea made numerical.
Two practical cautions
⚠️ Before you build on embeddings
- They inherit bias from training text. If the corpus associates certain professions with certain genders, the vectors encode that association — and any system built on them repeats it. See bias in machine learning.
- Never mix models. Vectors from two different embedding models are not comparable, even at the same dimensionality. Re-embed everything when you change model.
🔑 Key Takeaways
- An embedding is a vector whose position encodes meaning.
- Cosine similarity — a dot product over two magnitudes — measures how related two texts are.
- Embeddings find matches that share no words at all, which keyword search cannot.
- Contextual embeddings solve the “bank” problem that static ones cannot.
- They carry the biases of their training text, and vectors from different models never mix.
Further reading
The Sentence Transformers library is the practical starting point for generating your own embeddings in a few lines of Python, and Google’s embeddings course module covers how the vectors are learned.
Where to go next
- Attention, explained simply — the next post in this sequence
- Cosine similarity calculator — compare two vectors yourself
- Fine-tuning vs prompting vs RAG — embeddings in production
- Start Here — the full learning path