Embeddings: from identifiers to vectors with meaning

In the previous chapter we saw how to turn text into a sequence of numeric IDs with a tokenizer. But an ID like 5089 is nothing more than a label: it tells the model nothing about what that word means, or how it relates to others. A language model cannot do useful mathematics with a mere bin number.

We need to transform that ID into something the model can operate on: a vector of real numbers. That vector is what we call an embedding. And the fascinating part is that, during training, those vectors organize themselves into a semantic space where similar words sit close together, and relationships get encoded as geometric directions.

From identifier to vector: the lookup table

The step is simple: each ID acts as an index into a table (an embeddings matrix). That table has one row for each token in the vocabulary, and each row is a vector of fixed dimension (for example, 50, 300, or 4096 numbers).

"cat" → ID 5089 → embedding[5089] = [0.10, -0.11, -0.12, 0.00, 0.05, ...]  (50 numbers)

This vector is not computed; it is looked up in the table. It is like a dictionary that assigns each word a list of numbers. At the start of training, those numbers are random: cat and dog have vectors as different as cat and tuesday. There is no relationship at all.

But during training, the model adjusts these vectors (along with the rest of the parameters) to minimize prediction error. The gradient of the loss pushes the vectors of words that appear in similar contexts closer together, and those that appear in different contexts farther apart. That way, without anyone explicitly telling it that cat and dog are animals, the model discovers it on its own from the statistics of the text.

(Strictly speaking, LLMs embed tokens, not always complete words — here we use “words” to keep things simple.)

import torch.nn as nn

embedding_layer = nn.Embedding(num_embeddings=vocab_size, embedding_dim=dim)
# This layer internally has a matrix of shape (vocab_size, dim)

ids = tokenizer.encode("the cat slept")   # [464, 5089, 3332]
x = embedding_layer(ids)                    # shape: (3, dim)
# x[0] is the vector for "the", x[1] for "cat", x[2] for "slept"

Semantic space: closeness as similarity of use

Once trained, the embeddings space has a remarkable property: the distance between vectors reflects the semantic (and syntactic) similarity of the words.

If you ask for the nearest neighbors of cat, you get dog, rabbit, monkey, mouse… all animals. For red, you get other colors. For three, other numbers.

There is no dictionary and no category label; the model has inferred these groupings exclusively from the company each word keeps — the famous idea from the linguist J.R. Firth: “you shall know a word by the company it keeps”.

Illustrative table (GloVe embeddings, English, real results):

Notice how cat and dog sit very close, while table or red sit very far (or even in almost opposite directions). The model has never seen a picture of a cat and has never been given a definition; it has only read billions of texts in which cat and dog appear in similar contexts.

Zooming out, the entire space organizes itself into thematic neighborhoods — animals in one region, colors in another, numbers, cities, family, actions, each group in its own zone — and all of this emerges without supervision, with only the objective of predicting the next word.

Measuring closeness: cosine similarity, in depth

We have said that similar words sit close in the embeddings space. But how do we measure that closeness numerically?

The standard measure is cosine similarity. To understand it, forget vectors of hundreds of dimensions for a moment and think in a space of only 2 dimensions — a plane.

Imagine each word is an arrow leaving the origin (0,0). Cosine similarity measures the angle two arrows form with each other:

• If two arrows point in exactly the same direction (angle 0°), their cosine is 1 — identical in direction, even if one is longer.
• If they are perpendicular (angle 90°), their cosine is 0 — no relationship.
• If they point in opposite directions (angle 180°), their cosine is -1 — semantic antonyms.

Why cosine and not Euclidean distance (the straight line between two points)? Euclidean distance is very sensitive to vector length — and in trained embeddings, a vector’s length usually correlates with the word’s frequency (very common words tend to have longer vectors). If we used Euclidean distance, two rare but closely related words could look “far” just because their vectors are short. Cosine, by normalizing for length, looks only at direction — which is precisely what encodes meaning.

Formula, step by step:

1. Dot product (A · B): you multiply each component of A by its counterpart in B, and add: Σ (Aᵢ × Bᵢ).
2. Norm (‖A‖): square root of the sum of the squares of its components: √Σ Aᵢ² — the “length” of the arrow.
3. Normalization: you divide the dot product by the product of the norms.

cosine_similarity(A, B) = (A · B) / (‖A‖ × ‖B‖)

A concrete numeric example, in 2D:

Let cat = A = [2, 1] and dog = B = [3, 1.5] (it points in the same direction, but is 1.5 times longer):

• Dot product: (2×3) + (1×1.5) = 6 + 1.5 = 7.5
• Norm of A: √(2² + 1²) = √5 ≈ 2.236
• Norm of B: √(3² + 1.5²) = √11.25 ≈ 3.354
• Cosine: 7.5 / (2.236 × 3.354) = 7.5 / 7.5 = 1.0 — perfectly aligned!

Now let tuesday = C = [0, 1]:

• Dot product with A: (2×0) + (1×1) = 1
• Norm of C: 1
• Cosine: 1 / (2.236 × 1) ≈ 0.447 — an angle of about 63°, barely related.

Under the hood — the version we just computed by hand, in pure Python:

from math import sqrt

def cosine(a, b):
    dot_product = sum(x * y for x, y in zip(a, b))
    length_a = sqrt(sum(x * x for x in a))
    length_b = sqrt(sum(y * y for y in b))
    return dot_product / (length_a * length_b)  # 1 = same direction

And the NumPy version, which is how it is actually implemented in practice (same formula, using vectorized operations instead of explicit loops — faster over vectors of hundreds or thousands of dimensions):

import numpy as np

def cosine(a, b):
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

# Practical example: find the words most similar to "cat"
def nearest_neighbors(embedding, word, k=5):
    word_vec = embedding[word]
    similarities = {}
    for token, vec in embedding.items():
        if token == word:
            continue
        similarities[token] = cosine(word_vec, vec)
    return sorted(similarities.items(), key=lambda x: x[1], reverse=True)[:k]

Embeddings in action: semantic search, and the contrast with BM25

Now that we know how vectors are compared, we can understand their great practical use: semantic search. But it is worth knowing first the classical approach that came before.

The classical approach: BM25. Before embeddings became popular, search engines used algorithms like BM25 (Best Matching 25): they split the query into keywords, look for documents that contain them exactly, and score by term frequency, how rare that word is across the whole collection, and document-length normalization. It is fast, needs no training, and works very well for proper names, product codes, or very specific terms.

BM25’s limitation: it is blind to meaning. It depends on your using exactly the same words that appear in the document. If you search for “electric car” and a document talks about the “battery-powered automobile”, BM25 does not find it — it does not know they are synonyms. There is a lexical gap between what you write and what the document contains.

The modern approach: turn the query and the documents into vectors, compute cosine similarity between them, and return the closest ones — it does not matter whether the exact words match, only whether the meaning is similar.

Did BM25 become obsolete? No — in practice, both approaches are combined in hybrid search: BM25 captures exact matches (names, codes), embeddings capture general meaning, and the scores are combined to get the best of both worlds.

This duality — keyword search vs. semantic search — is exactly the mechanism we already took apart in depth in Part II: it is the basis of RAG and of how vector databases work (with their own dedicated chapter). There we pick up this same cosine similarity to explain how a system retrieves the document chunks most relevant to answering a question — the mechanics are identical to what we just computed by hand here, only operating over thousands or millions of vectors instead of three.

Directions mean something too

One of the most surprising properties of embeddings is that arithmetic operations with vectors capture semantic relationships:

The “shift” that goes from man to king (which encodes the idea of “royalty”) is very similar to the shift that goes from woman to queen. This works for several kinds of relationship:

It is important to be honest: this is an approximate geometric regularity, not a literal subtraction of “essence”. The model does not know anything and makes no judgments; the answer is the nearest vector, with no guarantee of correctness, and on hard analogies or infrequent words it fails often.

def analogy(word_a, word_b, word_c, excluded=None):
    vec_a = embedding[word_a]
    vec_b = embedding[word_b]
    vec_c = embedding[word_c]
    target = [b - a + c for a, b, c in zip(vec_a, vec_b, vec_c)]
    excluded = excluded or {word_a, word_b, word_c}
    best, best_sim = None, -1
    for word, vec in embedding.items():
        if word in excluded:
            continue
        sim = cosine(vec, target)
        if sim > best_sim:
            best_sim, best = sim, word
    return best

Analogy calculator (the lab)

The table above shows the idea. Below, the same operation B − A + C runs over real 50-dimensional GloVe vectors — the same ones that feed this chapter’s cat/dog table. It is arithmetic and cosine similarity, not training.

Caution: biases get learned too

This same geometry that encodes useful relationships also encodes stereotypes and biases present in the training texts. In some widely studied embeddings it has been observed that programmer or engineer sit closer to man, while nurse sits closer to woman.

This is not a deliberate decision by the model — it is a reflection of the statistical distributions in the data. If historical texts associate certain professions with genders, the embedding captures that. This has important implications: models that use these embeddings can perpetuate or amplify those biases. That is why, in practice, debiasing techniques are applied (neutralizing specific bias directions) or the training data is curated more carefully.

What about position? Positional embeddings

So far we have represented each token in isolation. But order matters: “The dog bites the man” and “The man bites the dog” have exactly the same tokens, and they mean opposite things.

Recurrent networks process text sequentially, so order is implicit in the temporal dynamics themselves. But transformers process every position in parallel, and attention by itself does not distinguish order — it has no notion of “first” or “second”. Without position information, the model would see every permutation of a sentence as identical.

The classical solution (GPT-1, GPT-2, BERT): add a position embedding to each token — input[t] = token_embedding[ID_t] + position_embedding[t] — where position_embedding is another learned table, one vector for each absolute position.

RoPE (Rotary Position Embeddings), the modern alternative: absolute position embeddings do not generalize well to sequences longer than those seen in training — if the model was trained with up to 512 tokens and receives a sequence of 1,000, the positions beyond 512 simply have no learned embedding. More recent models use RoPE: instead of adding a vector, it rotates the query and key vectors in each attention layer, with an angle proportional to the distance between tokens — making the comparison depend on relative position, not absolute.

Advantages of RoPE: it generalizes better to sequence lengths greater than those of training (the rotations are continuous functions of position), it adds no extra parameters (the rotation is deterministic, there is no table to learn), and it improves the handling of long-range dependencies.

q = q_projection(x)   # (length, head_dim)
k = k_projection(x)   # (length, head_dim)

q_rot = rotate(q, positions)   # rotates according to each token's absolute position
k_rot = rotate(k, positions)

# The dot product q_rot · k_rot will depend on the difference of positions,
# not on the absolute position of each one

Static vs. contextual embeddings

So far we have talked about static embeddings: each token has a single fixed vector, regardless of context. But that initial vector is only the starting point — as the word passes through the layers of the transformer, its representation gets enriched with the context of the words around it.

For example, bank has the same initial embedding in “bank of fish” and in “financial bank” — but after the first attention layer, its representation is already different depending on the context. That is why we say transformers produce contextual embeddings. In this chapter we focus on the initial embedding, the base everything else is built on — we will see contextual embeddings in the Anatomy of a transformer chapter.

Chapter summary

• Token IDs are turned into vectors through a trainable lookup table.
• Those vectors organize themselves into a semantic space where closeness reflects similarity of use.
• Similarity is measured with the cosine of the angle, which ignores vector length and looks only at direction.
• Embeddings enable semantic search, unlike BM25 (keyword-based) — both complement each other in hybrid systems, and they are the basis of RAG (Part II).
• Directions in that space can represent relationships (analogies), but they also encode biases from the training text.
• For the model to distinguish order, position embeddings are added — absolute, or rotary like RoPE.
• Initial embeddings are static, but they become contextual across the layers of the transformer.

In the next chapter we will see how the transformer uses these embeddings as input to the attention mechanism, which lets each token look at the others and build rich contextual representations.