How Do Embeddings Work? – Training, Dimensions, and Distance
A mechanical walkthrough of embeddings: how encoders are trained, what pooling does, why dimensions are not interpretable, and how distance is computed.
An embedding model is a neural network trained so that related inputs land near each other. Understanding how that arrangement is learned explains both what embeddings capture and where they fail.
1. From Tokens to One Vector
- The input text is tokenised.
- Each token becomes a vector, refined through transformer layers that let tokens attend to one another.
- The resulting per-token vectors are pooled into a single vector — by averaging, or by taking a designated summary token.
- The result is usually normalised to unit length, so comparison depends on direction alone.
Pooling is why a long document embeds poorly: averaging many topics produces a vector near the centre of the space, close to everything and specific to nothing.
2. How the Arrangement Is Learned
Training uses pairs and triplets rather than labels.
- Positive pairs — a question and its answer, a title and its body, a sentence and its paraphrase.
- Negatives — unrelated texts, ideally hard ones that look similar but are not.
- Objective — pull positives together, push negatives apart.
Everything an embedding "knows" comes from which pairs were treated as related during training. That is why domain-specific models outperform general ones on specialised corpora.
3. Dimensions Are Not Features
No single dimension corresponds to a concept. Meaning is distributed across all of them, and only relationships between whole vectors carry information.
More dimensions can capture finer distinctions and cost more to store and search, with gains flattening quickly. The right size is an empirical question for the corpus, not a maxim.
4. Measuring Distance
| Measure | Formula idea | Notes |
|---|---|---|
| Cosine similarity | Angle between vectors | The standard for text; ignores length |
| Dot product | Projection | Equals cosine when vectors are normalised |
| Euclidean | Straight-line distance | Sensitive to magnitude; less common for text |
With normalised vectors these largely coincide, which is why most stores default to cosine and most models normalise their output.
5. Why the Same Model Is Required
Each model arranges its space differently. Two models can produce vectors of identical length whose coordinates mean entirely different things, so a query embedded by one cannot be compared against documents embedded by another.
Changing the model therefore means re-embedding the whole corpus — treat it as part of the index's schema.
6. What This Explains
- Negation fails because "supports X" and "does not support X" appear in nearly identical contexts during training.
- Rare identifiers fail because they were too infrequent to acquire a well-placed vector.
- Paraphrase works because paraphrases were explicit positive pairs.
- Long inputs blur because pooling averages them.
- Domain models win because their positive pairs came from that domain.
7. Practical Consequences
Embed passages rather than documents, keep them topically coherent, pair vector search with lexical matching for exact tokens, and evaluate candidate models on your own queries rather than on public leaderboards.
8. Summary
An embedding model encodes text through transformer layers, pools the result into one vector, and is trained so related pairs sit close together in the resulting space.
That training procedure explains the strengths and the blind spots: paraphrase is captured because it was taught, negation and rare tokens are missed because nothing taught them.