Embeddings Explained
What a vector actually is, how cosine similarity works, how to choose an embedding model, and the mistakes that silently destroy retrieval
Embeddings Explained
TL;DR
An embedding is a list of numbers representing the meaning of a piece of text. Texts about similar things get similar numbers. That one property is what lets a computer search by meaning rather than by matching words. You turn both the question and the documents into numbers, then find the numbers pointing in the same direction.
| Property | Value |
|---|---|
| Level | Beginner → Advanced |
| Reading time | ~25 minutes |
| Prerequisites | Chunking Strategies |
| You will understand | Vectors, cosine similarity, model selection, and the errors that break retrieval silently |
Starting From Zero
Computers compare numbers, not meaning. So the task is to convert text into numbers such that similar meanings produce similar numbers.
Imagine describing foods with just two numbers: sweetness and crunchiness.
| Food | Sweetness | Crunchiness |
|---|---|---|
| Apple | 0.8 | 0.9 |
| Pear | 0.7 | 0.6 |
| Carrot | 0.3 | 0.95 |
| Honey | 1.0 | 0.0 |
Apple and pear have similar numbers, and they are similar foods. Honey and carrot are far apart in these numbers, and far apart as foods. You could now answer "what is most like an apple?" with arithmetic — no understanding of fruit required.
An embedding does exactly this for text, with two changes: the dimensions are learned rather than hand-designed, and there are hundreds or thousands of them instead of two.
Nobody decided what dimension 47 of an embedding means, and it usually has no clean human label. The model learned, from enormous quantities of text, whatever set of dimensions best separates meanings. The dimensions are not interpretable; the distances between points are.
Vocabulary
| Term | Meaning |
|---|---|
| Vector | An ordered list of numbers: [0.021, -0.118, 0.077, …] |
| Embedding | A vector that represents meaning. Used interchangeably with "vector" in RAG |
| Dimensions | How many numbers are in the vector — 384, 768, 1536, 3072 are common |
| Vector space | The imaginary space all your vectors live in |
| Cosine similarity | A measure of the angle between two vectors: 1 = same direction, 0 = unrelated, −1 = opposite |
| Dot product | Multiply matching positions and sum. Equals cosine similarity when vectors are normalised |
| Euclidean distance | Straight-line distance between two points |
| Normalised | Scaled to length 1, so only direction carries meaning |
| Bi-encoder | Embeds query and document separately. Fast, pre-computable — this is what search uses |
| Cross-encoder | Reads query and document together. Accurate, slow — this is what rerankers use |
See It Working
Pick a question and watch the shapes. Every sentence sits at a fixed point; the closer the angle to the query, the higher it ranks.
Embedding similarity playground
Pick a question. Every sentence sits at a fixed point in meaning-space; the closer the angle, the higher the cosine similarity, and the higher it ranks. No keywords are compared at any point.
Ibuprofen can increase the effect of blood thinners.
0.997Warfarin dosage should be adjusted based on INR results.
0.996Take one tablet twice daily with food.
0.967Unexplained bruising may indicate a clotting problem.
0.031The patient reported chest pain and shortness of breath.
-0.227The clinic is open from 9am to 5pm on weekdays.
-0.267Call reception to reschedule your appointment.
-0.375Parking is free for patients in the north lot.
-0.490Notice that raising k past the genuinely relevant chunks starts pulling in unrelated topics. Those extra chunks still get sent to the model, where they cost tokens and can distract it.
Notice that the medication query ranks medication sentences highly without sharing keywords with all of them. That is the whole point of embeddings: "blood thinner" and "warfarin" land near each other because they mean related things, not because they look alike.
Measuring Similarity
Cosine Similarity — the One You Will Use
Cosine similarity measures the angle between two vectors, ignoring their length.
cosine(A, B) = (A · B) / (|A| × |B|)
where A · B = a₁b₁ + a₂b₂ + … + aₙbₙ (the dot product)
|A| = √(a₁² + a₂² + … + aₙ²) (the length)| Value | Interpretation |
|---|---|
| 1.0 | Identical direction — the same meaning |
| 0.8–0.95 | Strongly related. Typical for a good retrieval match |
| 0.5–0.8 | Loosely related, same general topic |
| 0.0 | Unrelated — perpendicular |
| Negative | Opposing direction. Rare in practice with modern text models |
Why angle and not distance? Because length tends to encode length, not meaning. A long document and a short sentence about the same subject point in a similar direction but can differ in magnitude. Ignoring magnitude focuses the comparison on topic.
The widget below makes this concrete. Real embeddings have hundreds of dimensions and cannot be drawn, but the maths is identical in 2 dimensions — so here it is, drawn.
Cosine similarity is just an angle
Comparing
dog vs puppy
Almost the same direction — near-identical meaning
Notice that only direction matters, never length. That is the whole reason cosine similarity is used for text: a long document and a short sentence about the same topic point the same way, so they score as similar. Notice too that “invoice” and “dog” score close to -1. Real embedding models rarely produce strong negatives like this — most unrelated text lands near 0 — but the geometry is the same.
Similarity scores are not comparable across models or corpora. One model may call a good match 0.85; another may call the same pair 0.42. There is no universal "good score" threshold. Always calibrate against your own data by examining real matched pairs — never hard-code a threshold you read in a blog post.
The Three Metrics Compared
| Metric | Measures | Use when |
|---|---|---|
| Cosine | Angle | Almost always for text. The default |
| Dot product | Angle × magnitudes | When vectors are already normalised — then it equals cosine, and is faster |
| Euclidean (L2) | Straight-line distance | Rarely for text. Common in image and geometric applications |
Most modern embedding models output normalised vectors (length 1). When they do, cosine similarity and dot product are mathematically identical, and dot product is cheaper to compute — which is why many vector databases default to it.
Choosing an Embedding Model
The Dimensions Question
| Dimensions | Storage per million vectors | Trade-off |
|---|---|---|
| 384 | ~1.5 GB | Fast, cheap, surprisingly capable |
| 768 | ~3 GB | Common balance point |
| 1536 | ~6 GB | Strong quality, standard for hosted models |
| 3072 | ~12 GB | Marginal gains over 1536 for most corpora; 2× the cost |
More dimensions is not reliably better. Beyond a point, gains flatten while storage, memory, and search latency keep climbing linearly. Many teams pay for 3072 dimensions and measure no improvement over 1536 on their own data.
Matryoshka Embeddings
Some models are trained so that truncating the vector still works. A 3072-dimension vector can be cut to its first 512 numbers and remain useful, because the model was trained to pack the most important information into the earliest dimensions — like nested Russian dolls.
This enables a powerful two-stage pattern: search with truncated vectors for speed, then re-score the survivors with the full vectors for accuracy.
Selection Criteria
| Criterion | Question to ask |
|---|---|
| Quality | How does it rank on retrieval benchmarks — and more importantly, on your data? |
| Max input length | Will your chunks be silently truncated? Check this explicitly |
| Dimensions | What are the storage and latency consequences at your scale? |
| Hosted or local | Is sending your data to a third-party API acceptable? |
| Cost | Per-token API cost, or GPU cost if self-hosted |
| Language | Does it handle your languages? Many models are English-first |
| Domain | Medical, legal, and code text often benefit from specialised models |
| Licence | Can you use it commercially? |
On benchmarks
Public retrieval leaderboards are a reasonable starting filter and a poor final answer. They measure performance on generic corpora, and models can be tuned to score well on them. Build a small evaluation set from your own documents and real questions — 50 examples is enough to be informative — and test the top three candidates on it. That measurement beats any leaderboard.
The Mistakes That Break Retrieval Silently
These are the ones that produce no error message and quietly ruin quality.
Silent embedding failures
Using different models for indexing and querying
The catastrophic one. Different models produce coordinates in incompatible spaces. The arithmetic still runs and still returns numbers, so nothing crashes — the results are simply noise. If you change embedding models, you must re-embed the entire corpus. Store the model name and version in your index metadata so this is detectable.
Silent truncation of long chunks
Many libraries truncate input over the model's limit without warning. A 1,000-token chunk given to a 512-token model becomes an embedding of only its first half. The second half is invisible to search forever. Check your chunk token counts against the model limit explicitly.
Ignoring asymmetry between questions and documents
A question ("what is the maximum dose?") and its answer ("the maximum dose is 10 mg daily") are different kinds of text. Some models offer separate query and document prefixes — often literally query: and passage: — and skipping them measurably reduces quality. Read the model card.
Over-normalising the text first
Lowercasing, stripping punctuation, and removing stop words are habits from keyword-search systems. Embedding models are trained on natural text and use those signals. Feed them clean natural text, not stemmed tokens.
Assuming embeddings handle exact identifiers
They do not. Order numbers, error codes, part numbers, and rare proper nouns embed poorly because the model has little or no training signal for them. This is not a fixable weakness — it is a reason to run keyword search alongside vector search.
What Embeddings Are Bad At
Knowing the failure modes tells you exactly when to add other retrieval methods.
| Weakness | Example | Mitigation |
|---|---|---|
| Exact identifiers | "Invoice INV-2024-8871" | Keyword search |
| Rare proper nouns | An unusual surname or internal project name | Keyword search |
| Negation | "drugs not safe with warfarin" often embeds close to "drugs safe with warfarin" | Reranking; explicit query handling |
| Numeric comparison | "patients over 65" vs "patients over 75" look very similar | Metadata filters on structured fields |
| Very long text | One vector cannot represent a whole document faithfully | Chunking |
| Fresh jargon | Terms coined after the model was trained | Fine-tune, or rely on keyword search |
| Structure and order | "A causes B" vs "B causes A" can embed similarly | Reranking |
Negation is the one that surprises people most. To an embedding model, "safe with warfarin" and "not safe with warfarin" share nearly all their content words and land close together in vector space. In a clinical or legal setting that is a serious failure mode, and it is a strong argument for always reranking.
Domain Adaptation
If generic models underperform on your specialised corpus, three options in ascending cost:
| Option | Effort | When it pays off |
|---|---|---|
| Use a domain model | Low | A published biomedical, legal, or code model exists for your field |
| Fine-tune on your pairs | Medium | You have a few thousand real query-document pairs, e.g. from search logs |
| Train from scratch | Very high | Almost never justified |
Fine-tuning an embedding model is one of the more reliable quality upgrades available if you have real usage data — teaching the model that your users' phrasing maps to your documents' phrasing. The Embedding Fine-Tuning project walks through it.
Concept Checks
Check yourself
You swapped embedding models and only re-embedded new documents. What happens?
Retrieval quality collapses, with no error raised anywhere. Old chunks live in the old model's coordinate space and new chunks in the new one; queries are embedded by the new model, so similarity against every old chunk is meaningless noise. The fix is a full re-embed of the corpus, and the prevention is storing the model name and version in your index metadata.
Why does 'drugs not safe with warfarin' retrieve the opposite of what was asked?
Because embeddings capture topical content much more strongly than logical operators. "Not" is a single short function word among a dozen content words, so both phrases land in nearly the same region of vector space. Vector search alone cannot reliably distinguish them — you need a reranker, which reads query and chunk together and can weigh the negation.
Your team wants to move from 1536 to 3072 dimensions for better quality. Your response?
Measure first on your own evaluation set. Doubling dimensions doubles storage and memory and increases search latency for gains that are frequently negligible past 1536. If the measured improvement is within noise, the money and latency are better spent on hybrid search or reranking.
A similarity score of 0.62 — is that a good match?
Unanswerable without context. Score scales differ by model and by corpus; 0.62 might be an excellent match for one model and near-random for another. Calibrate by taking real query-document pairs you know are good and bad, and observing where the scores actually fall for your setup.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Embedding | Numbers representing meaning; similar meanings get similar numbers |
| Dimensions | 384–3072; more is not reliably better |
| Cosine similarity | Angle between vectors; the default text metric |
| Dot product | Equals cosine when vectors are normalised, and is faster |
| Bi-encoder | Embeds separately; fast; used for search |
| Cross-encoder | Reads together; accurate; used for reranking |
| Matryoshka | Truncatable vectors — search short, verify long |
| Same model rule | Index and query must use the identical model, always |
| Silent truncation | Chunks over the model's limit lose their tails without warning |
| Embedding blind spots | IDs, rare names, negation, numeric comparison — use keyword search and reranking |
Next
You have vectors. Now store them somewhere that can search millions of them in milliseconds: Vector Databases & Indexes.
Chunking Strategies
How to split documents — fixed, recursive, semantic, parent-document, and late chunking — and why boundaries decide what your system can ever answer
Vector Databases & Indexes
How approximate nearest neighbour search works — HNSW, IVF, quantization — plus metadata filtering, and how to choose a database