Embeddings

Take the word "dog." By itself, it's just a sequence of characters. But what does it mean? How does an AI system understand it's similar to "puppy" but different from "elephant"? Embeddings are how.

An embedding is basically a list of numbers, typically between 300 and 3,072 values long (depending on the embedding model), that represents the semantic meaning of a piece of text. The embedding model is trained so that similar meanings produce similar numbers. If you embed "dog," "puppy," and "canine," they cluster together in high-dimensional space. If you embed "dog," "apple," and "Tuesday," they're far apart. The distance between embeddings corresponds to semantic similarity.

This is wild when you first think about it. Somehow a neural network trained on massive text corpora learned to compress meaning into numerical form. The distances between embeddings correlate with how a human would judge meaning. "King" minus "man" plus "woman" is close to "queen." This isn't programmed logic. It's what the model learned from billions of examples.

Embeddings are the foundation of semantic search, the retrieval piece of RAG, similarity clustering, duplicate detection, anomaly detection, and a hundred other tasks. Want to find documents related to a user query? Embed both, calculate distances, boom. Similarity. Want to group similar customer support tickets? Embed them all, cluster, watch patterns emerge. Want a chatbot that understands when two sentences mean the same thing even with different wording? Embeddings.

The embedding space itself becomes your data structure. You're not searching documents with keywords anymore. You're searching a geometric space. Document databases become vector databases where each document is a point in space, and queries become navigation problems. Find me the nearest neighbors. This is conceptually simpler than traditional search for many applications, though definitely not without tradeoffs.

Different embedding models produce different embeddings. OpenAI's text-embedding-3-small. Google's embedding models. Open source options like all-MiniLM-L6-v2. They vary in dimension (smaller is cheaper to store and search), training approach, and what domains they're optimized for. A financial embedding model trained on earnings reports and SEC filings outperforms a general-purpose model on finance questions. Domain-specific embedding matters. We've seen teams spend weeks debugging RAG systems only to discover the embedding model was the bottleneck because it wasn't trained on their domain.

Embeddings get expensive at scale. A large knowledge base means embedding every single document and storing all those vectors. A semantic search system needs to compute embeddings for queries in real time. The cost of embedding APIs adds up. Open source embedding models are free to run but require infrastructure. The tradeoff between cost, quality, and latency forces tough decisions.

Embeddings also have failure modes nobody talks about until they hit them. Adversarial text can produce embeddings that don't reflect actual meaning. Homographs (words with multiple meanings) get a single embedding. Sarcasm and context-dependent meaning don't map cleanly to vector space. Embeddings capture statistical patterns but not true understanding. They work great for similarity tasks but poorly for interpretability. You can't easily ask "why did this embedding match that query?" The space is too high-dimensional for human intuition.