GeekCoding101 make your way to geek

What Is an Embedding? Text to Vectors, Explained

An embedding turns text into vectors so AI can compare meaning. How Word2Vec, GloVe and BERT build them, and how dot product and cosine similarity work.

7 min read
Daily Ai Insight Issue 7 20241208
On this pageOn this page

An embedding is a list of numbers (a vector) that represents a word, sentence or other piece of data so an AI model can work with it. Things with similar meaning get vectors that sit close together, so the model can compare meaning with plain math.

AI doesn’t comprehend words, sentences or syntax. It only works with numbers. Embeddings are the “translator” between the two.

On this page

  1. What is an embedding?
  2. Why do we need embeddings?
  3. A hands-on example: comparing vectors
  4. How are embeddings created?
  5. Static vs contextual embeddings
  6. Where embeddings are used
  7. How dot products relate to embeddings
  8. Challenges of embeddings
  9. Visualization resources
  10. One-line summary
  11. FAQ

1. What Is an Embedding?

Think of an embedding as a language map. Each word is a point on the map, and its position reflects its relationship with other words:

  • “cat” and “dog” sit close together.
  • “cat” and “car” sit far apart, even though they’re spelled almost the same.

An embedding assigns a numerical representation (a vector) to a word, phrase or sentence. The model never sees the letters “c-a-t”, only the vector.

2. Why Do We Need Embeddings?

Human language is rich and abstract, but models need something mathematical to work with. Embeddings solve three problems.

(1) Vectorizing language

Words become vectors, which are lists of numbers. With toy 3-dimensional values:

  • “cat” → [0.1, 0.3, 0.5]
  • “dog” → [0.1, 0.32, 0.51]

Once words are vectors, a model can compare, cluster and predict relationships between them.

(2) Capturing semantic relationships

The real power of embeddings is that directions in the space can carry meaning. The classic Word2Vec example:

king − man + woman ≈ queen

Subtracting “man” from “king” and adding “woman” lands near “queen.” The relationship “royal + gender” is encoded as geometry.

(3) Addressing data sparsity

Without embeddings, you could give every word its own index, a one-hot vector. It is as long as the whole vocabulary, all zeros except a single 1. These vectors are huge, mostly empty, and every word is equally distant from every other word.

Embeddings compress words into a fixed, much smaller number of dense dimensions (classic word embeddings often use 100 or 300). That is far more efficient, and it lets similarity show up as distance.

One-hot vectorEmbedding
LengthSize of the vocabularyFixed and small (e.g. 100–300 for classic word embeddings)
ValuesAll 0 except one 1Real numbers, mostly non-zero
Similar words close together?No, every pair is equally farYes
Learned from data?NoYes

3. A Hands-On Example: Comparing Vectors

Here are the toy “cat” and “dog” vectors, plus a made-up “car” vector, compared on your own machine. The numbers are invented for illustration; real embeddings have many more dimensions.

import numpy as np
cat = np.array([0.1, 0.3, 0.5])
dog = np.array([0.1, 0.32, 0.51])
car = np.array([0.9, -0.2, 0.1])
def cosine(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print("dot(cat, dog): ", round(np.dot(cat, dog), 3)) # 0.361
print("dot(cat, car): ", round(np.dot(cat, car), 3)) # 0.08
print("cosine(cat, dog):", round(cosine(cat, dog), 3)) # ~1.0
print("cosine(cat, car):", round(cosine(cat, car), 3)) # ~0.146

“cat” and “dog” point in almost the same direction (cosine ≈ 1). “cat” and “car” barely overlap. That is the language map, expressed as math.

4. How Are Embeddings Created?

Embeddings are learned by machine learning models trained on large amounts of text. Three well-known approaches:

(1) Word2Vec

Word2Vec is one of the earliest and most successful methods. It is built on the idea that similar words appear in similar contexts.

  • Sentences: “Cats love milk” and “Dogs love bones”
  • “cats” and “dogs” share similar surroundings, so Word2Vec places them close together.

(2) GloVe

GloVe (Global Vectors for Word Representation) learns from co-occurrence statistics across a whole corpus.

  • “apple” and “orange” often co-occur with “fruit,” and that shared pattern is encoded in their vectors.

(3) Transformer models (e.g. BERT, GPT)

Modern models create embeddings dynamically, based on context.

  • “bank” in “river bank” and “bank” in “money in the bank” get different embeddings, which lets the model tell the two meanings apart.

5. Static vs Contextual Embeddings

Static (Word2Vec, GloVe)Contextual (BERT, GPT-style Transformers)
One vector per word?Yes, the same everywhereNo, it changes with the surrounding text
Handles “bank” (river vs money)?No, both meanings share one vectorYes
Typical useLightweight similarity, classic NLPModern language models, semantic search
CostCheap lookupRequires running the model

6. Where Embeddings Are Used

Queries and documents are both turned into embeddings. The system then measures similarity (dot product or cosine) to return the closest matches, even when they don’t share exact keywords. The same idea powers retrieval in retrieval-augmented generation (RAG), where relevant passages are found by vector similarity and handed to an LLM.

(2) Recommendation systems

Users and items such as videos, movies and products can be represented as vectors. The system recommends items whose vectors are close to a user’s preferences.

(3) Generative AI

Models like ChatGPT start by turning every token you type into an embedding. Image models like DALL-E use embeddings to connect text prompts with images.

(4) Clustering and classification

Grouping similar support tickets, detecting duplicate questions, or labeling text all become distance problems once the text is embedded.

7. How Dot Products Relate to Embeddings

The dot product is the core operation for comparing vectors: multiply matching positions and add them up.

(1) Similarity measurement

In search and recommendation, a higher dot product between two vectors (say, a query and a document) means they’re more similar.

One catch: the dot product also grows with vector length. Cosine similarity divides by both lengths, so it only measures direction:

MeasureFormulaRangeAffected by vector length?
Dot producta · bUnboundedYes
Cosine similarity(a · b) / (‖a‖ ‖b‖)−1 to 1No

If vectors are normalized to length 1, the dot product and cosine similarity are the same number.

(2) Attention mechanism

In Transformers, each token produces a query vector and a key vector. Dot products between queries and keys produce attention scores, which decide which parts of the input matter most for each token. Embeddings and dot products work together at the heart of every Transformer layer.

8. Challenges of Embeddings

  • Data dependency: Embedding quality depends heavily on training data. Biased data produces biased embeddings.
  • Dimensional trade-offs: High-dimensional embeddings cost more to compute and store. Low-dimensional ones may lose information.
  • Semantic ambiguity: Even advanced embeddings can struggle with nuance, sarcasm and metaphor.
  • Model lock-in: Vectors from different embedding models live in different spaces. You can’t compare a vector from one model with a vector from another.

9. Visualization Resources

Seeing an embedding space makes it click. Try these searches:

  • Embedding space: search embedding space visualization or word embedding map. These show words plotted in 2D or 3D, clustered by meaning.
  • Dot product similarity: search dot product similarity visualization or cosine similarity embedding. These show how vectors are compared mathematically.
  • Attention mechanisms: search transformer attention scores or attention mechanism visualization. These show how embeddings and dot products work together inside Transformers.
  • Interactive projector: TensorFlow’s Embedding Projector lets you explore real word embeddings in your browser.

For a gentler, more math-focused follow-up, I recommend An Intuitive 101 Guide to Vector Embeddings.

10. One-Line Summary

Embeddings are the bridge between human language and machine understanding: they map words and sentences into a mathematical space where similar meanings sit close together.

FAQ

What is an embedding in simple terms? It’s a list of numbers that stands in for a word, sentence or item. Similar things get similar numbers, so a computer can measure meaning as distance.

What’s the difference between an embedding and a vector? A vector is just a list of numbers. An embedding is a vector that a model has learned so that its position carries meaning.

Should I use dot product or cosine similarity? Cosine similarity ignores vector length and compares only direction, which is usually what you want for meaning. If your vectors are already normalized, the two give the same result.

How many dimensions does an embedding have? It depends on the model. Classic word embeddings often use 100 or 300 dimensions, while Transformer models typically use hundreds or more (BERT-base uses 768).

Are ChatGPT’s embeddings the same as Word2Vec? No. Word2Vec gives each word one fixed vector. Transformer models like the ones behind ChatGPT produce contextual embeddings that change depending on the surrounding text.

Final Thoughts

Embeddings might seem like a dry technical concept, but they’re the unsung heroes behind AI’s ability to generate text, find documents and recommend content. Next time you type into ChatGPT, remember that every word you enter is turned into a dense vector before the model does anything else. Behind the magic is a lot of math, and now you’ve seen a little of it.

Let’s keep breaking down AI one piece at a time.

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts