GeekCoding101 make your way to geek

Tokens in AI: How LLM Tokenization Works

What tokens are, how LLMs split text with word, character and subword (BPE) tokenization, and why tokens shape context windows, cost and speed.

8 min read
Daily Ai Insight Issue 6 20241207
On this pageTokens at a glance

A token is the small chunk of text an AI language model actually reads and writes. Depending on the tokenizer, it can be a whole word, part of a word, a single character or a punctuation mark. Before a model like ChatGPT can do anything with your prompt, a tokenizer chops the text into tokens and turns each one into a number. That’s why tokens decide how much text a model can handle at once, how much a request costs and how fast you get an answer.

Today’s topic might sound a bit technical, but don’t worry: we’re keeping it down-to-earth.

llm token limit
llm token limit

Tokens at a glance

QuestionShort answer
What is a token?The unit of text a model processes: a word, a piece of a word, a character or a symbol
What do most modern LLMs use?Subword tokenization, usually a form of Byte Pair Encoding (BPE)
How many words is a token?In English, very roughly ¾ of a word, or about 4 characters
Why should I care?Context windows, API pricing and response speed are all measured in tokens

1. What are tokens?

AI models don’t read language the way we do. They read tokens. A token is the smallest unit of text a given model works with.

Take this sentence:

“AI is incredibly smart.”

Depending on the tokenization method, it could be split like this (the splits below are illustrative; real tokenizers vary):

MethodTokens
Word-level["AI", "is", "incredibly", "smart", "."]
Character-level["A", "I", " ", "i", "s", " ", "i", "n", "c", "r", "e", "d", "i", "b", "l", "y", " ", "s", "m", "a", "r", "t", "."]
Subword-level (most common)["AI", " is", " incred", "ibly", " smart", "."]

Notice two things in the subword row:

  • Leading spaces travel with the word. Many tokenizers attach the space to the following token, so " is" and "is" can be different tokens.
  • Punctuation often gets its own token.

Without tokens, a model is like a brain without neurons: it has nothing to work with.

How many tokens is my text?

A common rule of thumb for English text:

  • 1 token ≈ 4 characters
  • 1 token ≈ ¾ of a word
  • 100 tokens ≈ 75 words

These are only rough averages. Code, unusual words, numbers and many non-English languages usually take more tokens per word. For an exact count, run your text through the tokenizer of the specific model you’re using.

2. Why are tokens so important?

Language models aren’t magical. They work by predicting the next token. You feed in a sequence of tokens, and the model “guesses” what comes next, one token at a time.

It’s like texting a friend “I’m feeling…” and they instantly reply “tired.” Is it empathy? Nope. It’s a guess based on everything they’ve seen before.

Why does AI need tokens?

Math can’t operate on words directly, so tokens are the bridge. Each token is mapped to a number (a token ID), and those numbers are what the model processes:

  • Input: “AI is amazing!”
  • Token IDs (illustrative): [1234, 5678, 91011]
  • Prediction: given [1234, 5678], the model predicts the next ID is 91011.

Inside the model, each ID is then turned into a vector (an embedding) that carries meaning. For more on that step, see The Building Blocks of LLMs: Vectors, Tokens and Embeddings.

3. How does AI tokenize? It’s not just random chopping

Tokenization isn’t smashing sentences with a hammer. There are three main families of methods, each with trade-offs.

(1) Word-based tokenization

This is the simplest approach: split on spaces (and usually punctuation).

  • Input: “AI is awesome.”
  • Tokens: ["AI", "is", "awesome", "."]
  • Pros: fast and easy to understand.
  • Cons:
    • The vocabulary explodes, because “run”, “runs”, “running” and “runner” are all separate entries.
    • Any word not in the vocabulary becomes “unknown”.
    • It struggles with languages that build long compound words, like German, or that don’t use spaces, like Chinese.

(2) Character-based tokenization

Every character is a token.

  • Input: “GPT”
  • Tokens: ["G", "P", "T"]
  • Pros: a tiny vocabulary, and it never meets an “unknown” word, which makes it robust to typos.
  • Cons: sequences get much longer, so the model needs far more steps and compute for the same text, and each single character carries little meaning on its own.

(3) Subword-based tokenization (the go-to approach)

This is what modern models like GPT and BERT use. Common words stay whole, while rare words are broken into familiar pieces.

  • Input: “awesomesauce”
  • Tokens (illustrative): ["awesome", "sauce"]

Even if the model has never seen “awesomesauce”, it knows both parts, so it can still make a sensible guess at the meaning. Subword tokenization sits in the sweet spot: a manageable vocabulary, short-enough sequences and no “unknown word” problem.

There are a few popular ways to build a subword vocabulary:

AlgorithmIdea in one lineKnown for
Byte Pair Encoding (BPE)Start from characters (or bytes) and repeatedly merge the most frequent adjacent pairGPT-family models (byte-level BPE)
WordPieceSimilar merging, but chooses merges that best improve the likelihood of the training dataBERT
Unigram / SentencePieceStart with a big vocabulary and prune pieces that matter least; works directly on raw text, including spacesMany multilingual models

How Byte Pair Encoding works, step by step

Despite the fancy name, BPE is just counting and merging:

  1. Start with every word split into single characters (or bytes).
  2. Count which adjacent pair appears most often across the training text, for example t + h.
  3. Merge that pair into a new token, th, and add it to the vocabulary.
  4. Repeat: maybe th + e → the comes next.
  5. Stop when the vocabulary reaches the chosen size.

A very frequent word like “the” ends up as a single token, while a rare word stays split into several pieces.

In short: tokenization is a carefully designed trade-off between precision and efficiency.

4. The real impact of tokens on AI

Tokens aren’t just jargon. They directly shape what a model can do for you.

(1) Context window

A model’s context window is the maximum number of tokens it can consider at once. It usually covers both your input and the model’s output. Early GPT-3-era models handled a few thousand tokens; newer models handle far more. Limits differ from model to model, so check the documentation of the one you’re using.

What does it mean in practice? If your input fits in the window, the model can “see” the whole document, such as a long contract, while answering. If it doesn’t fit, something has to be cut, truncated or summarized, and the model effectively forgets what you said earlier.

(2) Generation quality

Tokenization affects how well a model handles related words. Subword tokenization lets it see that “amazing” and “amazingly” share a piece, which helps it produce coherent text. A cruder tokenizer may treat them as unrelated.

(3) Cost and speed

Every token adds work:

  • Speed: longer prompts take longer to process, and every output token is generated one step at a time. A long answer simply takes longer to write.
  • Compute: in standard Transformers, attention cost grows roughly with the square of the sequence length. Doubling the input more than doubles that part of the work.
  • Money: many AI APIs price usage per token, often with separate rates for input and output tokens.

That’s why long inputs can feel like “computational fatigue”: more tokens, more processing.

5. The limitations of tokenization

Tokenization is essential, but it has quirks:

  • Semantic splitting: breaking “unbelievable” into ["un", "believ", "able"] makes statistical sense but can blur meaning. This is also why models sometimes stumble on spelling, letter counting or reversing words: they see chunks, not letters.
  • Language diversity: tokenizers trained mostly on English text tend to split other languages, such as Chinese, Arabic or Hindi, into more tokens for the same content. The result is higher cost and less fitting into the context window.
  • Numbers and code: digits and symbols can be split in inconsistent ways, which is one reason arithmetic can trip models up.
  • Overhead: long texts mean many tokens, which slows inference and increases compute demand.

6. One-line summary

Tokens are the building blocks of an AI model’s language understanding, and tokenization is the bridge that turns human language into math. Without tokens, an AI is just a heap of clueless parameters.

FAQ

How many words is 1,000 tokens? For typical English text, roughly 750 words. Code, rare words and many non-English languages use more tokens per word, so treat this as a ballpark figure.

Do spaces and punctuation count as tokens? Yes, they take up tokens too. Punctuation is often its own token, and spaces are usually merged into the token that follows them.

Is a token the same as a word? Not usually. Common short words are often one token, but longer or rarer words are split into several subword tokens.

Does every model use the same tokenizer? No. GPT-family models use byte-level BPE, BERT uses WordPiece, and many others use SentencePiece. The same sentence can produce different token counts on different models.

What is a context window in tokens? It’s the maximum number of tokens a model can consider in one go, usually including both your prompt and its reply. Anything beyond that limit gets cut off or must be shortened.

Final thoughts

AI may seem like magic, but it really comes down to details like these. Next time you use ChatGPT, try asking yourself: how many tokens did my question use? Did my document fit in the context window? These hidden mechanics play a big role in how accurate and useful the answer will be.

That’s it for today’s AI dissection! This post is part of the Daily AI Insight series: bite-sized explanations of the nuts and bolts of AI. Start with Why is the Transformer Model Called an “AI Revolution”?, then go deeper with The Building Blocks of LLMs: Vectors, Tokens and Embeddings.

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts