A token is the small chunk of text an AI language model actually reads and writes. Depending on the tokenizer, it can be a whole word, part of a word, a single character or a punctuation mark. Before a model like ChatGPT can do anything with your prompt, a tokenizer chops the text into tokens and turns each one into a number. That’s why tokens decide how much text a model can handle at once, how much a request costs and how fast you get an answer.
Today’s topic might sound a bit technical, but don’t worry: we’re keeping it down-to-earth.

Tokens at a glance
| Question | Short answer |
|---|---|
| What is a token? | The unit of text a model processes: a word, a piece of a word, a character or a symbol |
| What do most modern LLMs use? | Subword tokenization, usually a form of Byte Pair Encoding (BPE) |
| How many words is a token? | In English, very roughly ¾ of a word, or about 4 characters |
| Why should I care? | Context windows, API pricing and response speed are all measured in tokens |
1. What are tokens?
AI models don’t read language the way we do. They read tokens. A token is the smallest unit of text a given model works with.
Take this sentence:
“AI is incredibly smart.”
Depending on the tokenization method, it could be split like this (the splits below are illustrative; real tokenizers vary):
| Method | Tokens |
|---|---|
| Word-level | ["AI", "is", "incredibly", "smart", "."] |
| Character-level | ["A", "I", " ", "i", "s", " ", "i", "n", "c", "r", "e", "d", "i", "b", "l", "y", " ", "s", "m", "a", "r", "t", "."] |
| Subword-level (most common) | ["AI", " is", " incred", "ibly", " smart", "."] |
Notice two things in the subword row:
- Leading spaces travel with the word. Many tokenizers attach the space to the following token, so
" is"and"is"can be different tokens. - Punctuation often gets its own token.
Without tokens, a model is like a brain without neurons: it has nothing to work with.
How many tokens is my text?
A common rule of thumb for English text:
- 1 token ≈ 4 characters
- 1 token ≈ ¾ of a word
- 100 tokens ≈ 75 words
These are only rough averages. Code, unusual words, numbers and many non-English languages usually take more tokens per word. For an exact count, run your text through the tokenizer of the specific model you’re using.
2. Why are tokens so important?
Language models aren’t magical. They work by predicting the next token. You feed in a sequence of tokens, and the model “guesses” what comes next, one token at a time.
It’s like texting a friend “I’m feeling…” and they instantly reply “tired.” Is it empathy? Nope. It’s a guess based on everything they’ve seen before.
Why does AI need tokens?
Math can’t operate on words directly, so tokens are the bridge. Each token is mapped to a number (a token ID), and those numbers are what the model processes:
- Input: “AI is amazing!”
- Token IDs (illustrative):
[1234, 5678, 91011] - Prediction: given
[1234, 5678], the model predicts the next ID is91011.
Inside the model, each ID is then turned into a vector (an embedding) that carries meaning. For more on that step, see The Building Blocks of LLMs: Vectors, Tokens and Embeddings.
3. How does AI tokenize? It’s not just random chopping
Tokenization isn’t smashing sentences with a hammer. There are three main families of methods, each with trade-offs.
(1) Word-based tokenization
This is the simplest approach: split on spaces (and usually punctuation).
- Input: “AI is awesome.”
- Tokens:
["AI", "is", "awesome", "."] - Pros: fast and easy to understand.
- Cons:
- The vocabulary explodes, because “run”, “runs”, “running” and “runner” are all separate entries.
- Any word not in the vocabulary becomes “unknown”.
- It struggles with languages that build long compound words, like German, or that don’t use spaces, like Chinese.
(2) Character-based tokenization
Every character is a token.
- Input: “GPT”
- Tokens:
["G", "P", "T"] - Pros: a tiny vocabulary, and it never meets an “unknown” word, which makes it robust to typos.
- Cons: sequences get much longer, so the model needs far more steps and compute for the same text, and each single character carries little meaning on its own.
(3) Subword-based tokenization (the go-to approach)
This is what modern models like GPT and BERT use. Common words stay whole, while rare words are broken into familiar pieces.
- Input: “awesomesauce”
- Tokens (illustrative):
["awesome", "sauce"]
Even if the model has never seen “awesomesauce”, it knows both parts, so it can still make a sensible guess at the meaning. Subword tokenization sits in the sweet spot: a manageable vocabulary, short-enough sequences and no “unknown word” problem.
There are a few popular ways to build a subword vocabulary:
| Algorithm | Idea in one line | Known for |
|---|---|---|
| Byte Pair Encoding (BPE) | Start from characters (or bytes) and repeatedly merge the most frequent adjacent pair | GPT-family models (byte-level BPE) |
| WordPiece | Similar merging, but chooses merges that best improve the likelihood of the training data | BERT |
| Unigram / SentencePiece | Start with a big vocabulary and prune pieces that matter least; works directly on raw text, including spaces | Many multilingual models |
How Byte Pair Encoding works, step by step
Despite the fancy name, BPE is just counting and merging:
- Start with every word split into single characters (or bytes).
- Count which adjacent pair appears most often across the training text, for example
t+h. - Merge that pair into a new token,
th, and add it to the vocabulary. - Repeat: maybe
th+e→thecomes next. - Stop when the vocabulary reaches the chosen size.
A very frequent word like “the” ends up as a single token, while a rare word stays split into several pieces.
In short: tokenization is a carefully designed trade-off between precision and efficiency.
4. The real impact of tokens on AI
Tokens aren’t just jargon. They directly shape what a model can do for you.
(1) Context window
A model’s context window is the maximum number of tokens it can consider at once. It usually covers both your input and the model’s output. Early GPT-3-era models handled a few thousand tokens; newer models handle far more. Limits differ from model to model, so check the documentation of the one you’re using.
What does it mean in practice? If your input fits in the window, the model can “see” the whole document, such as a long contract, while answering. If it doesn’t fit, something has to be cut, truncated or summarized, and the model effectively forgets what you said earlier.
(2) Generation quality
Tokenization affects how well a model handles related words. Subword tokenization lets it see that “amazing” and “amazingly” share a piece, which helps it produce coherent text. A cruder tokenizer may treat them as unrelated.
(3) Cost and speed
Every token adds work:
- Speed: longer prompts take longer to process, and every output token is generated one step at a time. A long answer simply takes longer to write.
- Compute: in standard Transformers, attention cost grows roughly with the square of the sequence length. Doubling the input more than doubles that part of the work.
- Money: many AI APIs price usage per token, often with separate rates for input and output tokens.
That’s why long inputs can feel like “computational fatigue”: more tokens, more processing.
5. The limitations of tokenization
Tokenization is essential, but it has quirks:
- Semantic splitting: breaking “unbelievable” into
["un", "believ", "able"]makes statistical sense but can blur meaning. This is also why models sometimes stumble on spelling, letter counting or reversing words: they see chunks, not letters. - Language diversity: tokenizers trained mostly on English text tend to split other languages, such as Chinese, Arabic or Hindi, into more tokens for the same content. The result is higher cost and less fitting into the context window.
- Numbers and code: digits and symbols can be split in inconsistent ways, which is one reason arithmetic can trip models up.
- Overhead: long texts mean many tokens, which slows inference and increases compute demand.
6. One-line summary
Tokens are the building blocks of an AI model’s language understanding, and tokenization is the bridge that turns human language into math. Without tokens, an AI is just a heap of clueless parameters.
FAQ
How many words is 1,000 tokens? For typical English text, roughly 750 words. Code, rare words and many non-English languages use more tokens per word, so treat this as a ballpark figure.
Do spaces and punctuation count as tokens? Yes, they take up tokens too. Punctuation is often its own token, and spaces are usually merged into the token that follows them.
Is a token the same as a word? Not usually. Common short words are often one token, but longer or rarer words are split into several subword tokens.
Does every model use the same tokenizer? No. GPT-family models use byte-level BPE, BERT uses WordPiece, and many others use SentencePiece. The same sentence can produce different token counts on different models.
What is a context window in tokens? It’s the maximum number of tokens a model can consider in one go, usually including both your prompt and its reply. Anything beyond that limit gets cut off or must be shortened.
Final thoughts
AI may seem like magic, but it really comes down to details like these. Next time you use ChatGPT, try asking yourself: how many tokens did my question use? Did my document fit in the context window? These hidden mechanics play a big role in how accurate and useful the answer will be.
That’s it for today’s AI dissection! This post is part of the Daily AI Insight series: bite-sized explanations of the nuts and bolts of AI. Start with Why is the Transformer Model Called an “AI Revolution”?, then go deeper with The Building Blocks of LLMs: Vectors, Tokens and Embeddings.