Tokenizer Lab
Learn Byte-Pair Encoding from scratch
Training corpus
Paste a short sample of text. The computer will learn which letter chunks appear together most often.
Training steps
Step 1 of 0
Starting point: every letter is its own chunk. The computer will now look for the most common pair of neighbors.
Train a tokenizer to see merge steps here.
Try it
Type any word or sentence to see how the trained tokenizer splits it.
Train a tokenizer first to enable live tokenization.
Why can't GPT count the r's in "strawberry"?
This example trains on a small corpus (including "straw", "strawber", "strawberries") but not the full word "strawberry" — then tokenizes it.
"strawberry" →
Because "strawberry" was not common in the training text, the computer never learned it as one chunk. It gets split into weird pieces — and the model cannot easily count letters inside a chunk it never saw letter-by-letter.