Mini GPT Tokenizer Lab

Goal

Understand how Byte Pair Encoding (BPE) works by building a tiny tokenizer from scratch.

By the end, you should understand:

  • Why word-level tokenization is problematic
  • Why character-level tokenization is inefficient
  • How BPE merges frequent symbols
  • How GPT tokenizers compress text
  • Why byte-level tokenization can represent all languages

Step 1 - Character Tokenization

Start with a small corpus:

dog dog dog dog cat cat mouse

Convert each word into characters:

dog -> d o g
cat -> c a t

Questions:

  1. How many unique symbols exist?
  2. How many tokens are needed to represent the corpus?
  3. What are the most common character pairs?

Step 2 - Implement One BPE Merge

Count adjacent pairs:

d o
o g
c a
a t

Find the most frequent pair and merge it.

Example:

(d,o)

becomes:

do

New sequence:

do g

Questions:

  1. Did the number of tokens decrease?
  2. Which pair is now most common?

Step 3 - Repeat Merges

Run 10 merges.

Expected result:

dog
cat
mouse

become tokens.

Print after every iteration:

Iteration 1:
Most common pair = (d,o)

Iteration 2:
Most common pair = (do,g)

...

Observe how larger tokens emerge.

Step 4 - Encode New Words

Try encoding:

dogs
doggy
cats

Questions:

  1. Which parts become known tokens?
  2. Which parts remain split?

Example:

dogs

might become:

[dog, s]

This demonstrates subword tokenization.

Step 5 - Compare With Word Tokenization

Create:

dog
dogs
doggy
dogging

Word vocabulary:

4 tokens

BPE vocabulary:

dog
s
gy
ing

Questions:

  1. Which vocabulary is smaller?
  2. Which generalizes better?

Step 6 - Add Punctuation

Corpus:

dog
dog.
dog!
dog?

Run BPE again.

Observe:

dog.
dog!
dog?

may become separate tokens.

This demonstrates the problem discussed in the GPT-2 paper.

Step 7 - Prevent Bad Merges

Add a rule:

  • letters may merge with letters
  • punctuation may merge with punctuation
  • letters may NOT merge with punctuation

Compare results.

Questions:

  1. How many vocabulary slots are saved?
  2. Does token reuse improve?

Step 8 - Byte-Level Encoding

Choose a non-English word:

café

Convert to UTF-8 bytes:

"café".encode("utf-8")

Observe:

[99, 97, 102, 195, 169]

Try:

你好
مرحبا
😀

Questions:

  1. Can every symbol be represented?
  2. How many base symbols are required?

Answer:

256

Step 9 - Build a Tiny Tokenizer Class

Implement:

fit(text, num_merges)

encode(text)

decode(tokens)

Usage:

tokenizer.fit(corpus, 50)

tokens = tokenizer.encode(
    "the dog runs"
)

text = tokenizer.decode(tokens)

Verify:

decoded == original_text

Step 10 - Final Experiment

Train on:

The dog runs.
The dog sleeps.
The cat runs.
The cat sleeps.

Inspect learned tokens.

Expected tokens:

 the
 dog
 cat
 runs
 sleeps

Notice that common patterns become single tokens.

This is exactly the intuition behind GPT-style tokenization.

Stretch Goal

Build a tiny language model after the tokenizer.

Pipeline:

Raw Text
    ↓
Byte Encoding
    ↓
BPE Tokenizer
    ↓
Token IDs
    ↓
Next Token Prediction

At this point you’ll have recreated the first stage of GPT training on a miniature scale.

Comments