Tokenization, Transformers and Context Windows: A Practical Mental Model

You don't need to derive backpropagation to build on LLMs. You do need an accurate picture of what happens between sending a string and getting a string back. That picture explains your bills, your latency, your truncation bugs and many odd model failures. This section builds it using concepts from Chapters 1 and 2 of the Hugging Face LLM course.

The pipeline in one line:

code

text -> tokenizer -> token IDs -> transformer -> scores for every possible next token -> pick one -> repeat

1. Tokenization: the model never sees your text

An LLM works on integer token IDs, not characters or words. A tokenizer converts text into IDs from a fixed vocabulary, typically 30k to 250k entries, and converts IDs back into text.

Modern tokenizers use subword algorithms: BPE (GPT family, Llama), WordPiece (BERT) and Unigram via SentencePiece (T5 and many multilingual models). Common words become a single token. Rare words are split into pieces.

code

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('gpt2')

text = 'Tokenization is not free.'

ids = tok(text)['input_ids']

print(ids)                              # list of integers

print(tok.convert_ids_to_tokens(ids))   # the subword pieces

print(len(ids), 'tokens for', len(text), 'characters')

Run this and you will see that 'Tokenization' is split into several pieces, and that leading spaces are attached to the start of tokens.

Why an engineer should care

Consequence	What it means in practice
Billing and limits are in tokens	API prices, rate limits and context windows are counted in tokens, not characters. For English prose, a rough estimate is about 4 characters or 0.75 words per token.
Token counts vary with content	Code, JSON, URLs, numbers and many non-English languages often use noticeably more tokens per character. The same prompt in Hindi or Japanese can cost several times more than in English.
Tokenizers are model-specific	The same text produces different IDs and different counts across model families. Always count with the target model's tokenizer, and never mix tokenizers and models.
Character-level tasks are hard	The model sees pieces like straw + berry, not individual letters. Counting characters, reversing strings and exact arithmetic on long numbers are weak spots. Do those in code instead.

Tokenization, Transformers and Context Windows: A Practical Mental Model

1. Tokenization: the model never sees your text

Why an engineer should care

2. The transformer, at the level you need

Inside a decoder-only model

Two phases that explain latency

3. Context windows: one shared budget

Checklist you should be able to explain