01 - tokenisers: the first rabbit hole

lecture 1 of stanford cs336: language modeling from scratch

lecture 1 was mostly a quick overview. a map of what the course is going to cover, how the pieces fit together, and enough context to not feel completely lost later.

honestly, most of the lecture was meant to be skipped. there wasn't much value in writing down every slide from a fast tour through the syllabus and the setup. so i skimmed it, kept the broad picture, and moved on.

then tokenisers appeared

towards the end, we moved a little closer to tokenisers. that was enough to send me into the usual rabbit hole: i opened a few tabs, started looking up how tokenisation works, and somehow ended up reading about the different ways text gets broken down before a language model sees it.

the basic problem is pretty simple. a language model does not read a sentence as a sentence. it needs numbers. a tokeniser turns text into small pieces and maps those pieces to token ids.

the bpe rabbit hole

bpe, or byte pair encoding, was the one that pulled me in. the rough idea is very readable:

i mostly knew how bpe works already, at least in the “yeah, it merges frequent pairs” kind of way. but that is exactly the sort of knowledge that becomes dangerous when you start implementing it. so i decided to pretend i knew nothing and start from square zero.

1. start with small units, usually bytes;
2. find the pair that appears most often;
3. merge that pair into a new token;
4. repeat until the vocabulary reaches the size you want.

repeated patterns gradually become useful chunks. a frequent piece like ing might become one token, but a rare or new word can still fall back to smaller pieces. during encoding, the learned merges are applied again to turn new text into token ids. decoding reverses that process.

this sounds simple enough until you start thinking about unicode, bytes, whitespace, special tokens, merge ordering, vocab files, and all the little edge cases hiding behind “just split the text”. classic.

the tabs i opened

i came across awesome-tokenizers, a curated list of tokenizer implementations.

i also went through this great post by @rasbt: bpe tokenizer from scratch.

what happens next

after a super long break because of college midterms (fahh, them), i'm finally starting to build one. it's called takeshi, and i'll be building it in rust.

another self-imposed challenge: i'm building the whole thing in nvim. idk, let's see.