一起实现 GPT 的分词器

Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.

Why HN likes it: Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.

“Worth spending 2h for the deeper understanding if you deal with LLMs.”