Tokenisation: how text turns into numbers
The first thing a language model sees. And why it matters especially in agglutinative languages like Turkish.
2 steps
110 XP
Open without an account
Start the lesson →
Sources
- Sennrich, Haddow, Birch 2016 · Neural Machine Translation of Rare Words with Subword Units (BPE) · ACL 2016
- Kudo, T. & Richardson, J. 2018 · SentencePiece: A Simple and Language Independent Subword Tokenizer · EMNLP 2018
- Gage, P. 1994 · A New Algorithm for Data Compression (özgün BPE) · The C Users Journal, 12(2)
- Rust, P. et al. 2021 · How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models · ACL 2021
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons