ML Academy · Track 3 · Large Language Models

The blind spot of multilingual models

The model does not "not know" a language; the tokeniser does not recognise it. The difference shows up on the bill.

4 steps 225 XP A free account is needed

Sources

Ahia, O. et al. 2023 · Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models · EMNLP 2023
Petrov, A. et al. 2023 · Language Model Tokenizers Introduce Unfairness Between Languages · NeurIPS 2023
Rust, P. et al. 2021 · How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models · ACL 2021
Conneau, A. et al. 2020 · Unsupervised Cross-lingual Representation Learning at Scale (XLM-R) · ACL 2020

ML Academy · an interactive machine learning course that runs in your browser · All lessons