Encoder or decoder: to understand, or to generate?
The same architecture, with one difference: the attention mask. Bidirectional context improves understanding measurably; a causal mask is the precondition for generation.
4 steps
175 XP
A free account is needed
Start the lesson →
Sources
- Vaswani, A. et al. 2017 · Attention Is All You Need · NeurIPS 2017
- Devlin, J. et al. 2019 · BERT: Pre-training of Deep Bidirectional Transformers · NAACL 2019
- Raffel, C. et al. 2020 · Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer · JMLR 21(140)
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons