Multi-head attention and positional encoding
A single attention catches one kind of relationship. What if you need grammar, reference and position at the same time?
2 steps
105 XP
A free account is needed
Start the lesson →
Sources
- Vaswani, A. et al. 2017 · Attention Is All You Need · NeurIPS 2017
- Clark, K. et al. 2019 · What Does BERT Look At? An Analysis of BERT's Attention · BlackboxNLP 2019
- Voita, E. et al. 2019 · Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting · ACL 2019
- Su, J. et al. 2021 · RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE) · arXiv:2104.09864
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons