Scaling laws: what growth buys and what it costs
We know the true entropy of the source from the perplexity lesson, which lets us compare the floor our fitted scaling law reaches against the truth.
4 steps
175 XP
A free account is needed
Start the lesson →
Sources
- Kaplan, J. et al. 2020 · Scaling Laws for Neural Language Models · arXiv:2001.08361
- Hoffmann, J. et al. 2022 · Training Compute-Optimal Large Language Models · NeurIPS 2022
- Hestness, J. et al. 2017 · Deep Learning Scaling is Predictable, Empirically · arXiv:1712.00409
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons