LSTM: the cell path and the forget gate
In the RNN lesson we measured the influence decaying at a rate of 0.59 per step. LSTM pushes that rate towards 1, and how it does so is visible in a single multiplication.
4 steps
175 XP
A free account is needed
Start the lesson →
Sources
- Hochreiter, S. & Schmidhuber, J. 1997 · Long Short-Term Memory · Neural Computation 9(8)
- Gers, F. A., Schmidhuber, J. & Cummins, F. 2000 · Learning to Forget: Continual Prediction with LSTM · Neural Computation 12(10)
- Jozefowicz, R., Zaremba, W. & Sutskever, I. 2015 · An Empirical Exploration of Recurrent Network Architectures · ICML 2015
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons