Long Short-Term Memory (LSTM)
Long Short-Term Memory (LSTM)
Outline
Background
LSTM Network
Extended LSTM
LSTM vs. RNN
Examples
References
Background
Definition
LSTM Network
For detailed derivation: http://cs224d.stanford.edu/
Extended LSTM---GRU
LSTM vs. RNN
References
1.A. Graves. Supervised Sequence Labelling with Recurrent Neural Networks. Textbook, Studies in Computational Intelligence, Springer, 2012
2.Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780
3.Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural computation, 12(10), 2451-2471
4.Graves, Alex. Supervised sequence labelling with recurrent neural networks. Vol. 385. Springer, 2012
5.Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian, Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2012
Long Short-Term Memory (LSTM)
Outline
Background
LSTM Network
Extended LSTM
LSTM vs. RNN
Examples
References
Background
Definition
LSTM Network
For detailed derivation: http://cs224d.stanford.edu/
Extended LSTM---GRU
LSTM vs. RNN
References
1.A. Graves. Supervised Sequence Labelling with Recurrent Neural Networks. Textbook, Studies in Computational Intelligence, Springer, 2012
2.Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780
3.Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural computation, 12(10), 2451-2471
4.Graves, Alex. Supervised sequence labelling with recurrent neural networks. Vol. 385. Springer, 2012
5.Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian, Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2012