Introduction to Recurrent Neural Networks

What is RNN

The networks are recurrent because they performance same computations for all the elements of a sequence of input, and the output of each element dependents, in addition to current input, from all the previous commutations.

Why RNN

• Sequential type information of the inputs
  Video Analysis
  Speech Recognition
  Machine Translation
• RNN have proved to have excellent performance in such problems

RNN Procedure

Sigmoid Gradient

The Vanish Gradient Problem

Consider the recurrent networks:

then,

LSTM Cell

• Input Gate
  

  g=tanh(bg+xtUg+ht−1Vg)$$g=tanh(b^g+x_t{U}^g+h_{t-1}{V}^g)$$
  
  
  i=σ(bi+xtUi+ht−1Vi)$$i=\sigma (b^i+x_t{U}^i+h_{t-1}{V}^i)$$
  
  
  outi=g∘i$$ou{t}_i=g\circ i$$

• forget gate
  

  f=σ(bf+xtUf+ht−1Vf)$$f=\sigma (b^f+x_t{U}^f+h_{t-1}{V}^f)$$
  
  
  st=st−1∘f+g∘i$$s_t=s_{t-1}\circ f+g\circ i$$

• output gate
  

  o=σ(bo+xtUo+ht−1Vo)$$o=\sigma (b^o+x_t{U}^o+h_{t-1}{V}^o)$$
  
  
  ht=tanh(st)∘o$$h_t=tanh(s_t)\circ o$$

Reducing The Problem

Reference

• http://adventuresinmachinelearning.com/recurrent-neural-networks-lstm-tutorial-tensorflow/
• Deep Learning with Tensorflow