Neural Network

1. Model Summary

At a very simple level, neurons are basically computational units that take inputs (dendrites) as electrical inputs (called “spikes”) that are channeled to outputs (axons). In our model, our dendrites are like the input features x1⋯xn$x_1\cdots x_n$ and the output is the result of our hypothesis function. In this model our x0=1$x_0=1$ input node is sometimes called the “bias unit.” It is always equal to 1. In neural networks, we use the same logistic function as in classification, 11+e−θTx$\frac{1}{1+e^{-{\theta }^{T}x}}$, yet we sometimes call it a sigmoid (logistic) activation function. In this situation, our “theta” parameters are sometimes called “weights”.
Visually, a simplistic representation looks like:

three layers: input layer / hidden layer / output layer
a(j)i$a_i^{(j)}$ : activation unit i in layer j
Θ(j)${\mathrm{\Theta }}^{(j)}$ : Matrix that controls function mapping from j-th layer to (j+1)-th layer
If layer j has sj$s_j$ units, layer j+1 has sj+1$s_{j+1}$ units, then size of Θ(j)${\mathrm{\Theta }}^{(j)}$ is sj+1∗(sj+1)$s_{j+1}\ast (s_j+1)$
L$L$ : Number of Layers
sl$s_l$ : Number of units in l-th layer
Number of Inputs: the dimension of features in x(i)$x^{(i)}$
Binary Classification: 1 output unit
K-classes Classification: K output unit

2. Forward Propagation

1) Add a(0)x=1$a_x^{(0)}=1$ first
2) zx+1=Θ(x)ax$z_{x+1}={\mathrm{\Theta }}^{(x)}{a}_x$
3) ax+1=g(zx+1)$a_{x+1}=g(z_{x+1})$ — g(x) : Sigmoid

3. Cost Function

Excluding Bias Term

4. Backpropagation Algorithm

δ(l)j${\delta }_j^{(l)}$ error of node j in layer l, then

where g′(z(i))=a(i).∗(1−a(i))$g^{\prime }(z^{(i)})=a^{(i)}.\ast (1-a^{(i)})$

One thing to note: use one training set to train the model at one time!

5. Unrolling Parameters

Enroll them to vectors/Get back:

6.Gradient Checking

When learning, turn off gradient checking!!!

7. Random Initialization

8.Network Architecture

one hidden layer/
more than one hidden layer with same number of units