机器学习 用一个易懂的小栗子解释神经网络的BP算法

神经网络之BP算法

• BP算法也叫做$δ$δ算法
• 以三层的感知器为例：
  

输出层误差：$E=\frac{1}{2}(d-O)^2=\frac{1}{2}\sum_{k=1}^ζ(d_k-O_k)^2$E=21​(d−O)2=21​∑k=1ζ​(dk​−Ok​)2

隐层的误差：$E=\frac{1}{2}\sum_{k=1}^ζ(d_k-f(net_k))^2 = \frac{1}{2}\sum_{k=1}^ζ(d_k-f(\sum_{j=1}^mw_{jk}y_j))$E=21​∑k=1ζ​(dk​−f(netk​))2=21​∑k=1ζ​(dk​−f(∑j=1m​wjk​yj​))

输入层误差：
$E=\frac{1}{2}\sum_{k=1}^ζd_k-f[\sum_{j=0}^mf(net_i)]^2 = \frac{1}{2}\sum_{k=1}^ζd_k-f[\sum_{j=0}^mw_{jk}f(\sum_{i=1}^nv_{ij}x_i)]$E=21​∑k=1ζ​dk​−f[∑j=0m​f(neti​)]2=21​∑k=1ζ​dk​−f[∑j=0m​wjk​f(∑i=1n​vij​xi​)]

---

神经网络之SGD

误差E有了，为了使误差越来越小，可以采用随机梯度下降的方式进行$w和v$w和v的求解，即求得$w$w和$v$v使得误差E最小。

---

BP算法例子

$net_{h1} = w_1*l_1+w_2*l_2+b_1*1 = 0.1*5+0.15*10+0.35*1=2.35$neth1​=w1​∗l1​+w2​∗l2​+b1​∗1=0.1∗5+0.15∗10+0.35∗1=2.35

$out_{h1} = \frac{1}{1+e^{-net_{h1}}} = \frac{1}{1+e^{-2.35}} = 0.912934$outh1​=1+e−neth1​1​=1+e−2.351​=0.912934

同理可求：$out_{h2} = 0.979164$outh2​=0.979164、$out_{h3} = 0.995275$outh3​=0.995275

$net_{o1} = w_7 * out_{h1} + w_9 * out_{h2} + w_11 * out_{h3} + b_2 * 1$neto1​=w7​∗outh1​+w9​∗outh2​+w1​1∗outh3​+b2​∗1

$net_{o1} = 0.4*0.912934+0.5*0.979164+0.6*0.995275 = 2.1019206$neto1​=0.4∗0.912934+0.5∗0.979164+0.6∗0.995275=2.1019206

$out_{o1} = \frac{1}{1+e^{-net_{o1}}} = \frac{1}{1+e^{-2.1019206}} = 0.891090$outo1​=1+e−neto1​1​=1+e−2.10192061​=0.891090

同理可求：$out_{o2} = 0.904330$outo2​=0.904330

即输出层误差
$E_{o1} = \frac{1}{2}(o1-out_{o1})^2$Eo1​=21​(o1−outo1​)2

$E_{o1} = \frac{1}{2}(o2-out_{o2})^2$Eo1​=21​(o2−outo2​)2

$E_{total} = E_{o1} + E_{o2} = \frac{1}{2}(0.01-0.891090)^2 + \frac{1}{2}(0.99-0.904330)^2 = 0.391829$Etotal​=Eo1​+Eo2​=21​(0.01−0.891090)2+21​(0.99−0.904330)2=0.391829

---

以计算$w_7^+$w7+​为例子

• $\frac{∂E_{total}}{∂w_7} = \frac{∂E_{total}}{∂out_{o1}}*\frac{∂out_{o1}}{∂net_{o1}}*\frac{∂net_{o1}}{∂w_7}$∂w7​∂Etotal​​=∂outo1​∂Etotal​​∗∂neto1​∂outo1​​∗∂w7​∂neto1​​

1. $\frac{∂E_{total}}{∂out_{o1}} = 2*\frac{1}{2}(target_{o1} - out_{o1})^{2-1} * -1 + 0 = -(0.01-0.891091) = 0.88109$∂outo1​∂Etotal​​=2∗21​(targeto1​−outo1​)2−1∗−1+0=−(0.01−0.891091)=0.88109
2. $\frac{∂out_{o1}}{∂net_{o1}} = out_{o1}(1-out_{o1}) = 0.891090(1-0.891090) = 0.097049$∂neto1​∂outo1​​=outo1​(1−outo1​)=0.891090(1−0.891090)=0.097049
3. $\frac{∂net_{o1}}{∂w_7} = 1*out_{h1}*w_7^{1-1}+0+0+0 = 0.912934$∂w7​∂neto1​​=1∗outh1​∗w71−1​+0+0+0=0.912934

• $\frac{∂E_{total}}{∂w_7} = 0.88109 * 0.097049 * 0.912934 = 0.078064$∂w7​∂Etotal​​=0.88109∗0.097049∗0.912934=0.078064
• $w_7^+ = w_7 + △w_7 = w_7 - η\frac{∂E_{total}}{∂w_7} =0.4 - 0.5*0.078064 = 0.360968（η自定义的值）$w7+​=w7​+△w7​=w7​−η∂w7​∂Etotal​​=0.4−0.5∗0.078064=0.360968（η自定义的值）

同理可求：
$w_1^+ = 0.094534$w1+​=0.094534

$w_2^+ = 0.139069$w2+​=0.139069

$w_3^+ = 0.198211$w3+​=0.198211

$w_4^+ = 0.246422$w4+​=0.246422

$w_5^+ = 0.299497$w5+​=0.299497

$w_6^+ = 0.348993$w6+​=0.348993

$w_7^+ = 0.360968$w7+​=0.360968

$w_8^+ = 0.453383$w8+​=0.453383

$w_9^+ = 0.458137$w9+​=0.458137

$w_{10}^+ = 0.553629$w10+​=0.553629

$w_{11}^+ = 0.557448$w11+​=0.557448

$w_{12}^+ = 0.653688$w12+​=0.653688

---

第十次迭代结果：$O = (0.662866,0.908195)$O=(0.662866,0.908195)
第百次迭代结果：$O = (0.073889,0.945864)$O=(0.073889,0.945864)
第千次迭代结果：$O = (0.022971,0.977675)$O=(0.022971,0.977675)

---