给定数据集D = { ( x ⃗ 1 , y 1 ) , ( x ⃗ 2 , y 2 ) , … , ( x ⃗ m , y m ) } \mathcal{D} = \left\{ (\vec x_1,y_1),(\vec x_2,y_2),…,(\vec x_m,y_m) \right\} D = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x m , y m ) } x ⃗ i = ( x i 1 , x i 2 , … , x i d ) \vec x_i = (x_{i1},x_{i2},…,x_{id}) x i = ( x i 1 , x i 2 , … , x i d ) y i ∈ R \quad y_i \in R y i ∈ R f ( x ⃗ ) = w 1 x 1 + w 2 x 2 + … + w d x d + b
f(\vec x) = w_1x_1 + w_2x_2 +\ldots +w_dx_d + b
f ( x ) = w 1 x 1 + w 2 x 2 + … + w d x d + b f ( x ⃗ ) = w ⃗ T x ⃗ + b
f(\vec x) = \vec w^\mathrm{T} \vec x + b
f ( x ) = w T x + b w ⃗ = ( w 1 , w 2 , … , w d ) \vec w = (w_1,w_2,\ldots,w_d) w = ( w 1 , w 2 , … , w d )
通过训练，求得最优的 w ⃗ , b \vec w, b w , b
为了理解方便，我们先考虑一种简单的情形，输入数据的属性只有一个，即研究房屋的面积与房屋价格之间的关系，不考虑价格与其他房屋属性之间的关系。线性回归模型试图寻找最优的 w , b w, b w , b f ( x i ) = w x i + b f(x_i) = wx_i + b f ( x i ) = w x i + b f ( x i ) f(x_i) f ( x i ) y i y_i y i
如何训练求得最优的 w , b w,b w , b
关键在于如何衡量f ( x ) f(x) f ( x ) y y y w , b w, b w , b ( w ∗ , b ∗ ) = a r g min w , b ∑ i = 1 m ( f ( x i ) − y i ) 2 = a r g min w , b ∑ i = 1 m ( y i − w x i − b ) 2
\begin{aligned}
(w^*,b^*) & = arg \min \limits_{w,b} \sum_{i=1}^m(f( x_i) - y_i)^2 \\& = arg \min \limits_{w,b} \sum_{i=1}^m(y_i - wx_i - b)^2
\end{aligned}
( w ∗ , b ∗ ) = a r g w , b min i = 1 ∑ m ( f ( x i ) − y i ) 2 = a r g w , b min i = 1 ∑ m ( y i − w x i − b ) 2
只要我们能够做到极小化误差函数，就能够得到最优解。求解w , b w,b w , b E ( w , b ) = ∑ i = 1 m ( y i − w x i − b ) 2 E_{(w,b)} = \sum_{i=1}^m(y_i - wx_i - b)^2 E ( w , b ) = ∑ i = 1 m ( y i − w x i − b ) 2
因为这里的误差函数E ( w , b ) E_{(w,b)} E ( w , b ) w , b w,b w , b w 和 b w和 b w 和 b w w w b b b ∂ E ( w , b ) ∂ w = ∂ ∂ w [ ∑ i = 1 m ( y i − w x i − b ) 2 ] = ∑ i = 1 m ∂ ∂ w [ ( y i − w x i − b ) 2 ] = ∑ i = 1 m [ 2 ⋅ ( y i − w x i − b ) ⋅ ( − x i ) ] = ∑ i = 1 m [ 2 ⋅ ( w x i 2 − y i x i + b x i ) ] = 2 ⋅ ( w ∑ i = 1 m x i 2 − ∑ i = 1 m y i x i + b ∑ i = 1 m x i ) = 2 ( w ∑ i = 1 m x i 2 − ∑ i = 1 m ( y i − b ) x i )
\begin{aligned} \cfrac{\partial E_{(w, b)}}{\partial w}&=\cfrac{\partial}{\partial w} \left[\sum_{i=1}^{m}\left(y_{i}-w x_{i}-b\right)^{2}\right] \\ &= \sum_{i=1}^{m}\cfrac{\partial}{\partial w} \left[\left(y_{i}-w x_{i}-b\right)^{2}\right] \\ &= \sum_{i=1}^{m}\left[2\cdot\left(y_{i}-w x_{i}-b\right)\cdot (-x_i)\right] \\ &= \sum_{i=1}^{m}\left[2\cdot\left(w x_{i}^2-y_i x_i +bx_i\right)\right] \\ &= 2\cdot\left(w\sum_{i=1}^{m} x_{i}^2-\sum_{i=1}^{m}y_i x_i +b\sum_{i=1}^{m}x_i\right) \\ &=2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right) \end{aligned}
∂ w ∂ E ( w , b ) = ∂ w ∂ [ i = 1 ∑ m ( y i − w x i − b ) 2 ] = i = 1 ∑ m ∂ w ∂ [ ( y i − w x i − b ) 2 ] = i = 1 ∑ m [ 2 ⋅ ( y i − w x i − b ) ⋅ ( − x i ) ] = i = 1 ∑ m [ 2 ⋅ ( w x i 2 − y i x i + b x i ) ] = 2 ⋅ ( w i = 1 ∑ m x i 2 − i = 1 ∑ m y i x i + b i = 1 ∑ m x i ) = 2 ( w i = 1 ∑ m x i 2 − i = 1 ∑ m ( y i − b ) x i )
∂ E ( w , b ) ∂ b = ∂ ∂ b [ ∑ i = 1 m ( y i − w x i − b ) 2 ] = ∑ i = 1 m ∂ ∂ b [ ( y i − w x i − b ) 2 ] = ∑ i = 1 m [ 2 ⋅ ( y i − w x i − b ) ⋅ ( − 1 ) ] = ∑ i = 1 m [ 2 ⋅ ( b − y i + w x i ) ] = 2 ⋅ [ ∑ i = 1 m b − ∑ i = 1 m y i + ∑ i = 1 m w x i ] = 2 ( m b − ∑ i = 1 m ( y i − w x i ) )
\begin{aligned} \cfrac{\partial E_{(w, b)}}{\partial b}&=\cfrac{\partial}{\partial b} \left[\sum_{i=1}^{m}\left(y_{i}-w x_{i}-b\right)^{2}\right] \\ &=\sum_{i=1}^{m}\cfrac{\partial}{\partial b} \left[\left(y_{i}-w x_{i}-b\right)^{2}\right] \\ &=\sum_{i=1}^{m}\left[2\cdot\left(y_{i}-w x_{i}-b\right)\cdot (-1)\right] \\ &=\sum_{i=1}^{m}\left[2\cdot\left(b-y_{i}+w x_{i}\right)\right] \\ &=2\cdot\left[\sum_{i=1}^{m}b-\sum_{i=1}^{m}y_{i}+\sum_{i=1}^{m}w x_{i}\right] \\ &=2\left(m b-\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right)\right) \end{aligned}
∂ b ∂ E ( w , b ) = ∂ b ∂ [ i = 1 ∑ m ( y i − w x i − b ) 2 ] = i = 1 ∑ m ∂ b ∂ [ ( y i − w x i − b ) 2 ] = i = 1 ∑ m [ 2 ⋅ ( y i − w x i − b ) ⋅ ( − 1 ) ] = i = 1 ∑ m [ 2 ⋅ ( b − y i + w x i ) ] = 2 ⋅ [ i = 1 ∑ m b − i = 1 ∑ m y i + i = 1 ∑ m w x i ] = 2 ( m b − i = 1 ∑ m ( y i − w x i ) )
令：{ ∂ E ( w , b ) ∂ w = 2 ( w ∑ i = 1 m x i 2 − ∑ i = 1 m ( y i − b ) x i ) = 0 ∂ E ( w , b ) ∂ b = 2 ( m b − ∑ i = 1 m ( y i − w x i ) ) = 0
\left \{
\begin{array}{lr}
\cfrac{\partial E_{(w, b)}}{\partial w} = 2\left(w \sum_{i=1}^{m} x_{i}^{2}-\sum_{i=1}^{m}\left(y_{i}-b\right) x_{i}\right) =0 &\\
\cfrac{\partial E_{(w, b)}}{\partial b} =2\left(m b-\sum_{i=1}^{m}\left(y_{i}-w x_{i}\right)\right) = 0
\end{array}
\right.
⎩ ⎪ ⎨ ⎪ ⎧ ∂ w ∂ E ( w , b ) = 2 ( w ∑ i = 1 m x i 2 − ∑ i = 1 m ( y i − b ) x i ) = 0 ∂ b ∂ E ( w , b ) = 2 ( m b − ∑ i = 1 m ( y i − w x i ) ) = 0
解得：{ w = ∑ i = 1 m y i ( x i − x ˉ ) ∑ i = 1 m x i 2 − 1 m ( ∑ i = 1 m x i ) 2 b = 1 m ∑ i = 1 m ( y i − w x i )
\left \{
\begin{array}{lr}
w=\cfrac{\sum_{i=1}^{m}y_i(x_i-\bar{x})}{\sum_{i=1}^{m}x_i^2-\cfrac{1}{m}(\sum_{i=1}^{m}x_i)^2} &\\b=\cfrac{1}{m}\sum_{i=1}^{m}(y_i-wx_i)
\end{array}
\right.
⎩ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎧ w = ∑ i = 1 m x i 2 − m 1 ( ∑ i = 1 m x i ) 2 ∑ i = 1 m y i ( x i − x ˉ ) b = m 1 ∑ i = 1 m ( y i − w x i )
求得w , b w,b w , b
对于多变量回归的情况，也就是预测房价时，除了考虑房屋面积，还要考虑卧室数量，建造年份等因素。为了便于讨论。把偏置b和w整合到一个向量中(下文出现的w ⃗ \vec w w w ⃗ = ( w 1 , w 2 , … , w d ; b )
\vec w = (w_1, w_2,\ldots,w_d;b)
w = ( w 1 , w 2 , … , w d ; b ) w ⃗ \vec w w w ⃗ T \vec w^ \mathrm T w T
把m条数据的房屋价格也写成向量的形式：y ⃗ = ( y 1 , y 2 , … , y m )
\vec y = (y_1, y_2,\ldots,y_m)
y = ( y 1 , y 2 , … , y m )
相应的把数据 D \mathcal D D x ⃗ i = ( x i 1 , x i 2 , … , x i d , 1 ) \vec x_i = (x_{i1},x_{i2},\ldots,x_{id},1) x i = ( x i 1 , x i 2 , … , x i d , 1 ) x ⃗ i \vec x_i x i m × ( d + 1 ) m\times(d+1) m × ( d + 1 ) X \textbf X X X = [ x 11 x 12 ⋯ x 1 d 1 x 21 x 22 ⋯ x 2 d 1 ⋮ ⋮ ⋱ ⋮ ⋮ x m 1 x m 2 ⋯ x m d 1 ]
\textbf X = \begin{bmatrix}x_{11} & x_{12} & \cdots & x_{1d} &1 \\x_{21} & x_{22} & \cdots & x_{2d} &1 \\ \vdots & \vdots & \ddots & \vdots &\vdots \\x_{m1} & x_{m2} & \cdots\ & x_{md} &1 \\\end{bmatrix}
X = ⎣ ⎢ ⎢ ⎢ ⎡ x 1 1 x 2 1 ⋮ x m 1 x 1 2 x 2 2 ⋮ x m 2 ⋯ ⋯ ⋱ ⋯ x 1 d x 2 d ⋮ x m d 1 1 ⋮ 1 ⎦ ⎥ ⎥ ⎥ ⎤
此时对于单个样本有：f ( x ⃗ i ) = w ⃗ T x ⃗
f(\vec x_i) = \vec w ^T \vec x
f ( x i ) = w T x
此时，X w ⃗ = [ x 11 x 12 ⋯ x 1 d 1 x 21 x 22 ⋯ x 2 d 1 ⋮ ⋮ ⋱ ⋮ ⋮ x m 1 x m 2 ⋯ x m d 1 ] [ w 1 w 2 ⋮ b ] = [ w 1 x 11 + w 2 x 12 + … + w d x 1 d + b w 1 x 21 + w 2 x 22 + … + w d x 2 d + b ⋮ w 1 x m 1 + w 2 x m 2 + … + w d x m d + b ]
X\vec w =\begin{bmatrix}x_{11} & x_{12} & \cdots & x_{1d} &1 \\x_{21} & x_{22} & \cdots & x_{2d} &1 \\ \vdots & \vdots &
\ddots & \vdots &\vdots \\x_{m1} & x_{m2} & \cdots\ & x_{md} &1 \\\end{bmatrix} \begin{bmatrix}w_{1} \\w_{2} \\ \vdots \\b \\\end{bmatrix} = \begin{bmatrix} w_1x_{11} + w_2x_{12} +\ldots +w_dx_{1d} + b \\w_1x_{21} + w_2x_{22} +\ldots +w_dx_{2d} + b \\ \vdots \\w_1x_{m1} + w_2x_{m2} +\ldots +w_dx_{md} + b \\\end{bmatrix}
X w = ⎣ ⎢ ⎢ ⎢ ⎡ x 1 1 x 2 1 ⋮ x m 1 x 1 2 x 2 2 ⋮ x m 2 ⋯ ⋯ ⋱ ⋯ x 1 d x 2 d ⋮ x m d 1 1 ⋮ 1 ⎦ ⎥ ⎥ ⎥ ⎤ ⎣ ⎢ ⎢ ⎢ ⎡ w 1 w 2 ⋮ b ⎦ ⎥ ⎥ ⎥ ⎤ = ⎣ ⎢ ⎢ ⎢ ⎡ w 1 x 1 1 + w 2 x 1 2 + … + w d x 1 d + b w 1 x 2 1 + w 2 x 2 2 + … + w d x 2 d + b ⋮ w 1 x m 1 + w 2 x m 2 + … + w d x m d + b ⎦ ⎥ ⎥ ⎥ ⎤
此时的最优解为：
w ^ ∗ = arg min w ^ ( y − X w ^ ) T ( y − X w ^ ) \hat{\boldsymbol{w}}^*=\underset{\hat{\boldsymbol{w}}}{\arg\min}(\boldsymbol{y}-\mathbf{X}\hat{\boldsymbol{w}})^{\mathrm{T}}(\boldsymbol{y}-\mathbf{X}\hat{\boldsymbol{w}}) w ^ ∗ = w ^ arg min ( y − X w ^ ) T ( y − X w ^ )
均方误差为：E w ^ = ( y − X w ^ ) T ( y − X w ^ )
E_{\hat w}=(\boldsymbol{y}-\mathbf{X}\hat{\boldsymbol{w}})^{\mathrm{T}}(\boldsymbol{y}-\mathbf{X}\hat{\boldsymbol{w}})
E w ^ = ( y − X w ^ ) T ( y − X w ^ ) w ⃗ \vec w w ∂ E w ^ ∂ w ^ = 2 X T ( X w ^ − y )
\cfrac{\partial E_{\hat{\boldsymbol w}}}{\partial \hat{\boldsymbol w}}=2\mathbf{X}^{\mathrm{T}}(\mathbf{X}\hat{\boldsymbol w}-\boldsymbol{y})
∂ w ^ ∂ E w ^ = 2 X T ( X w ^ − y ) E w ^ = ( y − X w ^ ) T ( y − X w ^ )
E_{\hat{\boldsymbol w}}=(\boldsymbol{y}-\mathbf{X}\hat{\boldsymbol w})^{\mathrm{T}}(\boldsymbol{y}-\mathbf{X}\hat{\boldsymbol w})
E w ^ = ( y − X w ^ ) T ( y − X w ^ ) E w ^ = y T y − y T X w ^ − w ^ T X T y + w ^ T X T X w ^
E_{\hat{\boldsymbol w}}= \boldsymbol{y}^{\mathrm{T}}\boldsymbol{y}-\boldsymbol{y}^{\mathrm{T}}\mathbf{X}\hat{\boldsymbol w}-\hat{\boldsymbol w}^{\mathrm{T}}\mathbf{X}^{\mathrm{T}}\boldsymbol{y}+\hat{\boldsymbol w}^{\mathrm{T}}\mathbf{X}^{\mathrm{T}}\mathbf{X}\hat{\boldsymbol w}
E w ^ = y T y − y T X w ^ − w ^ T X T y + w ^ T X T X w ^
对w ^ \hat{\boldsymbol w} w ^ ∂ E w ^ ∂ w ^ = ∂ y T y ∂ w ^ − ∂ y T X w ^ ∂ w ^ − ∂ w ^ T X T y ∂ w ^ + ∂ w ^ T X T X w ^ ∂ w ^
\cfrac{\partial E_{\hat{\boldsymbol w}}}{\partial \hat{\boldsymbol w}}= \cfrac{\partial \boldsymbol{y}^{\mathrm{T}}\boldsymbol{y}}{\partial \hat{\boldsymbol w}}-\cfrac{\partial \boldsymbol{y}^{\mathrm{T}}\mathbf{X}\hat{\boldsymbol w}}{\partial \hat{\boldsymbol w}}-\cfrac{\partial \hat{\boldsymbol w}^{\mathrm{T}}\mathbf{X}^{\mathrm{T}}\boldsymbol{y}}{\partial \hat{\boldsymbol w}}+\cfrac{\partial \hat{\boldsymbol w}^{\mathrm{T}}\mathbf{X}^{\mathrm{T}}\mathbf{X}\hat{\boldsymbol w}}{\partial \hat{\boldsymbol w}}
∂ w ^ ∂ E w ^ = ∂ w ^ ∂ y T y − ∂ w ^ ∂ y T X w ^ − ∂ w ^ ∂ w ^ T X T y + ∂ w ^ ∂ w ^ T X T X w ^
由矩阵微分公式∂ a T x ∂ x = ∂ x T a ∂ x = a , ∂ x T A x ∂ x = ( A + A T ) x
\cfrac{\partial\boldsymbol{a}^{\mathrm{T}}\boldsymbol{x}}{\partial\boldsymbol{x}}=\cfrac{\partial\boldsymbol{x}^{\mathrm{T}}\boldsymbol{a}}{\partial\boldsymbol{x}}=\boldsymbol{a},\cfrac{\partial\boldsymbol{x}^{\mathrm{T}}\mathbf{A}\boldsymbol{x}}{\partial\boldsymbol{x}}=(\mathbf{A}+\mathbf{A}^{\mathrm{T}})\boldsymbol{x}
∂ x ∂ a T x = ∂ x ∂ x T a = a , ∂ x ∂ x T A x = ( A + A T ) x
可得∂ E w ^ ∂ w ^ = 0 − X T y − X T y + ( X T X + X T X ) w ^
\cfrac{\partial E_{\hat{\boldsymbol w}}}{\partial \hat{\boldsymbol w}}= 0-\mathbf{X}^{\mathrm{T}}\boldsymbol{y}-\mathbf{X}^{\mathrm{T}}\boldsymbol{y}+(\mathbf{X}^{\mathrm{T}}\mathbf{X}+\mathbf{X}^{\mathrm{T}}\mathbf{X})\hat{\boldsymbol w}
∂ w ^ ∂ E w ^ = 0 − X T y − X T y + ( X T X + X T X ) w ^
∂ E w ^ ∂ w ^ = 2 X T ( X w ^ − y )
\cfrac{\partial E_{\hat{\boldsymbol w}}}{\partial \hat{\boldsymbol w}}=2\mathbf{X}^{\mathrm{T}}(\mathbf{X}\hat{\boldsymbol w}-\boldsymbol{y})
∂ w ^ ∂ E w ^ = 2 X T ( X w ^ − y ) w ^ ∗ = ( X T X ) − 1 X T y
\hat{\boldsymbol{w}}^*=(\mathbf{X}^{\mathrm{T}}\mathbf{X})^{-1}\mathbf{X}^{\mathrm{T}}\boldsymbol{y}
w ^ ∗ = ( X T X ) − 1 X T y
这里我们发现：当且仅当X T X \mathbf{X}^{\mathrm{T}}\mathbf{X} X T X w ^ ∗ \hat{\boldsymbol{w}}^* w ^ ∗
实际情况是，X T X \mathbf{X}^{\mathrm{T}}\mathbf{X} X T X
我们也可以从梯度下降的角度来理解如何训练求得最优的 w w w
梯度下降的核心是：设定初始参数w i w_i w i J ( w ^ ) J({\hat w}) J ( w ^ ) w j : = w j − α ∂ J ( w ) ∂ w
w_j:=w_j-\alpha\frac{\partial{J(w)}}{\partial w}
w j : = w j − α ∂ w ∂ J ( w )
J ( w ^ ) = 1 2 ∑ i = 1 m ( w ⃗ T x ⃗ i − y i ) 2
J({\hat w})=\frac{1}{2}\sum_{i=1}^m(\vec w^\mathrm{T} \vec x_i - y_i)^2
J ( w ^ ) = 2 1 i = 1 ∑ m ( w T x i − y i ) 2
根据梯度下降法则:∂ J ( w ⃗ ) ∂ w j = ∂ ∂ w j 1 2 ∑ i = 1 n ( w ⃗ T x ⃗ i − y i ) 2 = 2 ∗ 1 2 ∑ i = 1 n ( w ⃗ T x ⃗ i − y i ) ∗ ∂ ∂ w j ( w ⃗ T x ⃗ i − y i ) = ∑ i = 1 n ( w ⃗ T x ⃗ i − y i ) ∗ ∂ ∂ w j ( ∑ j = 0 d w j x i j − y i ) ) = ∑ i = 1 n ( w ⃗ T x ⃗ i − y i ) x i j
\begin{aligned}
\frac{\partial{J(\vec w)}}{\partial w_j}
&= \frac{\partial}{\partial w_j}\frac{1}{2}\sum_{i=1}^{n}(\vec w^\mathrm{T} \vec x_i - y_i)^2 \\
&= 2*\frac{1}{2}\sum_{i=1}^{n}(\vec w^\mathrm{T}\vec x_i - y_i)*\frac{\partial}{\partial w_j}(\vec w^\mathrm{T}\vec x_i - y_i) \\
&= \sum_{i=1}^{n}(\vec w^\mathrm{T}\vec x_i - y_i)*\frac{\partial}{\partial w_j}(\sum_{j=0}^{d}w_jx_{ij}-y_i))\\
&= \sum_{i=1}^{n}(\vec w^\mathrm{T}\vec x_i - y_i)x_{ij} \\
\end{aligned}
∂ w j ∂ J ( w ) = ∂ w j ∂ 2 1 i = 1 ∑ n ( w T x i − y i ) 2 = 2 ∗ 2 1 i = 1 ∑ n ( w T x i − y i ) ∗ ∂ w j ∂ ( w T x i − y i ) = i = 1 ∑ n ( w T x i − y i ) ∗ ∂ w j ∂ ( j = 0 ∑ d w j x i j − y i ) ) = i = 1 ∑ n ( w T x i − y i ) x i j
w j = w j − α ∑ i = 1 n ( w ⃗ T x ⃗ i − y i ) x i j
w_j = w_j - \alpha\sum_{i=1}^{n}(\vec w^\mathrm{T}\vec x_i - y_i)x_{ij}
w j = w j − α i = 1 ∑ n ( w T x i − y i ) x i j
注：下标 i 表示第几个样本点。
梯度下降法沿着图中的1-2-3-4路径逐步达到最小J ( w ^ ) J({\hat w}) J ( w ^ )
介绍两种回归变体。
当样本数远小于特征数时，直接使用min w ∑ i = 1 m ( y i − w T x i ) 2
\min\limits_{\boldsymbol w} \sum_{i=1}^{m}\left(y_{i}-\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}_{i}\right)^{2}
w min i = 1 ∑ m ( y i − w T x i ) 2
作为损失函数，很容易导致过拟合，我们在原有的误差函数上引入正则化项。
（1）先引入L 2 L_2 L 2 min w ∑ i = 1 m ( y i − w T x i ) 2 + λ ∣ ∣ w ∣ ∣ 2 2
\min\limits_{\boldsymbol w} \sum_{i=1}^{m}\left(y_{i}-\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}\right)^{2}+\lambda||\boldsymbol{w}||_{2}^{2}
w min i = 1 ∑ m ( y i − w T x i ) 2 + λ ∣ ∣ w ∣ ∣ 2 2
其中 λ > 0 \lambda > 0 λ > 0
（2）引入L 1 L_1 L 1 min w ∑ i = 1 m ( y i − w T x i ) 2 + λ ∣ ∣ w ∣ ∣ 1
\min\limits_{\boldsymbol w} \sum_{i=1}^{m}\left(y_{i}-\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}\right)^{2}+\lambda||\boldsymbol{w}||_{1}
w min i = 1 ∑ m ( y i − w T x i ) 2 + λ ∣ ∣ w ∣ ∣ 1
其中 λ > 0 \lambda > 0 λ > 0
L 1 L_1 L 1 L 2 L_2 L 2 L 1 L_1 L 1 L 1 L_1 L 1 w w w
如何理解这一点？为了分析方便，我们假定x x x w ⃗ = ( w 1 , w 2 ) \vec w = (w_1,w_2) w = ( w 1 , w 2 ) w 1 , w 2 w_1,w_2 w 1 , w 2
（1） L 1 L_1 L 1 ∣ ∣ w ⃗ ∣ ∣ 1 = ∣ w 1 ∣ + ∣ w 2 ∣
||\vec w||_1 = |w_1| + |w_2|
∣ ∣ w ∣ ∣ 1 = ∣ w 1 ∣ + ∣ w 2 ∣ L 2 L_2 L 2 ∣ ∣ w ⃗ ∣ ∣ 2 = ∣ w 1 ∣ 2 + ∣ w 2 ∣ 2
||\vec w||_2 = \sqrt{|w_1|^2 + |w_2|^2}
∣ ∣ w ∣ ∣ 2 = ∣ w 1 ∣ 2 + ∣ w 2 ∣ 2
(3)平方误差项等值线E = ∑ i = 1 m ( y i − ( w 1 x i 1 + w 1 x i 2 + b ) ) 2
E= \sum_{i=1}^m (y_i - (w_1 x_{i1} + w_1 x_{i2} + b ))^2
E = i = 1 ∑ m ( y i − ( w 1 x i 1 + w 1 x i 2 + b ) ) 2
注：等值线即空间中取值相同的点连成的线，以L 1 L_1 L 1 ∣ w 1 ∣ + ∣ w 2 ∣ = 1 |w_1| + |w_2| = 1 ∣ w 1 ∣ + ∣ w 2 ∣ = 1 ∣ w 1 ∣ + ∣ w 2 ∣ = 2 |w_1| + |w_2| = 2 ∣ w 1 ∣ + ∣ w 2 ∣ = 2 ∣ w 1 ∣ + ∣ w 2 ∣ = k |w_1| + |w_2| = k ∣ w 1 ∣ + ∣ w 2 ∣ = k L 1 L_1 L 1
由图可知：
（1）无论是“岭回归”还是“Lasso回归”，其最优解都出现在图中平方误差项等值线与相应的正则化等值线相交处。如：“Lasso回归”最优解可能是图中的B点，“岭回归”的最优解可能出现在图中的A点。
（2）采用L 1 L_1 L 1 L 1 L_1 L 1 w 1 w_1 w 1 w 2 w_2 w 2 w 1 w_1 w 1 w 2 w_2 w 2 L 1 L_1 L 1 L 1 L_1 L 1 w w w
说明：如果 w ⃗ \vec w w w ⃗ \vec w w
为什么均方误差作为性能的度量是合理的？我们从概率的角度来分析一下。
假设预测值f ( x i ) f(x_i) f ( x i ) y i y_i y i ϵ i \epsilon_i ϵ i y i = f ( x ⃗ i ) + ϵ i
y_i = f(\vec x_i) + \epsilon_i
y i = f ( x i ) + ϵ i w , b w,b w , b f ( x i ) f(x_i) f ( x i ) y i y_i y i ϵ i \epsilon_i ϵ i ϵ i \epsilon_i ϵ i ϵ i ∼ N ( 0 , σ 2 )
\epsilon_i \sim \mathcal{N}(0,\sigma^2)
ϵ i ∼ N ( 0 , σ 2 ) P ( ϵ i ) = 1 2 π σ e x p ( − ( ϵ i ) 2 2 σ 2 )
P(\epsilon_i) = \frac{1}{\sqrt{2\pi} \sigma}exp(-\frac{(\epsilon_i)^2}{2\sigma^2})
P ( ϵ i ) = 2 π σ 1 e x p ( − 2 σ 2 ( ϵ i ) 2 ) P ( y i ∣ x i ; w ) = 1 2 π σ e x p ( − ( ϵ i ) 2 2 σ 2 ) = 1 2 π σ e x p ( − ( y i − f ( x ⃗ i ) ) 2 2 σ 2 )
\begin{aligned}
P(y_i|x_i;w) &= \frac{1}{\sqrt{2\pi} \sigma}exp(-\frac{(\epsilon_i)^2}{2\sigma^2}) \\
&= \frac{1}{\sqrt{2\pi} \sigma}exp(-\frac{(y_i - f(\vec x_i))^2}{2\sigma^2}) \\
\end{aligned}
P ( y i ∣ x i ; w ) = 2 π σ 1 e x p ( − 2 σ 2 ( ϵ i ) 2 ) = 2 π σ 1 e x p ( − 2 σ 2 ( y i − f ( x i ) ) 2 ) w , x i w,x_i w , x i y i ∼ N ( f ( x ⃗ i ) , σ 2 ) 其 中 ： f ( x ⃗ i ) = w ⃗ T x ⃗ i
y_i \sim \mathcal{N}(f(\vec x_i),\sigma^2)\\其中：f(\vec x_i) = \vec w^T \vec x_i
y i ∼ N ( f ( x i ) , σ 2 ) 其 中 ： f ( x i ) = w T x i
相当于我们已经知道y i y_i y i w ⃗ T \vec w^T w T σ \sigma σ 而极大似然估计法来正是用来解决此类问题的 。通过假设样本独立同分布，最大化似然函数，来进行参数估计。
补充一点：最大化似然函数的原理说简单点就是在一次观测中发生了的事件其概率应该大，概率大的事在观测中容易发生，所以我们希望让每一个P ( y i ∣ x i ; w ) P(y_i|x_i;w) P ( y i ∣ x i ; w )
构造似然函数为：L ( w ⃗ ) = ∏ i = 1 m P ( y i ∣ x i ; w ) = ∏ i = 1 m 1 2 π σ e x p ( − ( y i − w ⃗ T x ⃗ i ) 2 2 σ 2 )
L(\vec w) =\prod_{i=1}^m P(y_i|x_i;w) = \prod_{i=1}^m \frac{1}{\sqrt{2\pi} \sigma}exp(-\frac{(y_i - \vec w^T \vec x_i)^2}{2\sigma^2})
L ( w ) = i = 1 ∏ m P ( y i ∣ x i ; w ) = i = 1 ∏ m 2 π σ 1 e x p ( − 2 σ 2 ( y i − w T x i ) 2 )
两边同时取对数：I n L ( w ⃗ ) = I n ∏ i = 1 m 1 2 π σ e x p ( − ( y i − w ⃗ T x ⃗ i ) 2 2 σ 2 ) = n I n 1 2 π σ − 1 σ 2 ⋅ 1 2 ∑ i = 1 n ( ( y ( i ) − w ⃗ T x ⃗ i ) 2
\begin{aligned}In L(\vec w) &= In \prod_{i=1}^m \frac{1}{\sqrt{2\pi} \sigma}exp(-\frac{(y_i - \vec w^T \vec x_i)^2}{2\sigma^2}) \\&= nIn\frac{1}{{\sqrt{2\pi}\sigma}} - \frac{1}{\sigma^2} \cdot \frac{1}{2}\sum^n_{i=1}((y^{(i)}-\vec w^T \vec x_i)^2\end{aligned}
I n L ( w ) = I n i = 1 ∏ m 2 π σ 1 e x p ( − 2 σ 2 ( y i − w T x i ) 2 ) = n I n 2 π σ 1 − σ 2 1 ⋅ 2 1 i = 1 ∑ n ( ( y ( i ) − w T x i ) 2 1 2 ∑ i = 1 n ( ( y ( i ) − w ⃗ T x ⃗ i ) 2 \frac{1}{2}\sum^n_{i=1}((y^{(i)}-\vec w^T \vec x_i)^2 2 1 ∑ i = 1 n ( ( y ( i ) − w T x i ) 2
此结果即为均方误差，因此用这个值作为代价函数来优化模型在统计学的角度是合理的。
