Logistic Regression

1. Problems of Linear Regression When Applied to Classification Problem

1) h(x) may out of range
2) some unusual feature values lead to failure of classification

2. Logistic Regression Model

1)hθ(x)=g(θTx)=P(y=1|x;θ)$1)h_{\theta }(x)=g({\theta }^{T}x)=P(y=1|x;\theta )$

where g(z)=11+e−z$g(z)=\frac{1}{1+e^{-z}}$ is called Sigmoid Function / Logistic Function

3. Decision Boundary

y=1 → hθ(x)>0.5$h_{\theta }(x)>0.5$ → θTx>0${\theta }^{T}x>0$
y=0 → hθ(x)<0.5$h_{\theta }(x)<0.5$ → θTx<0${\theta }^{T}x<0$
decision boundary: hθ(x)=0.5$h_{\theta }(x)=0.5$ → θTx=0${\theta }^{T}x=0$ (may be nonlinear)

4. Cost Function

where h=g(Xθ)$h=g(X\theta )$

5.Iteration Formula

vectorized formula:

(identical to linear regression)

6. Some Optimization Algorithms

Conjugate Gradient / BFGS / L-BFGS
No need to pick α and faster, but more complex

7. Multiclass Classification: one-vs-all

Train h(i)θ$h_{\theta }^{(i)}$ for every individual i.
When predicting, using maxi(h(i)θ(x))$ma{x}_i(h_{\theta }^{(i)}(x))$

8.Overfitting Problems

underfit — high bias — too few features
overfit — high variance — too many features —- fail to predict

2 solutions:
1) Reduce features
2) Regularization: Keep all features while reduce the values of some features

9. Regularization

adding   λ2m∑nj=1θ2j$\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2$    to    J(θ)$J(\theta )$
Note that it does not contain θ0${\theta }_0$ !
λ$\lambda$ : regularization parameter, making θ$\theta$ small

10. Regularized Linear Regression

(1) Linear Regression

J(θ)=12m(∑mi=1(h(x(i))−y(i))2+λ∑nj=1θ2j)$J(\theta )=\frac{1}{2m}(\sum _{i=1}^m(h(x^{(i)})-y^{(i)}{)}^2+\lambda \sum _{j=1}^n{\theta }_j^2)$
Note that it does not contain θ0${\theta }_0$ !

θj:=θj−α(1m∑mi=1(h(x(i))−y(i))∗x(i)j+λmθj)${\theta }_j:={\theta }_j-\alpha (\frac{1}{m}\sum _{i=1}^m(h(x^{(i)})-y^{(i)})\ast x_j^{(i)}+\frac{\lambda }{m}{\theta }_j)$ for j≠0$j\ne 0$
which is also
θj:=θj(1−αλm)−α∗1m∑mi=1(h(x(i))−y(i))∗x(i)j+${\theta }_j:={\theta }_j(1-\alpha \frac{\lambda }{m})-\alpha \ast \frac{1}{m}\sum _{i=1}^m(h(x^{(i)})-y^{(i)})\ast x_j^{(i)}+$ for j≠0$j\ne 0$

(2) Normal Equation

θ=(XTX+λdiag(0,1,1,...1,1))−1XTy$\theta =(X^{T}X+\lambda diag(0,1,1,...1,1){)}^{-1}{X}^{T}y$ where size of diag() is (n+1)*(n+1)