chapter 27 Multiple features

start to talk about a new version of linear regression,more powerful one that works with multiple variables or with multiple features.

Notation:

n = number of features

x(i)$x^{(i)}$ = input (features) of i th training example.

x(i)j$x_j^{(i)}$ = value of feature j in i th training example.

Hypothesis:

hθ(x)=θ0+θ1x1+θ2x2+...+θnxn$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+...+{\theta }_n{x}_n$

for convenience of notation ,define x0$x_0$ = 1.

so

hθ(x)=θTx$h_{\theta }(x)={\theta }^{T}x$

Multivariate linear regression.

chapter 28 Gradient descent for multiple variables

how to fit the parameters of that hypothesis.how to use gradient descent for linear regression with multiple features

Hypothesis: hθ(x)=θTx=θ0x0+θ1x1+...+θnxn)$h_{\theta }(x)={\theta }^{T}x={\theta }_0{x}_0+{\theta }_1{x}_1+...+{\theta }_n{x}_n)$

Parameters: θ$\theta$ here is a n+1-dimensional vector.

Cost function:

J(θ)=12m∑mi=1(hθ(x(i))−y(i))$J(\theta )=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})$

Gradient descent:

​ Repeat{

θj:=θj−a∂∂θjJ(θ)${\theta }_j:={\theta }_j-a\frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }_j}J(\theta )$

​ } (simultaneously update for every j=0,...,n$j=0,...,n$)

New algorithm(n>=1):

Repeat{

θj:=θj−a1m∑mi=1(hθ(x(i))−y(i))${\theta }_j:={\theta }_j-a\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})$

}

θ0:=θ0−a1m∑mi=1(hθ(x(i))−y(i))x(i)0${\theta }_0:={\theta }_0-a\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)}$

θ1:=θ1−a1m∑mi=1(hθ(x(i))−y(i))x(i)1${\theta }_1:={\theta }_1-a\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_1^{(i)}$

chapter 29 Gradient descent in practice 1:feature scaling

practical tricks for making gradient descent work well

Feature Scaling:

Idea: Make sure features are on similar scale.

Get every feature into approximately a −1<=xi<=1$-1<=x_i<=1$ range.

Mean normalization

Replace xi$x_i$ with xi−ui$x_i-u_i$ to make features have approximately zero mean (Do not apply to x0=1$x_0=1$)

x1←x1−u1s1$x_1\leftarrow \frac{x_1-u_1}{s_1}$ u1$u_1$ is the average value of x1 in the training sets.

s1$s_1$ is the range of values of that feature or standard deviation

chapter 30 Gradient descent in practice 2:learning rate

around the learning rate a$a$

• “Debugging “:How to make sure gradient descent is working correctly.
• How to choose learning rate a$a$

Declare convergence if J(θ)$J(\theta )$ decreases by less than 10−3${10}^{-3}$ in one iteration.

but choose what this threshold is pretty difficult.So,in order to check your gradient descent has converged, actually tend to look at plots.

• For sufficiently small a$a$ ,J(θ)$J(\theta )$should decrease on every iteration.
• But if a$a$ is too small,gradient descent can be slow to converge.
• if a$a$ is too large ;J(θ)$J(\theta )$ may not decrease on every iteration ;may not converge.

chapter 31 Features and polynomial regression

the choice of features that you have and how you can get different learning algorithm

It is important to apply feature scaling if you’re using gradient descent to get them into comparable ranges of values.

broad choices in the features you use.

chapter 32 Normal equation

which for some linear regression problems ,will give us a much better way to solve for the optimal value of the parameters θ$\theta$

Normal equation : Method to solve for θ$\theta$ analytically.

正规方程推到过程：https://zhuanlan.zhihu.com/p/22474562

矩阵求导：https://blog.****.net/nomadlx53/article/details/50849941

Gradient Descent and Normal Equation advantages and disadvantages :

The normal equation method actually do not work for those more sophisticated learning algorithms.

chapter 33 Normal equation and non-invertibility (optional)

what if XTX$X^{T}X$ is non-invertible?

• Redundant features (linearly dependent)

e g : x1$x_1$ = size in feet2${}^2$ x2$x_2$ = size in m2${}^2$

• too many features (e.g m<=n)

Delete some features , or use regularization.