Improvements and Diagnostics on Algorithms

1. How to Evaluate A Hypothesis

Split training set into 2 parts: training set + test set
If Jtest(θ)$J_{test}(\theta )$ high, J(θ)$J(\theta )$ low, then overfitting occurs.


Linear Regression Test Error:
Same as J(θ)$J(\theta )$

Logistic Regression Test Error:

then

TestError=1Mtest∑err(hΘ(x),y)$$TestError=\frac{1}{M_{test}}\sum err(h_{\mathrm{\Theta }}(x),y)$$

2. Model Selection

Split training set into 3 parts: training set + cross validation set (CV) + test set
1) Optimize the parameters in Θ using the training set for each polynomial degree.
2) Find the polynomial degree d with the least error using the cross validation set.
3) Estimate the generalization error using the test set with Jtest(Θ(d))$J_{test}({\mathrm{\Theta }}^{(d)})$, (d$d$ = theta from polynomial with lower error);
In reality, CV set and test set should be randomly picked!

3. Diagnosing Bias & Variance

Training error decreases with d$d$ increases.
Validation error first decreases, then increases as d$d$ becomes bigger.

High Bias:
JCV(θ)≈Jtrain(θ)$J_{CV}(\theta )\approx J_{train}(\theta )$ is high
High Variance:
JCV(θ)$J_{CV}(\theta )$ high, Jtrain(θ)$J_{train}(\theta )$ low

4. Choosing λ$\lambda$ When Doing Regularization

Try λ:=λ∗2$\lambda :=\lambda \ast 2$, Pick the one wth least JCV(θ)$J_{CV}(\theta )$ and see its test error
High Bias:
JCV(θ)≈Jtrain(θ)$J_{CV}(\theta )\approx J_{train}(\theta )$ is high, λ$\lambda$ is big
High Variance:
JCV(θ)$J_{CV}(\theta )$ high, Jtrain(θ)$J_{train}(\theta )$ low, λ$\lambda$ is small

5. Learning Curves

x-axis is m, y-axis is error

High Bias:

If bias is high, adding more training data won’t help.

High Variance:

If variance is high, adding more training data may help.

6. Solutions for Bias & Variance

High Bias:
-more features;
-more polynomials;
-decreasing λ$\lambda$

High Variance:
-more examples;
-less features;
-increasing λ$\lambda$

7.Bias & Variance for Neural Network

Small Network: High Bias
Big Network: High Variance, using λ$\lambda$ doing regularization

8. Error Metrics: Precision & Recall

Put y=1 in presence of rare classes.
- Precision: Of all y=1 predictions, how many are correctly detected?
- Recall: Of all the rare cases, how many are correctly detected?

How to compare precision and recall? Using F score.
F score = 2PRP+R$2\frac{PR}{P+R}$