Regularization and Bias/Variance
Regularization and Bias/Variance

Regularization can avoid underfitting/overfitting. But how it does acttually affect the learning algorithms

Remember the regularization indexes from 1

Set lambda = 1000, and each parameters will be highly penalized and will be tend to flat graph, resulting to underfitting

In contrast, set lambda to 0, the parameters will not be penalized and resulting in overfitting problems

So how we choose the correct value of regularization (lambda)?

Using extra lambda, just using average of the training set

Jtrain, Jcv,Jtest in earlier without the regularization

Try variant range of lambda by multiple sets of two

Iterate each model of theta use it to cost function

Use the theta into cross validation set

Pick whichever model that has the lowest value in cross validation error

And compare it to Jtest error

Concretely, use the selection model of thetas with selection of lambda,(model 5 with lambda no.5), and pick whichever has the lowest error of Jcv

That's the summary of model selection for regularization