Regularization and Bias/Variance

  |   Source
Regularization and Bias/Variance
  • Regularization can avoid underfitting/overfitting. But how it does acttually affect the learning algorithms

  • Remember the regularization indexes from 1
  • Set lambda  = 1000, and each parameters will be highly penalized and will be tend to flat graph, resulting to underfitting
  • In contrast, set lambda to 0, the parameters will not be penalized and resulting in overfitting problems
  • So how we choose the correct value of regularization (lambda)?

  • Using extra lambda, just using average of the training set
  • Jtrain, Jcv,Jtest in earlier without the regularization


  • Try variant range of lambda by multiple sets of two
  • Iterate each model of theta use it to cost function
  • Use the theta into cross validation set
  • Pick whichever model that has the lowest value in cross validation error
  • And compare it to Jtest error
  • Concretely, use the selection model of thetas with selection of lambda,(model 5 with lambda no.5), and pick whichever has the lowest error of Jcv
  • That's the summary of model selection for regularization