Regularization and Bias/Variance

  |   Source
Regularization and Bias/Variance
  • Regularization can avoid underfitting/overfitting. But how it does acttually affect the learning algorithms

  • Remember the regularization indexes from 1
  • Set lambda  = 1000, and each parameters will be highly penalized and will be tend to flat graph, resulting to underfitting
  • In contrast, set lambda to 0, the parameters will not be penalized and resulting in overfitting problems
  • So how we choose the correct value of regularization (lambda)?

  • Using extra lambda, just using average of the training set
  • Jtrain, Jcv,Jtest in earlier without the regularization


  • Try variant range of lambda by multiple sets of two
  • Iterate each model of theta use it to cost function
  • Use the theta into cross validation set
  • Pick whichever model that has the lowest value in cross validation error
  • And compare it to Jtest error
  • Concretely, use the selection model of thetas with selection of lambda,(model 5 with lambda no.5), and pick whichever has the lowest error of Jcv
  • That's the summary of model selection for regularization

  • These show how variance/bias vary based on variant of regularization parameter
  • lambda small, regularization not being used == overfitting
  • lambda high, regularization highly used == underfitting


  • These are the example where the higher the lambda, making more underfitting, that is the cost function of Jtheta in training set is  higher

  • In choosing lambda, often plotting the graph .... making a better intuition of choosing the right lambda
  • Bias and variance by now is seen from a lot of different perspective
  • Next learning curve as a tool to indentify whether the learning algorithm has bias/variance problem