Backpropagation Intuition

  |   Source
Backpropagation Intuition
  • Make a better intuition about backpropagation
  • Technically a little more complicated

  • As from example, there's two unit + 1 unit biased unit from forward propagation
  • So here's in the example we first take x(1) take it to the neural networks down to the output.
  • z(3)1 is the sigmoid function in layer 3 by take summation of (matrix_weight(2)10 x 1(bias_unit) + matrix_weight(2)11 x activation_unit(2)1).
  • eventually a(4)1 is the prediction
  • Backpropagation is doing the reverse of the forward. The process is really similar

  • The formula of cost function
  • if multiclass, then the formula will be added with summation of K unit classification.
  • Because we are doing example of 1 output unit, and 1 example, we are also ignoring the regularization term
  • For the purpose of the intuition, log will be ignored. We just want to know how close our network in predicting the output

  • delta can be thought as an error for every activation value
  • delta is actually the partial derivative of z, that if we change z, change the cost function, and eventually changing the actual cost
  • The first step is intuitive, first final error  = the final actual ouput - the final predicted output
  • Then keep going backwards from last layer to first hidden layer
  • By going reverse (from right to left) we are acquiring the delta (error value) by calculating [the previous error_value * matrix_weight]
  • Layer indexed from 1, the input layer

  • Why? We don't know the value of d(4)1
  • Next, give a little better intuition about backpropagation
  • Very effective algorithm eventhough a little harder to visualize