A program is a chain of elementary operations. Pick a mode and press Step: forward mode carries a derivative
along with every value, reverse mode first computes all values and then walks back from the output.
Count how many passes each mode needs for the whole gradient.
|
|
The computational graph
Each box is one elementary operation. Black: its value. Blue: derivative with respect to the seeded input
(flows left to right).Red: derivative of the output with respect to this box (flows right to left).
Numbers on the arrows are the local derivatives of one operation.
Result and cost
The finite difference column is the gradient check from last class.