Gradient descent minimizes a cost function J(w, b) by repeatedly moving w and b a small step against their partial derivatives, with the step size set by the learning rate α. If α is too small, it crawls toward the minimum. If α is too big, it overshoots and can fail to converge or even diverge. That was the big idea of my Days 2 and 3 in Andrew Ng’s Supervised Machine Learning course, and by the end of Day 3 I had finished Week 1.
Part of my series “On My Way To Becoming A Machine Learning Person”. Published Apr 16, 2024.
What I studied on Day 2 and Day 3
- Day 2: A busy day. I managed only 15 minutes of the “Supervised Machine Learning” videos, plus 15 minutes of But what is a neural network? | Chapter 1, Deep learning from 3Blue1Brown.
- Day 3: 30+ minutes of the course, plus some reading. One article was Parul Pandey’s Understanding the Mathematics behind Gradient Descent, which is really good. I like math 😂
So these notes mix the course with outside reading.
Writing the notation in LaTeX
I struggled to write the formulas in LaTeX, then found this table useful:
Do you need calculus for this course?
Andrew said we don’t need to worry about derivatives and calculus at all. I trust him. I still dug the Advanced Mathematics books I used in college out of my bookcase and spent 15 minutes reviewing them. Verdict: yes, I don’t need them.
The derivative tells you which way is downhill
If we are able to compute the derivative of a function, we know in which direction to proceed to minimize it (for the cost function).
— Parul Pandey, Understanding the Mathematics behind Gradient Descent
Parul briefly introduces two rules. Luckily, I still remember both from college:
- Power rule: d/dx xⁿ = n·xⁿ⁻¹
- Chain rule: d/dx f(g(x)) = f′(g(x)) · g′(x)
After reading several explanations of gradient descent, I really appreciate Andrew’s simple and precise approach. He can be playful: he draws a stick man walking down a hill step by step and suggests imagining flowers in the valley and clouds in the sky. The method was so relaxed and engaging that I forgot I was in learning mode.
The gradient descent update rule
This is the part I need to master by the end:
- α (alpha) is the learning rate. It controls the size of each step.
- ∂J/∂w and ∂J/∂b are partial derivatives. They give the slope of the cost in each direction, and subtracting them moves you downhill.
For linear regression, the model is f(x) = w·x + b. The squared error cost is J(w,b) = (1/2m) Σ (f(x⁽ⁱ⁾) − y⁽ⁱ⁾)², where m is the number of training examples. Its derivatives are:
Update w and b at the same time
Both updates must use the old values of w and b. Compute both new values first, then assign them. Using temp variables like this is standard practice in programming:
# Correct: simultaneous updatetmp_w = w - alpha * dj_dwtmp_b = b - alpha * dj_dbw, b = tmp_w, tmp_b
# Wrong: b's gradient would then be computed with the new w# w = w - alpha * dj_dw# b = b - alpha * dj_db (if dj_db is recomputed after w changed)A minimal sketch of the whole loop for one-feature linear regression:
def compute_gradient(x, y, w, b): m = len(x) dj_dw, dj_db = 0.0, 0.0 for i in range(m): err = (w * x[i] + b) - y[i] dj_dw += err * x[i] dj_db += err return dj_dw / m, dj_db / m
def gradient_descent(x, y, w, b, alpha, num_iters): for _ in range(num_iters): dj_dw, dj_db = compute_gradient(x, y, w, b) w, b = w - alpha * dj_dw, b - alpha * dj_db # simultaneous return w, bLearning rate: too small vs. too big
| Learning rate α | What happens |
|---|---|
| Too small | Baby steps. It works, but takes a long time to reach the minimum. |
| Too big | Big strides. It may overshoot and never reach the minimum, failing to converge or even diverging. |
| Well chosen | The cost goes down steadily and settles at the minimum. |
Why the steps shrink near a minimum
What happens when w is close to a local minimum?
- The derivative gets smaller because the slope flattens out.
- So the update step α × derivative also gets smaller.
- Gradient descent can therefore reach the minimum without you having to lower the learning rate.
Andrew’s conclusion: this is the gradient descent algorithm, and you can use it to minimize any cost function J, not just the mean squared error cost we use for linear regression.
Batch gradient descent and the convex cost function
- Batch gradient descent: “batch” means each step uses all the training examples.
- Convex function: the squared error cost for linear regression is bowl-shaped, so it has a single global minimum and no other local minima to get stuck in.
Terminology
| Term | Comments |
|---|---|
| Squared error cost function | The cost J(w,b) used for linear regression. |
| Local minimum | A point lower than everything around it, but not necessarily the lowest overall. |
| Tangent line | Andrew introduced it to show how the derivative affects the cost. Its slope is the derivative at that point. |
| Converge / Diverge | Converge: the steps settle at a minimum. Diverge: the cost grows instead of shrinking. |
| Convex function | Has a single global minimum because of its bowl shape. The squared error cost for linear regression is convex. |
| Batch gradient descent | Each step of gradient descent uses all the training examples. |
Finished Week 1!
It’s a milestone for me! I finished the first week’s course in 3 days (technically two). I love ML! Let’s keep the momentum going!
FAQ
What does the learning rate do in gradient descent? It sets the size of each step. If it is too small, training is slow. If it is too big, the steps overshoot the minimum and may never converge, or may diverge.
Why must w and b be updated simultaneously? Both partial derivatives should be computed from the same old values of w and b. Compute both updates first, store them in temp variables, then assign them.
What is batch gradient descent? It is gradient descent where every step uses all the training examples to compute the gradient.
Does gradient descent always find the global minimum? Not for every cost function. It can settle in a local minimum. The squared error cost for linear regression is convex, so there is only one minimum, and it is the global one.
Do you need calculus for Andrew Ng’s Supervised Machine Learning course? Andrew says you don’t need to worry about derivatives and calculus. Knowing the power rule and chain rule still helps the update rule make sense.
Keep reading
Thanks for reading! More posts from this series:
- Master Gradient Descent and Binary Classification: Supervised Machine Learning – Day 9
- Grinding Through Logistic Regression: Exploring Supervised Machine Learning – Day 10
- Overfitting! Unlocking the Last Key Concept in Supervised Machine Learning – Day 11, 12