GeekCoding101 make your way to geek

Supervised ML Day 2 & 3: Gradient Descent and Learning Rate

Gradient descent minimizes a cost function J(w, b) by repeatedly moving w and b a small step against their partial derivatives, with the step size set by the learning rate α. If α is too small, it…

6 min read
Image 35
On this pageWhat I studied on Day 2 and Day 3

Gradient descent minimizes a cost function J(w, b) by repeatedly moving w and b a small step against their partial derivatives, with the step size set by the learning rate α. If α is too small, it crawls toward the minimum. If α is too big, it overshoots and can fail to converge or even diverge. That was the big idea of my Days 2 and 3 in Andrew Ng’s Supervised Machine Learning course, and by the end of Day 3 I had finished Week 1.

Part of my series “On My Way To Becoming A Machine Learning Person”. Published Apr 16, 2024.

What I studied on Day 2 and Day 3

  • Day 2: A busy day. I managed only 15 minutes of the “Supervised Machine Learning” videos, plus 15 minutes of But what is a neural network? | Chapter 1, Deep learning from 3Blue1Brown.
  • Day 3: 30+ minutes of the course, plus some reading. One article was Parul Pandey’s Understanding the Mathematics behind Gradient Descent, which is really good. I like math 😂

So these notes mix the course with outside reading.

Writing the notation in LaTeX

I struggled to write the formulas in LaTeX, then found this table useful:

LaTeX notation table

Do you need calculus for this course?

Andrew said we don’t need to worry about derivatives and calculus at all. I trust him. I still dug the Advanced Mathematics books I used in college out of my bookcase and spent 15 minutes reviewing them. Verdict: yes, I don’t need them.

Advanced Mathematics book Advanced Mathematics book, second page

The derivative tells you which way is downhill

If we are able to compute the derivative of a function, we know in which direction to proceed to minimize it (for the cost function).

— Parul Pandey, Understanding the Mathematics behind Gradient Descent

Parul briefly introduces two rules. Luckily, I still remember both from college:

  • Power rule: d/dx xⁿ = n·xⁿ⁻¹
  • Chain rule: d/dx f(g(x)) = f′(g(x)) · g′(x)

After reading several explanations of gradient descent, I really appreciate Andrew’s simple and precise approach. He can be playful: he draws a stick man walking down a hill step by step and suggests imagining flowers in the valley and clouds in the sky. The method was so relaxed and engaging that I forgot I was in learning mode.

The gradient descent update rule

This is the part I need to master by the end:

repeat until convergence: {w=w−α∂J(w,b)∂wb=b−α∂J(w,b)∂b}\text{repeat until convergence: } \left\{ \begin{aligned} w &= w - \alpha \frac{\partial J(w,b)}{\partial w} \\ b &= b - \alpha \frac{\partial J(w,b)}{\partial b} \end{aligned} \right\}
  • α (alpha) is the learning rate. It controls the size of each step.
  • ∂J/∂w and ∂J/∂b are partial derivatives. They give the slope of the cost in each direction, and subtracting them moves you downhill.

For linear regression, the model is f(x) = w·x + b. The squared error cost is J(w,b) = (1/2m) Σ (f(x⁽ⁱ⁾) − y⁽ⁱ⁾)², where m is the number of training examples. Its derivatives are:

∂J∂w=1m∑i=1m(f(x(i))−y(i))x(i)∂J∂b=1m∑i=1m(f(x(i))−y(i))\frac{\partial J}{\partial w} = \frac{1}{m}\sum_{i=1}^{m}\left(f(x^{(i)}) - y^{(i)}\right)x^{(i)} \qquad \frac{\partial J}{\partial b} = \frac{1}{m}\sum_{i=1}^{m}\left(f(x^{(i)}) - y^{(i)}\right)

Update w and b at the same time

Both updates must use the old values of w and b. Compute both new values first, then assign them. Using temp variables like this is standard practice in programming:

# Correct: simultaneous update
tmp_w = w - alpha * dj_dw
tmp_b = b - alpha * dj_db
w, b = tmp_w, tmp_b
# Wrong: b's gradient would then be computed with the new w
# w = w - alpha * dj_dw
# b = b - alpha * dj_db (if dj_db is recomputed after w changed)

A minimal sketch of the whole loop for one-feature linear regression:

def compute_gradient(x, y, w, b):
m = len(x)
dj_dw, dj_db = 0.0, 0.0
for i in range(m):
err = (w * x[i] + b) - y[i]
dj_dw += err * x[i]
dj_db += err
return dj_dw / m, dj_db / m
def gradient_descent(x, y, w, b, alpha, num_iters):
for _ in range(num_iters):
dj_dw, dj_db = compute_gradient(x, y, w, b)
w, b = w - alpha * dj_dw, b - alpha * dj_db # simultaneous
return w, b

Learning rate: too small vs. too big

Learning rate αWhat happens
Too smallBaby steps. It works, but takes a long time to reach the minimum.
Too bigBig strides. It may overshoot and never reach the minimum, failing to converge or even diverging.
Well chosenThe cost goes down steadily and settles at the minimum.

Why the steps shrink near a minimum

What happens when w is close to a local minimum?

  1. The derivative gets smaller because the slope flattens out.
  2. So the update step α × derivative also gets smaller.
  3. Gradient descent can therefore reach the minimum without you having to lower the learning rate.

Andrew’s conclusion: this is the gradient descent algorithm, and you can use it to minimize any cost function J, not just the mean squared error cost we use for linear regression.

Batch gradient descent and the convex cost function

  • Batch gradient descent: “batch” means each step uses all the training examples.
  • Convex function: the squared error cost for linear regression is bowl-shaped, so it has a single global minimum and no other local minima to get stuck in.

Terminology

TermComments
Squared error cost functionThe cost J(w,b) used for linear regression.
Local minimumA point lower than everything around it, but not necessarily the lowest overall.
Tangent lineAndrew introduced it to show how the derivative affects the cost. Its slope is the derivative at that point.
Converge / DivergeConverge: the steps settle at a minimum. Diverge: the cost grows instead of shrinking.
Convex functionHas a single global minimum because of its bowl shape. The squared error cost for linear regression is convex.
Batch gradient descentEach step of gradient descent uses all the training examples.

Finished Week 1!

It’s a milestone for me! I finished the first week’s course in 3 days (technically two). I love ML! Let’s keep the momentum going!

FAQ

What does the learning rate do in gradient descent? It sets the size of each step. If it is too small, training is slow. If it is too big, the steps overshoot the minimum and may never converge, or may diverge.

Why must w and b be updated simultaneously? Both partial derivatives should be computed from the same old values of w and b. Compute both updates first, store them in temp variables, then assign them.

What is batch gradient descent? It is gradient descent where every step uses all the training examples to compute the gradient.

Does gradient descent always find the global minimum? Not for every cost function. It can settle in a local minimum. The squared error cost for linear regression is convex, so there is only one minimum, and it is the global one.

Do you need calculus for Andrew Ng’s Supervised Machine Learning course? Andrew says you don’t need to worry about derivatives and calculus. Knowing the power rule and chain rule still helps the update rule make sense.

Keep reading

Thanks for reading! More posts from this series:

  • Master Gradient Descent and Binary Classification: Supervised Machine Learning – Day 9
  • Grinding Through Logistic Regression: Exploring Supervised Machine Learning – Day 10
  • Overfitting! Unlocking the Last Key Concept in Supervised Machine Learning – Day 11, 12

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts