GeekCoding101 make your way to geek

Gradient Descent for Multiple Linear Regression: ML Day 6

Gradient descent for multiple linear regression updates every weight wⱼ by subtracting the learning rate times the average of (prediction − target) × xⱼ⁽ⁱ⁾, and updates b the same way but without the…

5 min read
Derivation Process
On this pageThe update rules (what Andrew put on the board)

Gradient descent for multiple linear regression updates every weight wⱼ by subtracting the learning rate times the average of (prediction − target) × xⱼ⁽ⁱ⁾, and updates b the same way but without the xⱼ⁽ⁱ⁾. That extra xⱼ⁽ⁱ⁾ comes straight from the chain rule. I learned this the hard way today, after being sure I’d found a mistake in Andrew’s course.

GeekCoding101 · Apr 19, 2024 · Topics: notes, supervised-machine-learning, machine-learning

Today I started by reviewing my previous notes, and the review alone took a good chunk of time. I’m realizing that’s a lot of notes already.

I am amazing 🤩

Then on to the topic of the day: “Gradient descent for multiple linear regression”.

The update rules (what Andrew put on the board)

At the beginning of the video, Andrew threw out the update rules and said he hoped we still remembered them. I didn’t.

gradient descent update rules implementing gradient descent

Written out, with the model f_w,b(x) = w · x + b and the cost function

J(w, b) = (1 / 2m) · Σᵢ₌₁ᵐ ( f_w,b(x⁽ⁱ⁾) − y⁽ⁱ⁾ )²

gradient descent repeats until convergence:

ParameterUpdate rule
each weight wⱼ (j = 1 … n)wⱼ = wⱼ − α · (1/m) · Σᵢ ( f_w,b(x⁽ⁱ⁾) − y⁽ⁱ⁾ ) · xⱼ⁽ⁱ⁾
bias bb = b − α · (1/m) · Σᵢ ( f_w,b(x⁽ⁱ⁾) − y⁽ⁱ⁾ )

All parameters are updated simultaneously. You compute every new value from the old w and b first, then assign them.

My question was simple: where does this come from? Why does xⱼ⁽ⁱ⁾ show up in one rule and not the other?

Going back to find it

I spent 30 minutes re-watching earlier videos before I found the source. The two that matter are:

  1. Week 1 – Implementing gradient descent. Andrew writes the update rules down here without explaining them (he explains later).
  2. Gradient descent for linear regression. This is where the derivative is worked out.

”Holy! Found a mistake in Andrew’s course!”

gradient descent for linear regression derivative of the cost function

Looking at that screenshot, I was convinced Andrew had lost the x⁽ⁱ⁾ at the end of the first line!

WOW! I ROCK! Spent almost 60 minutes on it!

I was done for the day!

The situation reversed an hour later

Then I felt upset. I was NOT convinced I had found such a simple mistake, especially in Andrew’s most popular machine learning course.

So I tried to work through the formula myself.

And… I found out I was indeed too young, too naive… Andrew was right.

I got help from my college classmate, who has been dealing with calculus every day for more than 20 years. This is the derivation he wrote out for me:

derivation of the formula

He said this to me just like my math teacher in college:

Chain rule, deriving step by step.

Why x⁽ⁱ⁾ appears: the chain rule, step by step

Here is the single-feature version, where f_w,b(x) = w·x + b. The multi-feature version works exactly the same way for each wⱼ.

Derivative with respect to w:

  1. Start from the cost: ∂J/∂w = ∂/∂w · (1/2m) · Σᵢ ( w·x⁽ⁱ⁾ + b − y⁽ⁱ⁾ )²
  2. Power rule on the outer square: bring the 2 down, which gives (1/2m) · Σᵢ 2 · ( w·x⁽ⁱ⁾ + b − y⁽ⁱ⁾ ) · [inner derivative]
  3. Chain rule on the inner part: ∂/∂w ( w·x⁽ⁱ⁾ + b − y⁽ⁱ⁾ ) = x⁽ⁱ⁾
  4. The 2 cancels the ½, which leaves (1/m) · Σᵢ ( f_w,b(x⁽ⁱ⁾) − y⁽ⁱ⁾ ) · x⁽ⁱ⁾

Derivative with respect to b:

Steps 1 and 2 are the same. In step 3, the inner derivative is ∂/∂b ( w·x⁽ⁱ⁾ + b − y⁽ⁱ⁾ ) = 1. That leaves (1/m) · Σᵢ ( f_w,b(x⁽ⁱ⁾) − y⁽ⁱ⁾ ), with no x⁽ⁱ⁾.

So the x⁽ⁱ⁾ isn’t missing from anything. It only appears once you take the derivative of the inside of the square, and for b that inside derivative is just 1.

Power rule and chain rule, again

If you remember, I mentioned Parul Pandey’s Understanding the Mathematics behind Gradient Descent in my earlier post, Supervised Machine Learning – Day 2 & 3 – On My Way To Becoming A Machine Learning Person. In her post she says:

Primarily we shall be dealing with two concepts from calculus: Power rule and chain rule.

Well, she is right as well 😁

FAQ

Why does xⱼ⁽ⁱ⁾ appear in the update for wⱼ but not for b? Because of the chain rule. The derivative of (w·x⁽ⁱ⁾ + b − y⁽ⁱ⁾) with respect to w is x⁽ⁱ⁾, but with respect to b it is 1.

Is gradient descent for multiple linear regression different from the single-feature version? Not really. You apply the same rule to each weight w₁ … wₙ, each using its own feature xⱼ⁽ⁱ⁾. The rule for b stays the same.

Where does the 1/m come from instead of 1/2m? The cost function has 1/2m. Differentiating the square brings down a 2, which cancels the ½.

Do I update w and b one after the other? No. Update them simultaneously: compute all the new values using the old parameters, then assign them together.

What calculus do I need for this? Mainly the power rule and the chain rule.

So happy I learned a lot today! Thanks for reading. Feel free to check out my other posts in this series, or reach me at /contact.

Keep reading in this series:

  • Master Gradient Descent and Binary Classification: Supervised Machine Learning – Day 9
  • Grinding Through Logistic Regression: Exploring Supervised Machine Learning – Day 10
  • Overfitting! Unlocking the Last Key Concept in Supervised Machine Learning – Day 11, 12

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts