GeekCoding101 make your way to geek

Multiple Linear Regression & Vectorization – ML Day 4-5

Notes from Andrew Ng's Supervised ML course, days 4–5: multiple linear regression, dot product and NumPy vectorization, plus a self-quiz.

5 min read
Supervised Machine Learning Day4 5
On this pageSo difficult to manage some time on this

GeekCoding101 · Apr 18, 2024 · Topics: machine-learning, notes, numpy, supervised-machine-learning

Multiple linear regression extends linear regression from one input feature to many by writing the weights and features as vectors, so the model becomes f(x) = w · x + b. Vectorization means computing that dot product with NumPy (np.dot) instead of a for-loop, which gives shorter code that runs much faster on parallel hardware.

These are my notes from days 4 and 5 of Andrew Ng’s Supervised Machine Learning course. They cover the videos “Multiple features”, “Vectorization part 1” and “Vectorization part 2”, plus the lab.

So difficult to manage some time on this

Day 4 was a long day for me. I only had 15 minutes before bed to quickly skim through “Multiple features” and “Vectorization part 1”.

Day 5 was a longer day than the one before. I went to urgent care in the morning, then had back-to-back meetings after I came back. Then lunch, then back-to-back meetings again, then I needed to step out again…

Anyway, that’s life.

Multiple features (variables)

In “Multiple features”, Andrew uses crisp language to show how the multiple-features formula can be simplified with a vector and the dot product.

With one feature, the model is f(x) = wx + b. With n features, it becomes:

f(x) = w₁x₁ + w₂x₂ + … + wₙxₙ + b

If the weights and the features are each written as a vector, w = [w₁ … wₙ] and x = [x₁ … xₙ], the same model is just:

f(x) = w · x + b

Notation used in the course

SymbolMeaning
xⱼthe j-th feature
nnumber of features
x⁽ⁱ⁾all features of the i-th training example (a vector)
xⱼ⁽ⁱ⁾value of feature j in the i-th training example
w (with arrow)vector of weights, one per feature
ba single number (the bias)

Vectorization part 1: the dot product in NumPy

In “Part 1”, Andrew shows how to use NumPy to compute the dot product. He explains that NumPy can take advantage of the parallel hardware in your computer, and that GPUs are also good at this type of calculation.

import numpy as np
w = np.array([1.0, 2.5, -3.3])
b = 4
x = np.array([10, 20, 30])
# Without vectorization: one multiplication at a time
f = 0
for j in range(len(w)):
f = f + w[j] * x[j]
f = f + b
# With vectorization: one line, and much faster on large arrays
f = np.dot(w, x) + b

Both versions give the same result. The vectorized one is easier to read, and NumPy can run it in parallel instead of step by step.

Vectorization part 2: why it’s fast

In “Part 2”, Andrew explains why a computer can compute a dot product fast. A for-loop handles one pair of numbers per step. A vectorized operation multiplies all pairs at the same time and then adds the results together.

He uses gradient descent as the example. Every weight needs an update, wⱼ = wⱼ − α·dⱼ. Without vectorization, that is a loop over every feature. With vectorization, it is one line:

w = w - alpha * d # w and d are NumPy arrays of the same length

The lab

The lab was informative. I walked through all of it, even though I already knew most of it.

More links about vectorization can be found here: [keep: link to vectorization resources]

Questions for helping myself learning

I created the following questions to test my knowledge later.

Why is there an arrow hat on a variable? It’s optional, but nice to have. It shows the variable is a vector (a list of numbers), not a single number.

What does the dot product do? It takes two vectors of the same length, such as w and x, and multiplies each corresponding pair of numbers (w₁ with x₁, w₂ with x₂, and so on). Then it adds all those products together into a single number.

What is multiple linear regression? It’s the name for a linear regression model with multiple input features. It contrasts with “univariate regression”, which has only one feature.

Is multivariate regression the same as “multiple linear regression”? No. Multivariate regression is something else: it usually refers to predicting more than one output variable at a time. Multiple linear regression has many inputs but one output.

Why do we need vectorization? Vectorized code can run calculations in much less time than non-vectorized code, especially on specialized hardware. This matters most when you run algorithms on large data sets or train large models, which is often the case in machine learning.

vectorization and multiple feature

What is x₁⁽⁴⁾ in the graph above? Using the course notation, the superscript (4) means the 4th training example and the subscript 1 means the 1st feature. So x₁⁽⁴⁾ is the value of the first feature in the fourth training example: the cell in row 4, column 1 of the table.


P.S. Feel free to check out the rest of my Supervised Machine Learning journey series. Thanks for reading! Questions or corrections are welcome via the contact page.

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts