Week 36: Linear Regression and Gradient descent
Contents
Plans for week 36
Material for lecture Monday September 2
Mathematical Interpretation of Ordinary Least Squares
Residual Error
Simple case
The singular value decomposition
Linear Regression Problems
Fixing the singularity
Ridge and LASSO Regression
Deriving the Ridge Regression Equations
Basic math of the SVD
The SVD, a Fantastic Algorithm
Economy-size SVD
Codes for the SVD
Note about SVD Calculations
Mathematics of the SVD and implications
Example Matrix
Setting up the Matrix to be inverted
Further properties (important for our analyses later)
Back to Ridge and LASSO Regression
Interpreting the Ridge results
More interpretations
Deriving the Lasso Regression Equations
Optimization and gradient descent, the central part of any Machine Learning algortithm
Reminder on Newton-Raphson's method
The equations
Simple geometric interpretation
Extending to more than one variable
Steepest descent
More on Steepest descent
The ideal
The sensitiveness of the gradient descent
Convex functions
Convex function
Conditions on convex functions
More on convex functions
Some simple problems
Revisiting Ordinary Least Squares
Gradient descent example
The derivative of the cost/loss function
The Hessian matrix
Simple program
Gradient Descent Example
Gradient descent and Ridge
The Hessian matrix for Ridge Regression
Program example for gradient descent with Ridge Regression
Using gradient descent methods, limitations
Material for lab sessions sessions Tuesday and Wednesday
Linear Regression and the SVD
What does it mean?
Ridge and LASSO Regression
From OLS to Ridge and Lasso
Deriving the Ridge Regression Equations
Note on Scikit-Learn
Comparison with OLS
SVD analysis
Interpreting the Ridge results
More interpretations
Deriving the Lasso Regression Equations
Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
Ridge Regression
Lasso Regression
Yet another Example
The OLS case
The Ridge case
Writing the Cost Function
Lasso case
The first Case
Simple code for solving the above problem
With Lasso Regression
Another Example, now with a polynomial fit
Plans for week 36
Material for the lecture on Monday September 1:
Linear Regression, ordinary least squares (OLS), Ridge and Lasso and mathematical analysis
Derivation of Gradient descent and discussion of implementations for
Video of lecture at
https://youtu.be/nVE_FRnGAHw
Whiteboard notes at
https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2025/FYSSTKweek36.pdf
Material for the lab sessions on Tuesday and Wednesday (see at the end of these slides):
Technicalities concerning Ridge and Lasso linear regression.
Presentation and discussion of the first project
Reading suggestion:
Goodfellow et al, Deep Learning, introduction to gradient descent, see chapter 4.3 at
https://www.deeplearningbook.org/contents/numerical.html
Rashcka et al, pages 37-44 and pages 278-283 with focus on linear regression.
Video on gradient descent at
https://www.youtube.com/watch?v=sDv4f4s2SB8
«
1
2
3
4
5
6
7
8
9
10
11
...
72
»