Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
Contents
Plans for week 35
Reading recommendations:
Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week
The equations for ordinary least squares
The cost/loss function
Interpretations and optimizing our parameters
Interpretations and optimizing our parameters
Some useful matrix and vector expressions
The Jacobian
Derivatives, example 1
Example 2
Example 3
Example 4
The mean squared error and its derivative
Other useful relations
Meet the Hessian Matrix
Interpretations and optimizing our parameters
Example relevant for the exercises
Own code for Ordinary Least Squares
Adding error analysis and training set up
Splitting our Data in Training and Test data
The complete code with a simple data set
Making your own test-train splitting
Reducing the number of degrees of freedom, overarching view
Preprocessing our data
Functionality in Scikit-Learn
More preprocessing
Frequently used scaling functions
Example of own Standard scaling
Min-Max Scaling
Testing the Means Squared Error as function of Complexity
More preprocessing examples, two-dimensional example, the Franke function
To think about, first part
More thinking
Still thinking
What does centering (subtracting the mean values) mean mathematically?
Further Manipulations
Wrapping it up
Linear Regression code, Intercept handling first
The Boston housing data example
Housing data, the code
Material for lecture Thursday, August 31
Mathematical Interpretation of Ordinary Least Squares
Residual Error
Simple case
The singular value decomposition
Linear Regression Problems
Fixing the singularity
Basic math of the SVD
The SVD, a Fantastic Algorithm
Economy-size SVD
Codes for the SVD
Note about SVD Calculations
Mathematics of the SVD and implications
Example Matrix
Setting up the Matrix to be inverted
Further properties (important for our analyses later)
Meet the Covariance Matrix
Introducing the Covariance and Correlation functions
Covariance and Correlation Matrix
Correlation Function and Design/Feature Matrix
Covariance Matrix Examples
Correlation Matrix
Correlation Matrix with Pandas
Correlation Matrix with Pandas and the Franke function
Rewriting the Covariance and/or Correlation Matrix
Linking with the SVD
What does it mean?
And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
Ridge and LASSO Regression
Deriving the Ridge Regression Equations
Interpreting the Ridge results
More interpretations
Deriving the Lasso Regression Equations
Material for lecture Thursday, August 31
«
1
...
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
...
74
»