Week 36: Statistical interpretation of Linear Regression and Resampling techniques
Contents
Plans for week 36
Material for the active learning sessions Tuesday and Wednesday
Linear Regression and the SVD
What does it mean?
And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
Code for SVD and Inversion of Matrices
Inverse of Rectangular Matrix
Ridge and LASSO Regression
From OLS to Ridge and Lasso
Deriving the Ridge Regression Equations
Note on Scikit-Learn
Comparison with OLS
SVD analysis
Interpreting the Ridge results
More interpretations
Deriving the Lasso Regression Equations
Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
Ridge Regression
Lasso Regression
Yet another Example
The OLS case
The Ridge case
Writing the Cost Function
Lasso case
The first Case
Simple code for solving the above problem
With Lasso Regression
Another Example, now with a polynomial fit
Material for lecture Thursday September 7
Important technicalities: More on Rescaling data
Test Function for what happens with OLS, Ridge and Lasso
Linking the regression analysis with a statistical interpretation
Assumptions made
Expectation value and variance
Expectation value and variance for \( \boldsymbol{\beta} \)
Deriving OLS from a probability distribution
Independent and Identically Distrubuted (iid)
Maximum Likelihood Estimation (MLE)
A new Cost Function
More basic Statistics and Bayes' theorem
Marginal Probability
Conditional Probability
Bayes' Theorem
Interpretations of Bayes' Theorem
Example of Usage of Bayes' theorem
Doing it correctly
Bayes' Theorem and Ridge and Lasso Regression
Ridge and Bayes
Lasso and Bayes
Ridge Regression
For Ridge regression our cost function is
$$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$
and minimizing we have that
$$ \hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. $$
«
1
...
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
...
49
»