Week 38: Logistic Regression and Optimization
Contents
Plans for week 38
Material from last week and relevant for the first project
Various steps in cross-validation
How to set up the cross-validation for Ridge and/or Lasso
Cross-validation in brief
Code Example for Cross-validation and \( k \)-fold Cross-validation
Material for lecture Thursday September 21
Logistic Regression
Classification problems
Optimization and Deep learning
Basics
Linear classifier
Some selected properties
Simple example
Plotting the mean value for each group
The logistic function
Examples of likelihood functions used in logistic regression and nueral networks
Two parameters
Maximum likelihood
The cost function rewritten
Minimizing the cross entropy
A more compact expression
Extending to more predictors
Including more classes
More classes
Friday September 23
Searching for Optimal Regularization Parameters \( \lambda \)
Grid Search
Randomized Grid Search
Wisconsin Cancer Data
Using the correlation matrix
Discussing the correlation data
Other measures in classification studies: Cancer Data again
Optimization, the central part of any Machine Learning algortithm
Revisiting our Logistic Regression case
The equations to solve
Solving using Newton-Raphson's method
Brief reminder on Newton-Raphson's method
The equations
Simple geometric interpretation
Extending to more than one variable
Steepest descent
More on Steepest descent
The ideal
The sensitiveness of the gradient descent
Convex functions
Convex function
Conditions on convex functions
More on convex functions
Some simple problems
Revisiting our first homework
Gradient descent example
The derivative of the cost/loss function
The Hessian matrix
Simple program
Gradient Descent Example
And a corresponding example using
scikit-learn
Gradient descent and Ridge
The Hessian matrix for Ridge Regression
Program example for gradient descent with Ridge Regression
Using gradient descent methods, limitations
Challenge yourself the coming weekend
Plans for week 38
Lecture from last week on the bias-variance tradeoff
Resampling techniques, cross-validation examples included here, see also the lectures from last week on the bootstrap method
Exercise for week 38, see also the whiteboard notes from week 37 at
https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep14.pdf
Work on project 1, in particular resampling methods like cross-validation and bootstrap.
Logistic regression as our first encounter of classification methods. From binary cases to several categories.
Start gradient and optimization methods
Video of lecture
Whiteboard notes at
https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep21.pdf
Readings and Videos:
Hastie et al 4.1, 4.2 and 4.3 on logistic regression
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
See also the whiteboard notes from week 37 at
https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep14.pdf
for a discussion and derivation of the bias-variance tradeoff.
Video on Logistic regression
Yet another video on logistic regression
Video on gradient descent
«
1
2
3
4
5
6
7
8
9
10
11
...
63
»