The main aim of this project is to study both classification and regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied in project 1. We will also include logistic regression for classification problems and write our own FFNN code for studying both regression and classification problems. The codes developed in project 1, including bootstrap and/or cross-validation as well as the computation of the mean-squared error and/or the \( R2 \) or the accuracy score (classification problems) functions can also be utilized in the present analysis.
The data sets that we propose here are (the default sets)
In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the Scikit-Learn site or at the University of California at Irvine.
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
In order to get started, we will now replace in our standard ordinary least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion algorithm with our own SGD code. You can choose whether you want to add the momentum SGD optionality or other SGD variants such as RMSprop or ADAgrad. The lecture notes from week 40 contain more details
Perform an analysis of the results for OLS and Ridge regression as function of the chosen learning rates, the number of mini-batches and epochs as well as algorithm for scaling the learning rate. You can also compare your own results with those that can be obtained using for example Scikit-Learn's various SGD options. Discuss your results. For Ridge regression you need now to study the results as functions of the hyper-parameter \( \lambda \) and the learning rate \( \gamma \). Discuss your results.
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.
Your aim now, and this is the central part of this project, is to write your own Feed Forward Neural Network code implementing the back propagation algorithm discussed in the lecture slides from week 41.
We will focus on a regression problem first and study either the Franke function or terrain data (or both or other data sets) from project 1. Discuss again your choice of cost function.
Write an FFNN code for regression with a flexible number of hidden layers and nodes using the Sigmoid function as activation function for the hidden layers. Initialize the weights using a normal distribution. How would you initialize the biases? And which activation function would you select for the final output layer?
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1. You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes from week 41) or tensorflow/keras.
Comment your results and give a critical discussion of the results obtained with the Linear Regression code and your own Neural Network code. Compare the results with those from project 1. Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and \( R2 \) scores.
A useful reference on the back progagation algorithm is Nielsen's book. It is an excellent read.
You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.
With a well-written code it should now be easy to change the activation function for the output layer.
Here we will change the cost function for our neural network code developed in parts b) and c) in order to perform a classification analysis.
We will here study the MNIST data set of hand-written numbers as discussed in the lecture notes from week 41. Use the Softmax function as activation function. Your code should however also be able to use a binary activation function as well.
To measure the performance of our classification problem we use the so-called accuracy score. The accuracy is as you would expect just the number of correctly guessed targets \( t_i \) divided by the total number of targets, that is $$ \text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} , $$
where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \) otherwise if we have a binary classification problem. Here \( t_i \) represents the target and \( y_i \) the outputs of your FFNN code and \( n \) is simply the number of targets \( t_i \).
Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \( \lambda \) (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
As stated in the introduction, it can also be useful to study other datasets. In particular, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the Scikit-Learn site or at the University of California at Irvine.
Again, we strongly recommend that you compare your own neural Network code for classification and pertinent results against a similar code using Scikit-Learn or tensorflow/keras or pytorch.
Finally, we want to compare the FFNN code we have developed with Logistic regression, that is we wish to compare our neural network classification results with the results we can obtain with another method.
Define your cost function and the design matrix before you start writing your code. Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates. Add also an \( l_2 \) regularization parameter \( \lambda \). Compare your results with those from your FFNN code as well as those obtained using Scikit-Learn's logistic regression functionality.
The weblink here https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article.
After all these glorious calculations, you should now summarize the various algorithms and come with a critical evaluation of their pros and cons. Which algorithm works best for the regression case and which is best for the classification case. These codes can also be part of your final project 3, but now applied to other data sets.
Here follows a brief recipe and recommendation on how to write a report for each project.
The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:
If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages, we recommend that you install the following Python packages via pip as
See below for a discussion of tensorflow and scikit-learn.
For OSX users we recommend also, after having installed Xcode, to install brew. Brew allows for a seamless installation of additional software via for example
If you don't want to install various Python packages with their dependencies separately, we recommend two widely used distrubutions which set up all relevant dependencies for Python, namely