Machine Learning and Data Analysis for Nuclear Physics, European ERASMUS+ Master of Science program

Why a course on Machine Learning for Nuclear Physics?

Probability theory and statistical methods play a central role in science. Nowadays we are surrounded by huge amounts of data. For example, there are about one trillion web pages; more than one hour of video is uploaded to YouTube every second, amounting to 10 years of content every day; the genomes of 1000s of people, each of which has a length of more than a billion base pairs, have been sequenced by various labs and so on. This deluge of data calls for automated methods of data analysis, which is exactly what machine learning provides. The purpose of thiscourse is to provide an introduction to the core concepts and tools of machine learning in a manner easily understood and intuitive to physicists and nuclear physicists in particular. We will start with some of the basic methods from supervised learning and statistical data analysis, such as various regression methods before we move into deep learning methods for both supervised and unsupervised learning, with an emphasis on the analysis of nuclear physics experiments and theoretical nuclear physics. The students will work on hands-on daily examples as well as projects than can result in final credits. Exercises and projects will be provided and the aim is to give the participants an overview on how machine learning can be used to analyze and study nuclear physics problems (experiment and theory). The major scope is to give the participants a deeper understanding on what Machine learning and Data Analysis are and how they can be used to analyze data from nuclear physics experiments and perform theoretical calculations of nuclear many-body systems.

The goals of this course on Machine Learning and Data Analysis are to give the participants a deeper understanding and critical view of several widely popular Machine Learning algorithms, covering both supervised and unsupervised learning. The learning outcomes involve an understanding of the following central methods:

• Basic concepts of machine learning and data analysis and statistical concepts like expectation values, variance, covariance, correlation functions and errors;
• Estimation of errors using cross-validation, blocking, bootstrapping and jackknife methods;
• Optimization of functions
• Linear Regression and Logistic Regression;
• Dimensionality reductions, from PCA to clustering
• Neural networks and deep learning;
• Convolutional Neural Networks
• Recurrent Neureal Networks and Autoencoders
• Decisions trees and random forests
• Support vector machines and kernel transformations

The course has two central parts

1. Statistical analysis and optimization of data
2. Machine learning

These topics will be scattered thorughout the course and may not necessarily be taught separately. Rather, we will often take an approach (during the lectures and project/exercise sessions) where say elements from statistical data analysis are mixed with specific Machine Learning algorithms.

Statistical analysis and optimization of data

The following topics will be covered

• Basic concepts, expectation values, variance, covariance, correlation functions and errors;
• Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;
• Central elements of Bayesian statistics and modeling;
• Gradient methods for data optimization,
• Estimation of errors and resampling techniques such as the cross-validation, blocking, bootstrapping and jackknife methods;
• Principal Component Analysis (PCA) and its mathematical foundation

Machine learning

The following topics will be covered:

• Linear Regression and Logistic Regression;
• Neural networks and deep learning, including convolutional and recurrent neural networks
• Decisions trees, Random Forests, Bagging and Boosting
• Support vector machines
• Unsupervised learning Dimensionality reduction, from PCA to cluster models

Hands-on demonstrations, exercises and projects aim at deepening your understanding of these topics.

Computational aspects play a central role and you are expected to work on numerical examples and projects which illustrate the theory and varous algorithms discussed during the lectures. We recommend strongly to form small project groups of 2-3 participants, if possible.

Prerequisites

Basic knowledge in programming and mathematics, with an emphasis on linear algebra. Knowledge of Python or/and C++ as programming languages is strongly recommended and experience with Jupiter notebook is recommended.

Practicalities

1. Lectures are Mondays from 815am-10am. Start September 25, 2022
2. Group meeting and work on projects and exercises to be determined
3. Grading scale: Grades are awarded on a scale from A to F, where A is the best grade and F is a fail. We are aiming at having two projects to be handed in. These will be graded and should be finalized not later than two weeks after the course is over. Both projects count 50% each of the final grade. We plan to make the grades available not later than February 1, hopefully the grades will be available before that.

Lecture material

The link https://compphysics.github.io/MLErasmus/doc/web/course.html gives you direct access to the learning material with lectures slides and jupyter notebooks. Videos of the lectures will be added.

Detailed Lecture Notes

Detailed notes at the link https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/intro.html

Teaching schedule, topics and teacher

Teachers: Morten Hjorth-Jensen (MHJ)

Monday September 25	- Lecture 815am-1000am: Introduction to Machine Learning and linear regression (MHJ)
Recommended readings	Goodfellow et al chapters 2 and 3; Bishop Sections 1.1, 1.2 and 1.3
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 1
	Video of Lecture TBA at https://youtu.be/NtHBoMoIBP8
	Handwritten notes at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesSep252023.pdf
	Video on using Git and GitHub https://mediaspace.msu.edu/media/t/1_8mgx3cyf
Monday October 2	- Lecture 815am-10am: Linear Regression, from ordinary least squares to Lasso and Ridge (MHJ)
Recommended readings	Goodfellow et al, Deep Learning, chapter 2 on linear algebra. Hastie et al, The elements of statistical learning, sections 3.1-3.4. Deisenroth et al, Mathematics for Machine Learning, see chapter 6, see https://mml-book.github.io/book/mml-book.pdf
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 2
	Video of Lecture at https://youtu.be/
	Handwritten notes at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesOct22023.pdf
Monday October 9	- Lecture 815am-10am: Linear Regression, Ridge and Lasso regression (MHJ)
Recommended readings	Bishop Section 3.2 and Hastie et al chapter 3
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 3
	Video of Lecture at https://youtu.be/SaQ1I-yyvIo
	Handwritten notes at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesOct92023.pdf
Monday October 16	- Lecture 815am-10am: Ridge and Lasso regression and statistical interpretations (MHJ)
Recommended readings	Hastie et al Chapter 3
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html sessions 3 and 4
	Video of Lecture at https://youtu.be/iqRKUPJr_bY
	Handwritten notes at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesOct162023.pdf
Monday October 23	- Lecture 815am-10am: Resampling Methods and Bias-Variance tradeoff (MHJ)
Recommended readings	Hastie et al chapter 7
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 4 material
	Video of Lecture at https://youtu.be/3DyqMQaMgvQ
	Handwritten notes at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesOct232023.pdf
Monday October 30	- Lecture 815am-10am: Resampling methods and Bias-variance tradeoff
Recommended readings	Hastie et al chapter 7
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 4 material
	Video of Lecture at https://youtu.be/-6Hpyj0dwC0
	Handwritten notes at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesOct302023.pdf
Monday November 6	- Lecture 815am-10am: Logistic Regression, Optimization and Gradient Descent, material from sessions 5 and 6
Recommended readings	Bishop Sections 5.1-5.5 and Hastie et al chapter 11, Goodfellow et al, chapter 4 and 8
	Video of Lecture, Morning session 8am-10am, https://youtu.be/NwVnnQ3SHD8
	Handwritten notes at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesNov62023.pdf
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html sessions 4 and 5
Sunday and Monday November 12-13	- Lecture 815am-10am: Optimization problems and start discussion of project 1
Recommended readings	Goodfellow et al, chaptes 4 and 8
	Video of Lecture from November 12 at https://youtu.be/e4bF7ztE0ic
	Handwritten notes November 12 at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesNov122023.pdf
	Video of Lecture from November 13 at https://youtu.be/8angq0E9-DA
	Handwritten notes November 13 at https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesNov132023.pdf
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 5
Monday November 20	- Lecture 815am-10am: Neural networks, basic elements
Recommended readings	Goodfellow et al, chapters 6-7, see also chapter 8
	Video of Lecture at https://youtu.be/0J0UEe-pPLc
	Handwritten notes https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesNov202023.pdf
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 6
Monday November 27	- Lecture 815am-10am: Neural networks, building our own code
Recommended readings	Goodfellow et al, chapters 6-7
	Video of Lecture at https://youtu.be/x7nHVCETJLs
	Handwritten notes https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesNov272023.pdf
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 6
Tuesday December 12	- Lecture 815am-10am: Solving differential equations with neural networks
Recommended readings	Slides for sessions 8
	Video of Lecture at https://youtu.be/FCPY9qdP5YU
	Handwritten notes https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesDec122023.pdf
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 8
Wednesday December 13	- Lecture 815am-10am: Convolutional Neural Networks
Recommended readings	Slides for sessions 7 and Goodfellow et al chapter 9, see also chapters 11 and 12
	Video of Lecture at https://youtu.be/IiBnw0Vrp0w
	Handwritten notes https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesDec132023.pdf
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 7
Monday December 18	- Lecture 815am-10am: Convolutional Neural Networks and Recurrent neural networks
Recommended readings	Slides for sessions 7 and Goodfellow et al chapters 9 and 10, see also chapters 11 and 12
	Video of Lecture at https://youtu.be/Us0z6ggPRbk
	Handwritten notes https://github.com/CompPhysics/MLErasmus/blob/master/doc/HandwrittenNotes/2023/NotesDec182023.pdf
	Lecture material at https://compphysics.github.io/MLErasmus/doc/web/course.html session 7
Wednesday January 17	- Lecture 815am-10am: Recurrent neural networks
Recommended readings	Slides for sessions 7 and Goodfellow et al chapter 10, see also chapters 11 and 12
	Video of Lecture https://www.youtube.com/watch?v=z0x-vgyAZUk&ab_channel=MortenHjorth-Jensen
	Handwritten notes https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesNov9.pdf

Required Technologies

Course participants are expected to have their own laptops/PCs. We use Git as version control software and the usage of providers like GitHub, GitLab or similar are strongly recommended.

We will make extensive use of Python as programming language and its myriad of available libraries. You will find Jupyter notebooks invaluable in your work. You can run R codes in the Jupyter/IPython notebooks, with the immediate benefit of visualizing your data. You can also use compiled languages like C++, Rust, Julia, Fortran etc if you prefer. The focus in these lectures will be mainly on Python.

If you have Python installed and you feel pretty familiar with installing different packages, we recommend that you install the following Python packages via pip as

• pip install numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas pillow

For OSX users we recommend, after having installed Xcode, to install brew. Brew allows for a seamless installation of additional software via for example

• brew install python3

For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution, you can use pip as well and simply install Python as

• sudo apt-get install python3

Python installers

If you don’t want to perform these operations separately and venture into the hassle of exploring how to set up dependencies and paths, we recommend two widely used distrubutions which set up all relevant dependencies for Python, namely

• Anaconda:https://docs.anaconda.com/,

which is an open source distribution of the Python and R programming languages for large-scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment. Package versions are managed by the package management system conda.

• Enthought canopy:https://www.enthought.com/product/canopy/

is a Python distribution for scientific and analytic computing distribution and analysis environment, available for free and under a commercial license.

Furthermore, Google’s Colab:https://colab.research.google.com/notebooks/welcome.ipynb is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud. Try it out!

Useful Python libraries

Here we list several useful Python libraries we strongly recommend (if you use anaconda many of these are already there)

• NumPy:https://www.numpy.org/ is a highly popular library for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays
• The pandas:https://pandas.pydata.org/ library provides high-performance, easy-to-use data structures and data analysis tools
• Xarray:http://xarray.pydata.org/en/stable/ is a Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun!
• Scipy:https://www.scipy.org/ (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
• Matplotlib:https://matplotlib.org/ is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.
• Autograd:https://github.com/HIPS/autograd can automatically differentiate native Python and Numpy code. It can handle a large subset of Python’s features, including loops, ifs, recursion and closures, and it can even take derivatives of derivatives of derivatives
• SymPy:https://www.sympy.org/en/index.html is a Python library for symbolic mathematics.
• scikit-learn:https://scikit-learn.org/stable/ has simple and efficient tools for machine learning, data mining and data analysis
• TensorFlow:https://www.tensorflow.org/ is a Python library for fast numerical computing created and released by Google
• Keras:https://keras.io/ is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano
• And many more such as pytorch:https://pytorch.org/, Theano:https://pypi.org/project/Theano/ etc

Textbooks

Recommended textbooks:
	- Goodfellow, Bengio and Courville, Deep Learning, see https://www.deeplearningbook.org. This textbook is excellent, in particular chapter 1-12 for this course.
	- Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer, https://www.springer.com/gp/book/9780387310732. This is the main textbook and this course covers chapters 1-7, 11 and 12.
	- Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer, https://www.springer.com/gp/book/9780387848570. This is a well-known text and serves as additional text.
	- Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O’Reilly, https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/. This text is very useful since it contains many code examples.

---

The books by Bishop and Hastie et al. can be downloaded for free if you access the university library via an IP number of your home university.

General learning book on statistical analysis:
	- Christian Robert and George Casella, Monte Carlo Statistical Methods, Springer

---

General Machine Learning Books:
	- Kevin Murphy, Machine Learning: A Probabilistic Perspective, MIT Press
	- David J.C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press
	- David Barber, Bayesian Reasoning and Machine Learning, Cambridge University Press

---