Week 47: Recurrent neural networks and Autoencoders
Contents
Plan for week 47
Reading recommendations RNNs
TensorFlow examples
Reading recommendations: Autoencoders (AE)
What is a recurrent NN?
Why RNNs?
More whys
RNNs in more detail
RNNs in more detail, part 2
RNNs in more detail, part 3
RNNs in more detail, part 4
RNNs in more detail, part 5
RNNs in more detail, part 6
RNNs in more detail, part 7
RNN Forward Pass Equations
Unrolled RNN in Time
Example Task: Character-level RNN Classification
PyTorch: Defining a Simple RNN, using Tensorflow
Similar example using PyTorch
Backpropagation Through Time (BPTT) and Gradients
Truncated BPTT and Gradient Clipping
Limitations and Considerations
PyTorch RNN Time Series Example
Tensorflow (Keras) RNN Time Series Example
The mathematics of RNNs, the basic architecture
Gating mechanism: Long Short Term Memory (LSTM)
Implementing a memory cell in a neural network
LSTM details
LSTM Cell and Gates
Core LSTM Equations
Gate Intuition and Dynamics
Basic layout (All figures from Raschka
et al.,
)
LSTM details
Comparing with a standard RNN
LSTM details I
LSTM details II
LSTM details III
Forget gate
The forget gate
Basic layout
Input gate
Short summary
Forget and input
Basic layout
Output gate
LSTM Implementation (Code Example)
Example: Modeling Dynamical Systems
Example: Biological Sequences
Training Tips and Variants
LSTM Summary
Summary of LSTM
LSTM implementation using TensorFlow
And the corresponding one with PyTorch
Dynamical ordinary differential equation
The Runge-Kutta-4 code
Using the above data to train an RNN
Similar code using PyTorch
Autoencoders: Overarching view
Powerful detectors
First introduction of AEs
Autoencoder structure
Schematic image of an Autoencoder
More on the structure
Decoder part
Typical AEs
Feed Forward Autoencoder
Mirroring
Output of middle layer
Activation Function of the Output Layer
ReLU
Sigmoid
Cost/Loss Function
Binary Cross-Entropy
Reconstruction Error
Implementation using TensorFlow
Implementation using PyTorch
Dimensionality reduction and links with Principal component analysis
Linear functions
AE mean-squared error
Dimensionality reduction
Reading recommendations: Autoencoders (AE)
Goodfellow et al chapter 14, see
https://www.deeplearningbook.org/contents/autoencoders.html
Rashcka et al. Their chapter 17 contains a brief introduction only.
Building AEs in Keras at
https://blog.keras.io/building-autoencoders-in-keras.html
Introduction to AEs in TensorFlow at
https://www.tensorflow.org/tutorials/generative/autoencoder
«
1
2
3
4
5
6
7
8
9
10
11
12
13
14
...
81
»