For a simple (vanilla) RNN with one hidden layer and no bias, the state update and output are:
$$ \mathbf{h}_t = \sigma(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}\mathbf{h}_{t-1})\,,\quad \mathbf{y}_t = \mathbf{W}_{yh}\mathbf{h}_t, $$where \( \sigma \) is an activation (e.g. tanh or ReLU) .
In matrix form,
$$ \mathbf{W}_{xh}\in\mathbb{R}^{h\times d}, \mathbf{W}_{hh}\in\mathbb{R}^{h\times h}, \mathbf{W}_{yh}\in\mathbb{R}^{q\times h}, $$for input dimension \( d \), hidden dimension \( h \), output dimension \( q \).
Because the same \( \mathbf{W} \) are used each step, gradients during training will propagate through time.