Before we proceed, we would like to discuss the approach called the standard Steepest descent, which again leads to us having to be able to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).
The success of the CG method for finding solutions of non-linear problems is based on the theory of conjugate gradients for linear systems of equations. It belongs to the class of iterative methods for solving problems from linear algebra of the type $$ \begin{equation*} \hat{A}\hat{x} = \hat{b}. \end{equation*} $$
In the iterative process we end up with a problem like $$ \begin{equation*} \hat{r}= \hat{b}-\hat{A}\hat{x}, \end{equation*} $$ where \( \hat{r} \) is the so-called residual or error in the iterative process.
When we have found the exact solution, \( \hat{r}=0 \).