BFGS Algorithm Overview
The BFGS method is an iterative procedure that approximates the
inverse Hessian matrix \( H_k \) at each iteration. The update of the
current solution \( \mathbf{x}_k \) involves computing a search
direction and step length. The general steps of the BFGS algorithm are
as follows:
- Initialize \( \mathbf{x}_0 \) and choose an initial guess for the inverse Hessian approximation \( H_0 \), typically \( H_0 = I \) (the identity matrix).
- For each iteration \( k \), do the following:
- Compute the gradient at the current point: \( \nabla f(\mathbf{x}_k) \).
- Compute the search direction:
$$
\mathbf{p}_k = -H_k \nabla f(\mathbf{x}_k)
$$
- Perform a line search to find an appropriate step size \( \alpha_k \), which minimizes \( f(\mathbf{x}_k + \alpha_k \mathbf{p}_k) \).
- Update the current point:
$$
\mathbf{x}_{k+1} = \mathbf{x}_k + \alpha_k \mathbf{p}_k
$$
- Compute the new gradient \( \nabla f(\mathbf{x}_{k+1}) \).