When considering gradient based methods for the solution of optimization problem it is useful to consider the level lines of the cost functional. If the level lines are close to circles, then gradient based algorithms will be fast to converge since the gradient (with a minus sign) points toward the minimum. If on the other hand, when those level curves are thin ellipses the gradient does not point toward the minimum in general, and therefore gradient based methods will be slow to converge. The thin ellipses correspond to bad conditioning (large ratio of largest to smallest eigenvalue) of the Hessian.
It is important to notice the following two cases. The first is when removing just a few eigenvalues, a well conditioned system is obtain. In such cases methods which use an approximate Hessian which is constructed during the iterative process, such as BFGS, lead to very effective solvers. The second case is when the condition number remains high even after removing a large number of eigenvalues. This is typical to problems arising from discretization of partial differential equations where the number of design variables is large. Iterative algorithms of the BFGS type cannot serve as a remedy in this case and different approaches are needed. Such approaches will be described in lectures 3 and 4 following this one.