Quasi-Newton Methods: A New Direction






Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newton methods can be interpreted as approximations of Bayesian linear regression under varying prior assumptions. This new notion elucidates some shortcomings of classical algorithms, and lights the way to a novel nonparametric quasi-Newton method, which is able to make more efficient use of available information at computational cost similar to its predecessors.

Author(s): Hennig, Philipp and Kiefel, Martin
Journal: Journal of Machine Learning Research
Volume: 14
Number (issue): 1
Pages: 843--865
Year: 2013
Month: March

Department(s): Empirical Inference, Perceiving Systems, Probabilistic Numerics
Research Project(s): Probabilistic Methods for Nonlinear Optimization
Bibtex Type: Article (article)
Paper Type: Journal

URL: http://www.jmlr.org/papers/volume14/hennig13a/hennig13a.pdf

Links: website+code
Attachments: pdf


  title = {Quasi-Newton Methods: A New Direction},
  author = {Hennig, Philipp and Kiefel, Martin},
  journal = {Journal of Machine Learning Research},
  volume = {14},
  number = {1},
  pages = {843--865},
  month = mar,
  year = {2013},
  url = {http://www.jmlr.org/papers/volume14/hennig13a/hennig13a.pdf},
  month_numeric = {3}