TR2000-27

An Idiosyncratic Journey Beyond Mean Field Theory


    •  Yedidia, J.S., "An Idiosyncratic Journey Beyond Mean Field Theory" in Advanced Mean Field Methods, Theory and Practice, Opper, M. and Saad, D., Eds., ISBN: 0-262-15045-9, chapter 3, pp. 21-36, The MIT Press, February 2001.
      BibTeX Download PDF
      • @incollection{Yedidia2001feb,
      • author = {Yedidia, J.S.},
      • title = {An Idiosyncratic Journey Beyond Mean Field Theory},
      • booktitle = {Advanced Mean Field Methods, Theory and Practice},
      • year = 2001,
      • editor = {Opper, M. and Saad, D.},
      • chapter = 3,
      • pages = {21--36},
      • month = feb,
      • publisher = {The MIT Press},
      • isbn = {0-262-15045-9},
      • url = {http://www.merl.com/publications/TR2000-27}
      • }

I try to clarify the relationships between different ways of deriving or correcting mean field theory, and present "translations" between the language of physicists and that of computer scientists. The connecting thread between the different methods described here is the Gibbs free energy. After introducing the inference problem we are interested in analyzing, I will define the Gibbs free energy, and describe how to derive a mean field approximation to it using a variational approach. I will then explain how one might re-derive and correct the mean field and TAP free energies using high temperature expansions with constrained one-node beliefs. I will explore the relationships between the high-temperature expansion approach, the Bethe approximation, and the belief propagation algorithm, and point out in particular the equivalence of the Bethe approximation and belief propagation. Finally, I will describe Kikuchi approximations to the Gibbs Free energy and advertise new belief propagation algorithms that efficiently compute beliefs equivalent to those obtained from the Kikuchi free energy.