19 September 2012

Learning Bayesian Probability Theory

Alex asked me about some introductory references on Bayesian probability theory. Here's my list (to be read in this order):

1) a historic intro:
Jaynes, 1986, "Bayesian Methods: General Background"

2) intro to Bayesian probability (and the connection to Boolean logic):
first two chapters in the book: Jaynes, 2003, Probability Theory: The Logic of Science
[the free alternative: chapters 3-6 from this Probability Theory With Applications in Science and Engineering (1973)]

3) intro on maximum entropy:
Jaynes, 1982, "On the Rationale of Maximum-Entropy Methods"
Gull, 1988, "Bayesian Inductive Inference and Maximum Entropy"

4) a general overview of the entire theory (including maximum entropy):
Jaynes, 1988, "How Does the Brain Do Plausible Reasoning?"

5) model selection and parameter estimation:
Bretthorst, 1996, "An introduction to model selection using probability theory as logic"
Bretthorst, 1989, "An Introduction to Parameter Estimation"
[the respective chapters in Jaynes' 2003 book]
[the advanced and more philosophical by the end: Jaynes, 1985, "Entropy and Search-Theory"]

6) on the connection between Bayes formula and Maximum entropy:
Caticha, 2003, "Relative Entropy and Inductive Inference"
[Caticha & Giffin, 2006, "Updating Probabilities"]

7) Bayesian theory of surprise:
Itti & Baldi, 2005-2006

8) Bayesian networks:
Jensen & Nielsen, 2007, Bayesian Networks and Decision Graphs