Past Event: Babuška Forum

A fresh look at the Bayesian theorem from information theory

Tan Bui, Aerospace Engineering and Engineering Mechanics, ICES, UT Austin

Abstract

We construct a convex optimization problem whose first order optimality condition is exactly the Bayes’ formula and whose unique solution is precisely the posterior distribution. In fact, the solution of our optimization problem includes the usual Bayes’ posterior as a special case and it is therefore more general. We provide the construction, and hence a generalized Bayes’ formula, for both finite and infinite dimensional settings. We shall show that the our posterior distribution, and the Bayes’ one as a special case, is optimal in the sense that it is the unique minimizer of an objective function. We provide the detailed and constructive derivation of the objective function using information theory and optimization technique. In particular, the objective is the compromise of two quantities: 1) the relative entropy between the posterior and the prior, and 2) the mean squared error between the computer model and the observation data. As shall be shown, our posterior minimizes these two quantities simultaneously.