In Bayesian inference, the Bernstein–von Mises theorem provides the basis for using Bayesian credible sets for confidence statements in parametric models. It states that under some conditions, a posterior distribution converges in total variation distance to a multivariate normal distribution centered at the maximum likelihood estimator
\widehat{\theta}n
n-1
| -1 | |
| l{I}(\theta | |
| 0) |
\theta0
l{I}(\theta0)
||P(\theta|x1,...xn)-l{N}({\widehat{\theta}}n,n-1
| -1 | |
| l{I}(\theta | |
| 0) |
)||{TV
The Bernstein–von Mises theorem links Bayesian inference with frequentist inference. It assumes there is some true probabilistic process that generates the observations, as in frequentism, and then studies the quality of Bayesian methods of recovering that process, and making uncertainty statements about that process. In particular, it states that asymptotically, many Bayesian credible sets of a certain credibility level
\alpha
\alpha
Let
(P\theta:\theta\in\Theta)
\Theta
Rk
X1,\ldots,Xn\inl{X}
| P | |
| \theta0 |
(p\theta:\theta\in\Theta)
\mu
l{I}(\theta0)
f:l{X} → Rk
\int\left[\sqrt{p\theta(x)}-
| \sqrt{p | |
| \theta0 |
(x)}-
| 1 | |
| 2 |
(\theta-
| \top | |
| \theta | |
| 0) |
| f(x)\sqrt{p | |
| \theta0 |
(x)}\right]2d\mu(x)=o(||\theta-
| 2) | |
| \theta | |
| 0|| |
\theta → \theta0
\varepsilon>0
| n | |
| \phi | |
| n:l{X} |
→ [0,1]
| E | |||||||||
|
\left[\phin(X)\right] → 0
| \sup | |
| \theta:||\theta-\theta0||>\varepsilon |
| E | |||||||||
|
\left[1-\phin(X)\right] → 0
n → infty
\theta0
\theta0
Then for any estimator
\widehat{\theta}n
\sqrt{n}({\widehat{\theta}}n-\theta0)\xrightarrow{d}l{N}(0,{l{I}}-1(\theta0))
\Pin
\theta\midX1,\ldots,Xn
as} 0.{\left|\left|\Pin-l{N}\left(\widehat{\theta}n,
1 n {l{I}}-1({\theta0})\right)\right|\right|}TV
\xrightarrow{P \theta0
n → infty
Under certain regularity conditions, the maximum likelihood estimator is an asymptotically efficient estimator and can thus be used as
\widehat{\theta}n
The most important implication of the Bernstein–von Mises theorem is that the Bayesian inference is asymptotically correct from a frequentist point of view. This means that for large amounts of data, one can use the posterior distribution to make, from a frequentist point of view, valid statements about estimation and uncertainty.
The theorem is named after Richard von Mises and S. N. Bernstein, although the first proper proof was given by Joseph L. Doob in 1949 for random variables with finite probability space.[2] Later Lucien Le Cam, his PhD student Lorraine Schwartz, David A. Freedman and Persi Diaconis extended the proof under more general assumptions.
In case of a misspecified model, the posterior distribution will also become asymptotically Gaussian with a correct mean, but not necessarily with the Fisher information as the variance. This implies that Bayesian credible sets of level
\alpha
\alpha
In the case of nonparametric statistics, the Bernstein–von Mises theorem usually fails to hold with a notable exception of the Dirichlet process.
A remarkable result was found by Freedman in 1965: the Bernstein–von Mises theorem does not hold almost surely if the random variable has an infinite countable probability space; however, this depends on allowing a very broad range of possible priors. In practice, the priors used typically in research do have the desirable property even with an infinite countable probability space.
Different summary statistics such as the mode and mean may behave differently in the posterior distribution. In Freedman's examples, the posterior density and its mean can converge on the wrong result, but the posterior mode is consistent and will converge on the correct result.
. John A. Hartigan . Asymptotic Normality of Posterior Distributions . Bayes Theory . New York . Springer . 1983 . 10.1007/978-1-4613-8242-3_11 .
. Lucien Le Cam . Asymptotic Methods in Statistical Decision Theory . Approximately Gaussian Posterior Distributions . 336–345 . New York . Springer . 1986 . 0-387-96307-3 .