Suboptimality of MDL and Bayes in Classification under Misspecification
published: Feb. 25, 2007, recorded: October 2005, views: 3198
Report a problem or upload filesIf you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
We show that forms of Bayesian and MDL learning that are often applied to classification problems can be *statistically inconsistent*. We present a large family of classifiers and a distribution such that the best classifier within the model has generalization error (expected 0/1-prediction loss) almost 0. Nevertheless, no matter how many data are observed, both the classifier inferred by MDL and the classifier based on the Bayesian posterior will behave much worse than this best classifier in the sense that their expected 0/1-prediction loss is substantially larger. Our result can be re-interpreted as showing that under misspecification, Bayes and MDL do not always converge to the distribution in the model that is closest in KL divergence to the data generating distribution. We compare this result with earlier results on Bayesian inconsistency by Diaconis, Freedman and Barron.
Link this pageWould you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !