PhD project
Robustness of Bayesian inference against violations of exchangeability

Dr Christian Hennig
chrish@stats.ucl.ac.uk

Many standard Bayesian techniques assume that the data are exchangeable, 
i.e., that the order of observations is not informative. This implies, e.g., 
that in a 0-1-sequence the probability for observing a 1 in the 
51st position if we have observed 20 ones and 30 zeroes before doesn't 
depend on whether these observations are seemingly randomly ordered or we 
observed 20 zeroes in a row first, then the 20 ones, and then 10 further 
zeroes. This is obviously neither realistic, nor can it be expected to
precisely reflect prior beliefs of subjective Bayesians in most situations.

This project is about investigating the effect of violations of 
exchangeability to standard Bayesian inference. It
is restricted to 0-1-sequences, which are the most archetypical
setup for which exchangeability/dependence considerations make sense. 
Furthermore, situations are considered in which there is no clear prior 
information suggesting any particular dependence model, that is, situations 
in which a Bernoulli standard analysis assuming exchangeability and some
prior distribution for the parameter p would normally be applied (see, e.g.
Gelman, Carlin, Stern and Rubin 2003).

In this project, the problem will be approached in a fashion inspired by 
the ideas of robust statistics (see, e.g., Huber 1981), namely that in 
standard parametric statistical analysis usually simple parametric models 
are applied to real situations in which the model assumptions do not hold 
precisely, though there is no alternative parametric model specified for the 
particular (often only very slight) violations of the assumptions.

As a first step, the impact of dependence on the results of a standard
Bernoulli Bayesian analysis should be explored by theoretically investigating
and simulating its behaviour on data generated by models modelling slight
dependences in a situation where the parameter p is still well defined (i.e.,
0-1-sequences generated from stationary processes). 

The project will then go on to investigate methodological possibilities to 
deal with such a situation without modelling the dependence. The idea here is 
that prior models could be considered under which a high probability 
(1-\epsilon) is assigned to the standard exchangeability model, while a
probability of \epsilon is reserved for some unrestricted violations of 
exchangeability. It is of interest whether some methodology can be developed
that keeps most of the simplicity of the standard analysis while safeguarding
against unexpected extreme situations. This is connected to the contamination
model and trimmed estimators in robust statistics.

The outcome of the project should have practical as well as philosophical 
implications. Methodological developments would be practically useful, because
0-1-sequences without any prespecified dependence structure are very common
while convincing arguments in favour of precise independence (conditionally 
on p) are rare. Philosophically, the project could shed some light on
Gillies' (2000) criticism of Bayesian statistics that the exchangeability 
assumption is actually not weaker than assuming the existence of an underlying 
frequentist model with independence by investigating how robust Bayesian
inference is against being confronted with data generated in a frequentist 
manner that doesn't fulfill the assumptions precisely.

Existing literature on Bayesian robustness (Berger 1995) and Bayesian model 
averaging (Hoeting, Madigan, Raftery and Volinsky 1999) will be taken 
into account as well as some work on robustness of standard statistical
methodology against violations of independence (Kuensch, Beran and Hampel 
1993).

References:
Berger, J. O. (1995) Bayesian robustness: Proceedings of the Workshop on 
Bayesian Robustness, May 22-May 25, 1995, Rimini, Italy, IMS.
Gelman, A., Carlin, J. B., Stern H. S., Rubin, D. B. (2003) Bayesian Data
Analysis (2nd ed.), Chapman and Hall.
Gillies, D. (2000) Philosophical Theories of Probability, Routledge.
Hoeting, J., Madigan, D., Raftery, A. and Volinsky, C. (1999) Bayesian model
averaging, Statistical Science 14, 382-401. 
Huber, P. J. (1981) Robust Statistics, Wiley.
Künsch, H., Beran, J. and Hampel, F. (1993) Contrasts under long-range 
correlations, The Annals of Statistics 21, 943-964.