Expert elicitation

Eliciting a probability distribution is the process of extracting an expert’s beliefs about some unknown quantity of interest, and representing his/her beliefs with a probability distribution. The challenges are, firstly, to help the expert consider uncertainty carefully, without being excessively overconfident or underconfident, and secondly, to find a way of constructing a full probability distribution based on a small number of simple probability judgements from the expert.

Elicitation can be used to construct prior distributions in Bayesian inference, though my interest is more in situations where we are using expert judgement because there is no data.

SHELF

Tony O’Hagan and I have developed the Sheffield Elicitation Framework (SHELF), a package of protocols, templates and guidance documents for conduction expert elicitation. In support of this, I maintain an R package SHELF which is available on CRAN (with a development version on GitHub).

Shiny apps for elicitation

The SHELF R package includes shiny apps for implementing various elicitation methods. If you are not an R user, you can try the following apps online. Access to these apps is time-limited (they are hosted on RStudio’s shinyapps.io service).

If you want to use the apps in an actual elicitation workshop, I strongly encourage you to use the offline versions in the SHELF R package.

The MATCH elicitation tool

As part of the MATCH project, we produced a web-based elicitation tool which is based on an earlier version of the SHELF R code. Multiple users can log into the same session, which can be useful when the facilitator and expert(s) can’t meet in the same room.

Papers on elicitation

Salsbury, J. A., Oakley J. E, Julious, S. A, Hampson, L. V. (2024). To appear in Statistics in Medicine. Assurance methods for designing a clinical trial with a delayed treatment effect

Supporting shiny app.

Jurek, L., Baltazar, M., Gulati, S., Novakovic N., Nunez, M., Oakley, J. and O’Hagan, A. (2021) Response (minimum clinically relevant change) in ASD symptoms after an intervention according to CARS-2: consensus from an expert elicitation procedure. Eur. Child Adolesc. Psychiatry.
DOI: https://doi.org/10.1007/s00787-021-01772-z

Wilson, K. J., Elfadaly, F. G., Garthwaite P. H. and Oakley J. E. (2021) Recent advances in the elicitation of uncertainty distributions for multinomial probabilities. In Expert judgement and risk analysis, Hanea, A. M., Nane, G. F., Bedford, T. and French, S. (eds.).

Alhussain, Z. A. and Oakley, J. E. (2020) Assurance for clinical trial design with normally distributed outcomes: eliciting uncertainty about variances. Pharmaceutical Statistics, 19, 827-839.
R package (assurance) on GitHub

Ren, S., Oakley, J. E. and Stevens, J. W. (2018) Incorporating genuine prior information about between-study heterogeneity in random effects pairwise and network meta-analyses. Medical Decision Making, 38(4), 531-542.

Morris, D. E., Oakley, J. E. and Crowe, J. A. (2014) A web-based tool for eliciting probability distributions from experts. Environmental Modelling & Software, 52, 1-4.

Ren, S. and Oakley, J. E. (2014) Assurance calculations for planning clinical trials with time-to-event outcomes. Statistics in Medicine, 33(1), 31-45.
Supporting R code (ZIP, 922KB)

Oakley, J. E. (2010) Eliciting univariate probability distributions, in Rethinking risk measurement and reporting: Volume 1, edited by Böcker, K., Risk Books, London.

Daneshkhah, A. and Oakley, J.E. (2010) Eliciting multivariate probability distributions, in Rethinking risk measurement and reporting: Volume 1, edited by Böcker, K., Risk Books, London.
Supporting R code (R, 4KB)

Nixon, R.M., O’Hagan, A., Oakley, J. E., Madan, J., Stevens, J.W. Bansback, N. and Brennan, A. (2009) The rheumatoid arthritis drug development model: A case study in Bayesian clinical trial simulation. Pharmaceutical Statistics, 8(4), 371-389.

Gosling, J.P., Oakley, J.E. and O’Hagan, A. (2007) Nonparametric elicitation for heavy-tailed prior distributions. Bayesian Analysis, 2, 693-718.

Oakley, J. and O’Hagan, A. (2007) Uncertainty in prior elicitations: A non-parametric approach. Biometrika, 94, 427-441.

O’ Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. E., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E. and Rakow, T. (2006) Uncertain judgements: Eliciting expert probabilities. Chichester: Wiley.

O’Hagan, A. and Oakley, J. E. (2004) Probability is perfect, but we can’t elicit it perfectly. Reliability Engineering and System Safety, 85, 239-248.

Oakley, J. (2002). Eliciting Gaussian process priors for complex computer codes. The Statistician, 51, 81-97.

I have also contributed to the following guidance document prepared by the European Food Safety Authority:

European Food Safety Authority (2014). Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA Journal 2014, 12(6):3734, 278 pp.
DOI: https://doi.org/10.2903/j.efsa.2014.3734

Other research

Health economics

Applications of Bayesian statistics in health economics.

Find out more about health economics

Uncertainty quantification

Quantifying uncertainty in computer model predictions.

Find out more about uncertainty quantification