Expert elicitation

Eliciting a probability distribution is the process of extracting an expert’s beliefs about some unknown quantity of interest, and representing his/her beliefs with a probability distribution. The challenges are, firstly, to help the expert consider uncertainty carefully, without being excessively overconfident or underconfident, and secondly, to find a way of constructing a full probability distribution based on a small number of simple probability judgements from the expert.

Elicitation can be used to construct prior distributions in Bayesian inference, though my interest is more in situations where we are using expert judgement because there is no data.

SHELF

Tony O’Hagan and I have developed the Sheffield Elicitation Framework (SHELF), a package of protocols, templates and guidance documents for conduction expert elicitation. In support of this, I maintain an R package SHELF which is available on CRAN (with a development version on GitHub).

Shiny apps for elicitation

The SHELF R package includes shiny apps for implementing various elicitation methods. If you are not an R user, you can try the following apps online. Access to these apps is time-limited (they are hosted on RStudio’s shinyapps.io service).

If you want to use the apps in an actual elicitation workshop, I strongly encourage you to use the offline versions in the SHELF R package.

• Eliciting a single distribution. Elicit a single distribution, directly specifying probabilities or quantiles, or using the roulette method.

• Eliciting individual distributions from multiple experts. Includes options for fitting a weighted linear pool.

• Eliciting bivariate distributions. Elicit a joint distribution for two dependent variables using a Gaussian copula.

• Eliciting Dirichlet distributions. Elicit a Dirichlet distribution for a set of proportions constrained to sum to 1.

• The extension method: continuous extension and target variables. This method involves eliciting a distribution for a ‘target’ variable X via a distribution for an ‘extension’ variable Y and a conditional distribution for X|Y. In this app, both X and Y are continuous.

• The extension method: discrete extension variable and continuous target variable. Similar to above, but with a discrete extension variable Y. In particular, this method allows the elicitation of multi-modal distributions: the density for X is of the form ∑yfx(x|Y=y)Pr(Y=y).

• Assurance for clinical trial design with normally distributed data. This implements the method in Alhussain and Oakley (2019).

The MATCH elicitation tool

As part of the MATCH project, we produced a web-based elicitation tool which is based on an earlier version of the SHELF R code. Multiple users can log into the same session, which can be useful when the facilitator and expert(s) can’t meet in the same room.

Papers on elicitation

Salsbury, J. A., Oakley J. E, Julious, S. A, Hampson, L. V. (2024). To appear in Statistics in Medicine. Assurance methods for designing a clinical trial with a delayed treatment effect. 

Supporting shiny app.

Jurek, L., Baltazar, M., Gulati, S., Novakovic N., Nunez, M., Oakley, J. and O’Hagan, A. (2021) Response (minimum clinically relevant change) in ASD symptoms after an intervention according to CARS-2: consensus from an expert elicitation procedure. Eur. Child Adolesc. Psychiatry.
DOI: https://doi.org/10.1007/s00787-021-01772-z

Wilson, K. J., Elfadaly, F. G., Garthwaite P. H. and Oakley J. E. (2021) Recent advances in the elicitation of uncertainty distributions for multinomial probabilities. In Expert judgement and risk analysis, Hanea, A. M., Nane, G. F., Bedford, T. and French, S. (eds.).

Alhussain, Z. A. and Oakley, J. E. (2020) Assurance for clinical trial design with normally distributed outcomes: eliciting uncertainty about variances. Pharmaceutical Statistics, 19, 827-839.
R package (assurance) on GitHub

Ren, S., Oakley, J. E. and Stevens, J. W. (2018) Incorporating genuine prior information about between-study heterogeneity in random effects pairwise and network meta-analyses. Medical Decision Making, 38(4), 531-542.

Morris, D. E., Oakley, J. E. and Crowe, J. A. (2014) A web-based tool for eliciting probability distributions from experts. Environmental Modelling & Software, 52, 1-4.

Ren, S. and Oakley, J. E. (2014) Assurance calculations for planning clinical trials with time-to-event outcomes. Statistics in Medicine, 33(1), 31-45.
Supporting R code (ZIP, 922KB)

Oakley, J. E. (2010) Eliciting univariate probability distributions, in Rethinking risk measurement and reporting: Volume 1, edited by Böcker, K., Risk Books, London.

Daneshkhah, A. and Oakley, J.E. (2010) Eliciting multivariate probability distributions, in Rethinking risk measurement and reporting: Volume 1, edited by Böcker, K., Risk Books, London.
Supporting R code (R, 4KB)

Nixon, R.M., O’Hagan, A., Oakley, J. E., Madan, J., Stevens, J.W. Bansback, N. and Brennan, A. (2009) The rheumatoid arthritis drug development model: A case study in Bayesian clinical trial simulation. Pharmaceutical Statistics, 8(4), 371-389.

Gosling, J.P., Oakley, J.E. and O’Hagan, A. (2007) Nonparametric elicitation for heavy-tailed prior distributions. Bayesian Analysis, 2, 693-718.

Oakley, J. and O’Hagan, A. (2007) Uncertainty in prior elicitations: A non-parametric approach. Biometrika, 94, 427-441.

O’ Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. E., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E. and Rakow, T. (2006) Uncertain judgements: Eliciting expert probabilities. Chichester: Wiley.

O’Hagan, A. and Oakley, J. E. (2004) Probability is perfect, but we can’t elicit it perfectly. Reliability Engineering and System Safety, 85, 239-248.

Oakley, J. (2002). Eliciting Gaussian process priors for complex computer codes. The Statistician, 51, 81-97.

I have also contributed to the following guidance document prepared by the European Food Safety Authority:

European Food Safety Authority (2014). Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA Journal 2014, 12(6):3734, 278 pp.
DOI: https://doi.org/10.2903/j.efsa.2014.3734