Quantifying Experts Uncertainty about the Future Cost of Exotic Diseases - Report Example

QUANTIFYING EXPERTS’ UNCERTAINTY ABOUT THE FUTURE COST OF EXOTIC DISEASES INTRODUCTION Variability and uncertainty are concepts that have been aroundfor several decades. In order for us to be able to predict future outcomes of an event or action, the two components must be determined. While variability is a physical phenomenon that can be measured, analyzed and explained, uncertainty involves knowledge (Vose, 2002). Decision making is the most important task for managers. Often, the degree of knowledge on the outcome of actions and events, form the basis for decision making. In the wake of uncertainty, decision-making becomes a difficult task. Therefore, managers rely on expert knowledge when planning the decision making process (Taghavifard et al., 2009). This report discusses various techniques that are commonly used for the quantification of uncertainty, and the role that expert knowledge plays in the process of quantification of uncertainty. The statistical techniques used in a study by Gosling et al., (2012) - “quantifying experts’ uncertainty about the future cost of exotic diseases”-, and their applications, are described in detail. The experts recruited in the study by Gosling et al., (2012) comprised of a team of veterinarians and economists in England, whose opinions were used to provide estimates of the average annual cost for a number of exotic diseases. The article reveals how formal elicitation techniques were applied in the quantification of the experts’ uncertainties about cost estimates of exotic diseases. All terms that have been used in this paper are clearly defined and where necessary, relevant examples of the techniques used in quantification of uncertainty are presented. Definition of terms The term expert is used to denote individuals who background knowledge in a particular subject area or field. When experts identify relevant data and information, such as models, experimental results and numerical methods, they are providing expertise. When the experts provide estimates of a given phenomenon (qualitative or quantities) or the uncertainties, they are providing expert judgment. Expert knowledge collectively refers to the information woven in the minds of experts , which is employed in solving complex, difficult (technical) questions, by formal elicitation (Booker et al., 2001, Meyer and Booker, 2001). In the context of statistical analysis, elicitation is the process by which a person’s beliefs about some uncertain quantities are translated into a probability distribution (Gosling et al., 2012). Probability is defined as the likelihood of an event occurring and can be expressed as a number between 0 (no chance) and1 (likelihood of happening) (Rice, 2007). Uncertainties are categorized into two: aleatory and epistemic. Basically, aleatory involves uncertainty due to physical or random variability present in the system or its environment. It is not strictly due to a lack of knowledge and hence cannot be reduced. In literature, aleatory uncertain uncertainty is often termed as stochastic uncertainty or irreducible uncertainty. On the other hand, epistemic uncertainty potential deficiency is due to lack of knowledge. Unlike aleatory uncertainty, epistemic uncertainty can be reduced. Epistemic uncertainty is often referred to as reducible certainty or incertitude (Booker et al., 2001, Laccarino ). Elicitation of expert judgments Experts perform assessment tasks that quantify their opinions in probabilistic terms, which are in turn used in statistical models to predict the capital investment in order to maintain and improve available assets (Garthwaite and OHagan., 2000). According to Booker et al., (2001), quantification of uncertainty involves the characterization, estimation, propagation and analysis of certain types of uncertainties (including variability), for a complex decision phenomenon. The elicitation process is completed by three individuals: the decision maker, the expert and the facilitator. Figure 1: Key players of the elicitation exercise The role of a facilitator is to question the experts on their beliefs and opinions regarding some phenomenon, and then use the answers given, to fit a representative model. The generated model is then handed to the decision maker, who in turn uses it to update their beliefs on the subject matter, in light of the expert’s opinion. Prior to carrying out the elicitation exercise, the facilitator must have adequate background and understanding of statistics (Gosling, 2005). Typically, an elicitation should consist of the following components, which are executed in an iterative way: 1) Pre-elicitation training – the facilitators should clearly explain to the experts the objectives of carrying out assessment so that they can understand how their judgment will be used. The facilitator conducts training for the experts, which can be divided into three: a general discussion of subjective probability distribution and their elicitation; a discussion of relevant known biases that could occur; and a practice elicitation session, which involves asking the experts some questions, outside the subject of concern. 2) The elicitation session-the experts are asked to make a small number of probabilistic judgments about a phenomenon 3) Post elicitation feedback-the facilitator fits a suitable parametric probability distribution to the experts’ judgments and reports the features of the distribution to the experts, so that they can confirm whether they are a true reflection of their beliefs. 4) Analysis of disparate views-if the experts agree with the probability distribution in step 3, the elicitation process is concluded. If the experts do not agree with the model, the facilitator fits an alternative distribution, which is usually based on modified or additional probabilistic judgments from the experts (Slottje et al., 2008, Oakley, 2010). The process of elicitation also has limitations, including : difficultly by experts to articulate their beliefs clearly ; or biases in the questioning process, especially when the people in question are not experts in the field of interest (Gosling, 2005, Gosling et al., 2012). Such-like limitations often confer a subjective nature the elicitation exercise. In order to overcome this problem, the elicitation exercise must be carried out with the highest transparency, as far as possible. The quantification of expert’s uncertainty about the future cost of exotic diseases The objective of Gosling et al., (2012), was to quantify the uncertainty of DEFRA’s in-house veterinarians and economists about the average annual costs of exotic diseases in England to the government. The elicitation process and methods / approaches that were used in study are presented in detail. As stated previously, the four -stage elicitation process described in Slottjie et al., (2008) and Oakley (2010), was adopted in this study. The experts’ categorization of disease outbreak as being either minor or major was influenced by findings of a previous study by the Department for Food Environment and Rural Affairs (DEFRA). Since the experts were accustomed to the conventional way of evaluating cost in terms of major and minor outbreaks, they were able to clearly distinguish between the two categories (Department for Food Environment and Rural Affairs). Four parameters were estimated, based on the assumption of a steady stated over an indefinite long time frame. The parameters included the following: average interval between outbreaks; probability that an outcome would be a major outbreak; the average cost of a minor outbreak; and the average cost of a major outbreak. The choice summaries, which include simple statistical measures such as the mean, median, mode and variances, constitute a part of the elicitation method. For each of the parameters listed above, the facilitators used a systematic approach to elicited range, median, and quartiles (Gosling et al., 2012). The statistical approaches are discussed in the section that follows: Median, Range and Quartiles In probability theory and statistics, measures of spread include measures that attempt to describe a whole set of data with a single value that represents the middle or centre. They include mean, mode and median (Statistics). The median is the value which occupies the central position, when all the observations of a distribution are arranged in an ascending or descending order. It divides the frequency distribution into exactly two halves. Often, 50 % of observations in a distribution have scores at or below the median. The median is therefore referred to as the 50th percentile or ‘positional average’. In order to calculate the median, the values are arranged, either in descending or ascending order, and the middle value is picked. If there is an even number of distributions, the mean of two middle values is calculated to give the median (Manikandan, 2011 ). Consider the following set of values: 10, 39, 48, 67, 11, 34, 67, 89, 23, 45, 56 ,87 and 89.The median will be computes as follows: First arrange the numbers 10, 11, 23, 34, 39,45, 48, 56,67,67,87 ,89; calculate the mean of 45 and48 (Mean= 45+48 / 2 = 46.5). It is important to note that the median is the most preferred measure when the distribution is not symmetrical. Unlike the mean, the median is less affected by outliers and skewed data (Manikandan, 2011 ). Measures of spread describe how similar or varied the set of data are for a particular variable. They include range, quartiles and the interquartile range, variance and standard deviation(Statistics). Range is given as the difference between the highest and lowest values in a data set. For example, for the given data set: 10, 39, 48, 67, 11, 34, 67, 89, 23, 45, 56, 87 and 89, the range will be calculated as 89-10=79. Quartiles divide an ordered data set in a given distribution, into four equal parts. The quartiles can be represented as follows: Table 1: Illustration of quartiles 25 % of values Q1 25 % of values Q2 25 % of values Q3 25 % of values Where, Q1: Lower quartile-the point between the lowest 25 % of values and the highest 75% of values. Sometimes, it is termed as the 25th percentile. Q1: Second quartile- the point between that represents the middle data set. Sometimes, it is termed as the 50th percentile or the median. Q1: Upper quartile-the point between the lowest 75 % of values and the highest 25% of values. Sometimes, it is termed as the 75th percentile (Statistics). The bisection method When determining a prior distribution, variances of known scalar quantities and/or sample errors, are required. One way in which variances can be elicited, which avoids their direct assessment, is by eliciting credible intervals; the intervals are useful as they can yield estimates of variances if suitable distributions are made. Generally, there are two ways of assessing credible intervals for scalar quantities; the fixed interval method and the variable interval method. The median and quartiles were elicited by use of the bisection method in the study by Gosling et al., (2012). To illustrate the bisection method, let F denotes a particular scalar of interest: The expert identifies points that correspond to the specified percentiles of their subjective distribution. The bisection method is often used with the variable interval method. The bisection method often entails a sequence of questions of the following form: Q1. Can you determine a value (the expert’s median) such that F is likely to be less than or greater than this point? Q2. Suppose F is below your assessed median, can you now determine a new value (the lower quartile) such that F is less than or greater than this value? Q3. Suppose F is above your assessed median, can you determine a new value (the upper quartile) such that it is equally likely that F is less than or greater tan this value? One major advantage of this line of questioning is that only judgments of equal odds are required. The bisection method does not require experts to have a good understanding of statistics (Garthwaite et al., 2005). Log normal distribution versus beta distribution The log-normal distribution is useful when raw data is highly skewed whereas the natural log of the data is normally distributed. Because the basic properties of log-normal distributions were established long ago, it is not difficult to express the log-distributions mathematically. Generally, a random number X is said to be log normally distributed if the log (X) is normally distributed. There are two parameters in log-normal distribution, that s, mean (µ: the location parameter) and standard deviation ( : the scale parameter). On the other hand, the beta distribution is a continuous distribution that occurs between 0 and 1. The beta distribution is characterized by two properties, namely: a=first shape parameter and b=second shape parameter. The Beta distribution is said to be identical to the uniform distribution on (0.1), when a=b=1.the common application of Beta distribution is in the measurement of success for a binomial distribution (Thompson, 1999, René van Dorp, 2009). The criteria for selection of lognormal and beta distributions in parametric distributions is summarized in Table 2. Table 2: Selected parametric distributions and the generative process that describes their selection criteria Type of distribution Range Criteria for selection Beta Fractional quantities including the probability of observing an event given data of n trials and r successes. Conjugate prior for a Bernoulli method. Uncertainty in the probability of an event occurring. Sometimes, a modified four parameter version is used for subjective distributions. Log normal Arises from multiplicative processes. Describes quantities that are the product of a large number of other quantities. Used to describe quantities that have occasionally large values (right skewed quantities) ,such as the concentration of a given substance in the environment, the number of species in a family of communities, the resistance of chemicals to plants e.t.c Source: (Thompson, 1999) The Monte Carlo Estimation procedure Monte Carlo analysis uses statistical sampling and probability distributions to simulate effects of uncertain variables on model outcomes. The Monte Carlo approach enables the decision makers are to understand and visualize and the following attributes: i. the expected value and range of variability due to the uncertainty on each of the variables modeled ii. the relationship between variables and estimated possible outcomes iii. the expected value and range of the possible outcomes , representing the combined effect of the multiple sources of uncertainty Firstly, the process involves the identification and assessment of the key variables. Secondly, a suitable probability distribution (or multivariate distribution) that best describes the uncertainty around the expected value is assigned for each variable. Input values for each of the variables being modeled are randomly drawn from the underlying probability distribution function. Thirdly, appropriate computer models are used to combine these inputs in order to generate an estimated outcome value. The process is usually repeated several times (thousand times) to generate a probability distribution of possible outcomes (New Zealand Treasury, 2005). Several commercial packages exist that run the Monte Carlo simulation; however a basic spreadsheet such as Microsoft EXCEL can be used to run the simulation. Accuracy and precision are dependent on size of distribution; the large the distribution, the more accurate the cost estimates. Gosling et al., (2005) used the Monte Carlo estimation procedure to analyze uncertainty in the total cost using the same procedure listed in New Zealand Treasury (2005). The study by Gosling et al., (2009) is a major improvement of a previous study by DEFRA. Firstly, the combination mechanism used for the estimates were incorrect in the previous study; it assumed that the product of average is equal to an average. Secondly, the previous study did not consider the fact the diseases’ costs were potentially dependable. Experts of the study by Gosling et al., (2009) comprised of the same group of DEFRA employees, which had originally formed the point estimates of the previous study. This means that there was a high chance that the experts’ judgments on the qualities of interest would be based on the original point estimates. According to Table III, the experts expressed fears that the involvement of a different group of experts would lead to different judgments altogether. All in all, this study provided DEFRA’s decision makers with an assessment of DEFRA’s expertise in estimation of annual costs from exotic diseases in livestock. Additional sources of uncertainty were considered as shown in Table III. The study involved a thorough quantification of the uncertainties; all aspects of the study (including the scope, model structure, possibility of change in scenarios, expert selection and potential bias) were critically analyzed (Gosling et al., 2012). REFERENCES BOOKER, J. M., ANDERSON, M. C. & MEYER, M. A. 2001. The Role of Expert Knowledge in Uncertainty Quantification (Are We Adding More Uncertainty or More Understanding?). Seventh Army Conference on Applied Statistics DEPARTMENT FOR FOOD ENVIRONMENT AND RURAL AFFAIRS. Impact assessment of an independent body for animal health in England, consultation document, 2009. . Available: http:/www.defra.gov.uk/corporate/consult/new-independentbody-ah/consultation-document.pdf. GARTHWAITE, P. H., KADANE, J. B. & OHAGAN, A. 2005. Statistical Methods for Eliciting Probability Distributions. Available: http://www.stat.cmu.edu/tr/tr808/tr808.pdf. GARTHWAITE, P. H. & OHAGAN., A. 2000. Quantifying Expert Opinion in the UK Water Industry: An Experimental Study. Journal of the Royal Statistical Society: Series D (The Statistician), 49, 455-477. GOSLING, J. P. 2005. Elicitation: a nonparametric view. Doctor of Philosophy, University of Sheffield. GOSLING, J. P., HART, A., MOUAT, D. C., SABIROVIC, M., SCANLAN, S. & SIMMONS, A. 2012. Quantifying Experts’ Uncertainty About the Future Cost of Exotic Diseases. Risk Analysis 32. LACCARINO , G. Introduction to Uncertainty Quantification RTO-AVT-VKI Short Course [Online]. Available: http://www.stanford.edu/group/uq/events/UQLS_stanford/Iaccarino.pdf. MANIKANDAN, S. 2011 Measures of central tendency: Median and mode. J Pharmacol Pharmacother, 2, 214-215. MEYER, M. A. & BOOKER, J. M. 2001. Eliciting and Analyzing Expert Judgment: A Practical Guide, SIAM, Philadelphia, PA. NEW ZEALAND TREASURY. 2005. Cost Benefit Analysis Primer [Online]. The Treasury: Business Analysis Team. Available: http://www.treasury.govt.nz/publications/guidance/planning/costbenefitanalysis/primer/cba-primer-v12.pdf [Accessed]. OAKLEY, J. E. 2010. Eliciting Univariate Probability Distributions. Available: http://www.rochester.edu/college/psc/clarke/506/Oakley10.pdf. RENÉ VAN DORP, J. 2009. On Some Elicitation Procedures for Distributions with Bounded Support with Applications in PERT. In: KALLEN, M. J. & KUNIEWSKI, S. P. (eds.) Risk and Decision Analysis in Civil Engineering. IOS Press, The Netherlands. RICE, J. A. 2007. Mathematical Statistics and data analysis. Berkeley, United States of America: Thomson Brooks/Cole. SLOTTJE, P., SLUIJS, J. P. & KNOL, A. B. 2008. Expert Elicitation: Methodological suggestions for its use in environmental health impact assessments RIVM letter report 630004001/2008. RIVM. STATISTICS, A. B. O. Statistical Language - Measures of Spread [Online]. Available: http://www.abs.gov.au/websitedbs/a3121120.nsf/home/statistical+language [Accessed]. TAGHAVIFARD, M. T., DAMGHANI, K. K. & MOGHADDAM, R. T. 2009. Decision Making Under Uncertain and Risky Situations, Society of Actuaries. THOMPSON, K. M. 1999. Developing univariate distributions from data for risk analysis. Human and Ecological Risk Assessment. 5, 755-783. VOSE, D. 2002. Risk Analysis - A Quantitative Guide. , England, John Wiley & Sons Ltd. Read More