Appendix A


Glossary

TABLE A.1 Glossary of Terms Related to Verification, Validation, and Uncertainty Quantification


Term, with Synonyms and Cross-References Definition Notes and Comments

 
accuracy
See also precision.
A measure of agreement between the estimated value of some quantity and its true value. (Adapted from Society for Risk Analysis [SRA] Glossary.a) See note under precision.
 
adjoint map Given a map (i.e., forward model) from an input vector space to an output vector space, the adjoint is an associated map between the vector space of linear real-valued functions on the output space to the vector space of linear real-valued functions on the input space. Given a linear real function on the output space, a linear real function on the input space is obtained by first applying the original map to any specified vector in the input space and then applying the given linear real function on the output space. The adjoint map is important for determining properties of the original map when the input and output vectors cannot be observed directly. It plays a fundamental role in the theory of maps, e.g., for determining solvability of inverse problems, stability and sensitivity, Green’s functions, and derivatives of the output of a map with respect to the input. The concrete formulation and evaluation of an adjoint depend heavily on the properties of the original map (i.e., forward model) and the input and output spaces, with extra care needed for nonlinear maps.
 
aleatoric uncertainty
Synonyms: aleatoric probability, aleatoric uncertainty, systematic error See also probability, epistemic uncertainty.
A measure of the uncertainty of an unknown event whose occurrence is governed by some random physical phenomena that are either (1) predictable, in principle, with sufficient information (e.g., tossing a die), or (2) essentially unpredictable (radioactive decay).b See epistemic uncertainty.


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 109
Appendix A Glossary TABLE A.1 Glossary of Terms Related to Verification, Validation, and Uncertainty Quantification Term, with Synonyms and Cross-References Definition Notes and Comments accuracy See note under precision. A measure of agreement between the See also precision. estimated value of some quantity and its true value. (Adapted from Society for Risk Analysis [SRA] Glossary.a) adjoint map Given a map (i.e., forward model) from The adjoint map is important for an input vector space to an output vector determining properties of the original space, the adjoint is an associated map map when the input and output vectors between the vector space of linear real- cannot be observed directly. It plays a valued functions on the output space to fundamental role in the theory of maps, the vector space of linear real-valued e.g., for determining solvability of inverse functions on the input space. Given a problems, stability and sensitivity, Green’s linear real function on the output space, functions, and derivatives of the output a linear real function on the input space of a map with respect to the input. The is obtained by first applying the original concrete formulation and evaluation of an map to any specified vector in the input adjoint depend heavily on the properties space and then applying the given linear of the original map (i.e., forward model) real function on the output space. and the input and output spaces, with extra care needed for nonlinear maps. aleatoric uncertainty A measure of the uncertainty of an See epistemic uncertainty. Synonyms: aleatoric probability, aleatoric unknown event whose occurrence is uncertainty, systematic error governed by some random physical See also probability, epistemic phenomena that are either (1) predictable, uncertainty. in principle, with sufficient information (e.g., tossing a die), or (2) essentially unpredictable (radioactive decay).b 109

OCR for page 109
110 ASSESSING THE RELIABILITY OF COMPLEX MODELS Term, with Synonyms and Cross-References Definition Notes and Comments algorithm A finite list of well-defined instructions The instructions and executions are not that, when executed, proceed through a necessarily deterministic; some algorithms incorporate random input (see Monte finite number of well-defined successive Carlo simulation). states, eventually terminating and producing an output. approximation The result of a computation or assessment See also estimation (of parameters in that may not be exactly correct but that is probability models). adequate for a particular purpose.c average The sum of n numbers divided by n.d,e,f The average is a simple arithmetic Synonyms: arithmetic mean, sample mean operation requiring a set of n numbers. See also mean. It is often confused with the mean (or expected value), which is a property of a probability distribution. One reason for this confusion is that the average of a set of realizations of a random variable is often a good estimator of the mean of the random variable’s distribution. Bayesian approach An approach that uses observations (data) In most problems the Bayesian See also prior probability. to constrain uncertain parameters in a approach produces a high-dimensional probability distribution describing probabilistic model. The constrained uncertainty is described by a posterior the joint uncertainty in all of the model probability distribution, produced parameters. Functionals or integrals of this using Bayes’s theorem to combine the posterior distribution are typically used prior probability distribution with the to summarize the posterior uncertainty. probabilistic model of the observations. These summaries are typically produced by means of numerical approximation or sampling methods such as Markov chain Monte Carlo. code verification The process of determining and See also verification, solution verification. documenting the extent to which a computer program (“code”) correctly solves the equations of the mathematical model. computational model Computer code that (approximately) In physically based applications the Synonym: computer model solves the equations of the mathematical computational model might encode See also model (simulation). model. physical rules such as conservation of mass or momentum. In other applications the computational model might also produce a Monte Carlo or a discrete-event realization. conditional probability The probability of an event supposing In the Bayesian approach the posterior See also probability. (i.e., “conditioned on”) the occurrence of distribution is a conditional probability other specified events. distribution, conditioned on the physical observations. It is important to note that subjectively assessed probabilities are based on the state of knowledge that holds at the time of the probability assessment.

OCR for page 109
111 APPENDIX A Term, with Synonyms and Cross-References Definition Notes and Comments confidence interval A range of values [a, b] determined from Confidence intervals should not be Synonym: interval a sample, using a predetermined rule interpreted as implying that the parameter chosen such that, in repeated random itself has a range of values; it has only one samples from the same population, the value. For any given sample the confidence fraction α of computed ranges will limits a and b define a random range include the true value of an unknown within which the parameter of interest parameter. The values a and b are will lie with probability a (provided that called confidence limits; α is called the the actual population satisfies the initial confidence coefficient (commonly chosen hypothesis). to be .95 or .99); and 1 − α is called the confidence level. (Adapted from SRA Glossary.)a constrained uncertainty Uncertainty about a parameter, For most of the examples in this report, See also Bayesian approach. prediction, or other entity that has been uncertainty is constrained using the reduced by incorporating additional Bayesian approach, conditioning on information, such as new physical physical observations, producing a observations. posterior distribution for parameters and predictions. continuous random variable A random variable, X, is continuous if it See also cumulative distribution function, has an absolutely continuous cumulative probability density function. distribution function.d cumulative distribution function The probability that a random variable The cdf always exists for any random Synonyms: cumulative distribution, cdf, X will be less than or equal to a value x; variable; it is monotonic nondecreasing in x, and (being a probability 0 ≤ P{X ≤ x} distribution function written as P{X ≤ x}.f,g ≤  1. If P{X ≤ x} is absolutely continuous See also probability density function, probability distribution. in x, then X is called a continuous random variable; if it is discontinuous at a finite or countably infinite number of values of x, and constant otherwise, X is called a discrete random variable. data assimilation A recursive process for producing The combination method is usually based predictions with uncertainty regarding on Bayesian inference. The Kalman filter, some process, commonly used in weather the ensemble Kalman filter, and particle forecasting and other fields of geoscience. filters are examples of approaches with At a given iteration, new physical which data assimilation is carried out. observations are combined with model- based predictions to produce updated predictions and updated estimates of the current state of the system. data verification and validation The process of verifying the internal consistency and correctness of data and validating that they represent real- world entities appropriate for their intended purpose or an expected range of purposes.h discrete random variable A random variable that has a nonzero See also cumulative distribution function. probability for only a finite, or countably infinite, set of values.b

OCR for page 109
112 ASSESSING THE RELIABILITY OF COMPLEX MODELS Term, with Synonyms and Cross-References Definition Notes and Comments epistemic uncertainty A representation of uncertainty Some examples of epistemic uncertainty are (1) a probability density function Synonym: epistemic probability about propositions due to incomplete See also aleatoric uncertainty. knowledge. Such propositions may be describing uncertainty regarding the about either past or future events.b acceleration due to gravity at Earth’s surface; (2) determination of the probability that a required maintenance procedure will, in fact, be carried out. estimation (of parameters in probability A procedure by which sample data are Estimation procedures are usually based models) used to assess the value of an unknown on statistical analyses that address See also approximation. quantity.e their efficiency, effectiveness, limiting behaviors, degrees of bias, etc. The most common methods of parameter estimation are “maximum likelihood” and the method of moments. Under the Bayesian approach estimates can be produced by taking the mean, median, or most likely value determined by the posterior distribution. expected value The first moment of the probability Synonym: expectation distribution of a random variable X; often denoted as E(X) and defined as ∑ xip(xi) See also mean. if X is a discrete random variable and ∫ xf(x)dx if X is a continuous random variable.d, f extrapolative prediction The use of a model to make statements See also interpolative prediction. about quantities of interest (QOIs) in settings (initial conditions, physical regimes, parameter values, etc.) that are outside the conditions for which the model validation effort occurred. face validation A nonquantitative “sanity check” on a Face validation should not be used by See also validation. model that requires both its structural itself as a formal validation process. content and outputs to be consistent with Instead, it should be used to guide well-understood and agreed-on forms, model development, design of sensitivity ranges, etc. analyses, etc. forward problem The use of a model, given the values of See also inverse problem. all necessary inputs (initial conditions, parameters, etc.), to produce potentially observable QOIs. forward propagation Quantifying the uncertainty of a model’s Synonym: uncertainty propagation (UP) responses that results from uncertainty See also forward problem. in the model’s inputs being propagated through the model.

OCR for page 109
113 APPENDIX A Term, with Synonyms and Cross-References Definition Notes and Comments global statistical sensitivity analysis The study of how the uncertainty in the Global statistical sensitivity analysis is See also sensitivity analysis. output or QOI of a model (numerical distinguished from local, or one-at-a-time, or otherwise) can be apportioned to sensitivity analyses in that interactions and different sources of uncertainty in the nonlinearities are considered. model input. The term global ensures that the analysis considers more than just local or one-factor-at-a-time effects. Hence interactions and nonlinearities are important components of a global statistical sensitivity analysis. input verification The process of determining that the See also verification. data entered into a model or simulation accurately represent what the developer intends. (Adapted from DOD, 2009.h) interpolative prediction The use of a model to make statements In practice, it may be difficult to determine See also extrapolative prediction. about QOIs in regimes within which the if a particular prediction is an interpolation model has been validated. or not. intrusive methods Approaches to exploring a computational See also nonintrusive methods model that require a recoding of the (black box methods). model. Such a recoding might be done in order to efficiently produce derivative information using the adjoint equation to facilitate a sensitivity analysis. inverse problem An estimation of a model’s uncertain An inverse problem is often formulated as See also forward problem. parameters by using data, measurements, an optimization problem that minimizes an or observations. appropriate measure of the “differences” between observed and model-predicted outputs (with constraints—or penalty costs—on the values of some of the parameters). level of fidelity The amount of detail with which a model A high level of fidelity does not See also validation. describes an actual process. Relevant necessarily imply that the model will give features might include the descriptions highly accurate predictions for the system. of geometry, model symmetries, dimensionality, or physical processes in the model. High-fidelity models attempt to capture more of these features than do low-fidelity models. likelihood The likelihood, L(A | D), of an event, A, In informal usage, “likelihood” is often a See also probability, uncertainty. qualitative description of probability or given the data, D, and a specific model, is often taken to be proportional to frequency. However, equally often these P(D | A), the constant of proportionality descriptions do not satisfy the axioms of being arbitrary.i probability. linear regression Regression when the function to be fit is Synonym: regression linear in the independent variables. See also nonlinear regression.

OCR for page 109
114 ASSESSING THE RELIABILITY OF COMPLEX MODELS Term, with Synonyms and Cross-References Definition Notes and Comments Markov chain Monte Carlo (MCMC) A sampling technique that constructs a MCMC typically requires many fewer Markov chain to produce Monte Carlo points than grid-based sampling methods. samples from a typically complicated, MCMC approaches become intractable as multivariate distribution. The resulting the complexity of the forward problem and sample is then used to estimate the dimensions of the parameter spaces functionals of the distribution. increase. mathematical model A model that uses mathematical language Synonym: conceptual model (sets of equations, inequalities, etc.) to See also model (simulation). describe the behavior of a system. mean The first moment of a probability See also expected value, average. distribution, with the same mathematical definition as that of expected value. The mean is a parameter that represents the central tendency of a distribution.d,e,g,j measurement error The discrepancy between a measurement Measurement error is often decomposed and the quantity that the measurement into two components: replicate variation instrument is intended to measure.k and bias. model (simulation) A representation of some portion of the Mathematical models are used to aid our See also simulation. world in a readily manipulated form. A understanding of some aspects of the mathematical model is an abstraction real world and to aid in decision making. that uses mathematical language to They are also valuable rhetorical tools for describe the behavior of a system.l presenting the rationale supporting various decisions, since they arguably allow for transparency and the reproduction of results by others. However, models are only as good as their (validated) relationship to the real world and within the context for which they are designed. model discrepancy A term accounting for or describing the In some cases, model discrepancy is the difference between a model of the system dominant source of uncertainty in model- Synonyms: model inadequacy, structural error and the true physical system. based predictions. When relevant physical data are available, model discrepancy can be estimated. Estimating this term when relevant physical observations are not available is difficult. Monte Carlo simulation Each set of “runs” of a simulation A model constructed so that the input of See also model (simulation). a large number of random draws from inherently represents the outcomes of a defined probability distributions will series of experiments. The analysis of generate outputs that are representative simulation output data therefore requires of the random behavior of a particular a proper experimental design, followed by system, phenomenon, consequences, etc., the use of statistical techniques to estimate of a series of events.m parameters, test hypotheses, etc. multiscale phenomena Equations representing the dynamics The analysis of multiscale phenomena of a nonlinear system that combine the presents many challenges to numerical behavior at many scales of physical analysis and associated software, so that dimension and/or time. the coupling of results from one scale to those of another may lead to instability in the model output that might not represent physical reality.

OCR for page 109
115 APPENDIX A Term, with Synonyms and Cross-References Definition Notes and Comments multivariate adaptive regression splines A form of nonparametric regression (MARS) analysis (usually presented as an See also regression. extension of linear regression) that automatically represents nonlinearities and interactions in terms of splines (e.g., functions having smooth first and second derivatives).n nonintrusive methods Methods to carry out sensitivity analysis (black box methods) or forward propagation or to solve the inverse problem that only require forward runs of the computational model, effectively treating the model as a black box. nonlinear regression Regression when the function to be fit is See also regression, linear regression. nonlinear in the independent variables. parameter Terms in a mathematical function that Often parameters are fixed at assumed remain fixed during any computational values, or they can be estimated using procedure. These may include initial physical observations. Alternatively, conditions, physical constants, boundary uncertainty regarding parameters may be values, etc. constrained with physical data. polynomial chaos A parameterization of random variables The coefficients in these representations Synonym: PC, Wiener chaos expansion and processes that lends itself to the can be estimated in a number of ways, See also Monte Carlo simulation. characterization of transformations including Galerkin projections, least between input and output quantities. squares, perturbation expansions, statistical The resulting representations are akin sampling, and numerical quadrature. to a response surface with respect to normalized random variables and can be readily evaluated, yielding very efficient procedures for sampling the output variables. posterior probability Probability distribution describing The Bayesian approach updates the prior See also Bayesian approach, prior probability distribution by conditioning uncertainty in parameters (and possibly probability. other random quantities) of interest in a on the data (often physical observations), statistical model after data are observed producing a posterior distribution for the and conditioned on. same parameters. Often of interest is the posterior predictive distribution for a QOI, describing uncertainty about the QOI for the physical system. precision The implied degree of certainty with Consider two statements assessing See also accuracy. which a value is stated, as reflected in Bill Gates’s net worth, W. A precise the number of significant digits used but inaccurate assessment is “W = to express the value—the more digits, $123,472.89.” An imprecise but accurate the more precision. (Adapted from SRA assessment is “W > $6 billion.” Glossary.a)

OCR for page 109
116 ASSESSING THE RELIABILITY OF COMPLEX MODELS Term, with Synonyms and Cross-References Definition Notes and Comments prediction uncertainty The uncertainty associated with a This is a statement about reality, given prediction about a QOI for the real-world information from an analysis typically involving a computational model, process. The prediction uncertainty could be described by a posterior distribution physical observations, and possibly other for the QOI, a predictive distribution, information sources. a confidence interval, or possibly some other representation. prior probability Bayesian approach updates this prior Probability distribution assigned to Synonym: a priori probability parameters (and possibly other random probability distribution by conditioning See also Bayesian approach, posterior quantities) of interest in a statistical on the physical observations, producing probability. model before physical observations are a posterior distribution for the same available. parameters. Obtaining the prior distribution may be done using expert judgment or previous data, or it may be specified to be “neutral” to the analysis. probability One of a set of numerical values between This definition holds for all quantification See also likelihood, conditional of uncertainty: subjective or frequentist. 0 and 1 assigned to a collection of probability, aleatoric uncertainty, random events (which are subsets of a subjective probability. sample space) in such a way that the assigned numbers obey two axioms: (1) 0 ≤ P{A} ≤ 1 for any A and (2) P{A} + P{B} = P{A  B} for two mutually exclusive events A and B.j probability density function (pdf) The pdf is the common way to represent The derivative of an absolutely the probability distribution of a continuous cumulative distribution continuous random variable, because function. j its shape often displays the central tendency (mean) and variability (standard For a scalar random variable X, a deviation). From its definition, P{a < X function f such that, for any two numbers, a and b, with a ≤ b, ≤ b} is the integral of the pdf between a P{a ≤ X ≤  b} = ∫ab f(x)dx. and b. probability distribution See cumulative distribution function. probability elicitation A process of gathering, structuring, There are many approaches for probability Synonyms: probability assessment, and encoding expert judgment (about elicitation, the most common of which subjective probability uncertain events or quantities) in the are those used for obtaining a priori form of probability statements about subjective probabilities. Note that the future events.o results of probability elicitations are sometimes called probability assessments or assignments. quantity of interest (QOI) A numerical characteristic of the system being modeled, the value of which is of interest to stakeholders, typically because it informs a decision. To be useful the model must be able to provide, as output, values of or probability statements about QOIs.

OCR for page 109
117 APPENDIX A Term, with Synonyms and Cross-References Definition Notes and Comments reduced model A low-fidelity model developed to A reduced model is particularly useful for Synonym: emulator replace (or augment) a computationally carrying out computationally demanding analysis (e.g., sensitivity analysis, demanding, high-fidelity model. forward propagation of uncertainty, solving the inverse problem) that would be infeasible with the original model. Sometimes a reduced model “collapses” aspects of a “physics-based” model so as to be referred to as a “physics-blind” model. regression A form of statistical analysis in which See also: linear regression, nonlinear observational data are used to statistically regression. fit a mathematical function that presents the data (i.e., dependent variables) as a function of a set of parameters and one or more independent variables. response surface A function that predicts outputs from a A response surface can be used See also sensitivity analysis. like a reduced model to carry out model as a function of the model inputs. A response surface is typically estimated computationally demanding analyses (e.g., sensitivity analysis, forward from an ensemble of model runs using a propagation, solving the inverse regression, Gaussian process modeling, problem). Since the response surface does or some other estimation or interpolation not exactly reproduce the computational procedure. model, there is typically additional error in results produced by response surface approaches. robustness analysis For a prescriptive model, a procedure that See also sensitivity analysis. analyzes the degree to which deviations from a “best” decision provide suboptimal values of the desired criterion. These deviations can be due to uncertainty in model formulation, assumed parameter values, etc. sensitivity analysis An exploration, often by numerical See also robustness analysis. (rather than analytical) means, of how model outputs (particularly QOIs) are affected by changes in the inputs (parameter values, assumptions, etc.). simulation Many uncertainty quantification (UQ) The execution of a computer code to Synonym: model mimic an actual system. methods use an ensemble of simulations, See also Monte Carlo simulation. or model runs, to construct emulators, carry out sensitivity analysis, etc. solution verification The process of determining as completely See also verification, code verification. as possible the accuracy with which the algorithms solve the mathematical-model equations for a specified QOI. standard deviation The square root of the variance of a See also variance. distribution. j

OCR for page 109
118 ASSESSING THE RELIABILITY OF COMPLEX MODELS Term, with Synonyms and Cross-References Definition Notes and Comments stochastic Pertaining to a sequence of observations, Often informally used as a synonym of See also probability. each of which can be considered to be a “probabilistic.” sample from a probability distribution. subjective probability Expert judgment about uncertain events See also probability elicitation. or quantities, in the form of probability statements about future events. It is not based on any precise computation but is often a reasonable assessment by a knowledgeable person. uncertainty The condition of being unsure about For the purpose of this report, uncertainty See also probability, aleatoric probability, something; a lack of assurance or is often described regarding a QOI of the epistemic uncertainty. conviction.c true, physical system. This uncertainty depends on a model-based prediction, as well as on other information included in the VVUQ assessment. This uncertainty can be described using probability. uncertainty quantification (UQ) The process of quantifying uncertainties More broadly, UQ can be thought of as the in a computed QOI, with the goals of field of research that uses and develops accounting for all sources of uncertainty theory, methodology, and approaches and quantifying the contributions of for carrying out inference, with the aid of computational models, on complex specific sources to the overall uncertainty. systems. validation The process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model.p variance The second moment of a probability The variance is a common measure See also standard deviation. distribution, defined as E(X – µ)2, of variability around the mean of a distribution. Its square root, the standard where µ is the first moment of the deviation, having dimensional units of random variable X. the random variable, is a more intuitively meaningful measure of dispersion from the mean. verification The process of determining whether a See also code verification, solution computer program (“code”) correctly verification. solves the mathematical-model equations. This includes code verification (determining whether the code correctly implements the intended algorithms) and solution verification (determining the accuracy with which the algorithms solve the mathematical-model equations for specified QOIs). a Society for Risk Analysis (SRA), Glossary of Risk Analysis Terms. Available at sra.org/resources_glossary.php. b Cornell LCS Statistics Laboratory. See http://instruct1.cit.cornell.edu:8000/courses/statslab/Stuff/indes.php. c American Heritage Dictionary. 2000. Boston: Houghton, Mifflin. d Glossary of Statistics Terms. Available at http://www.stat.berkeley.edu/ users/stark/SticiGui/Text/gloss.htm. e Statistical Education Through Problem Solving [STEP] Consortium. Available at http://www.stats.gla.ac.uk/steps/index.html. f W. Feller. 1968. An Introduction to Probability Theory and Its Applications. New York, N.Y.: Wiley. g J.L. Devore. 2000. Probability and Statistics for Engineering and the Sciences. Pacific Grove, Calif.: Duxbury Press.

OCR for page 109
119 APPENDIX A h DOD (Department of Defense). 2009. Instruction 5000.61. December 9. Washington, D.C. i A.W.F. Edwards. 1992. Likelihood. Baltimore, Md.: Johns Hopkins University Press. j S.M. Ross. 2000. Introduction to Probability Models. New York: Academic Press. k Duke University. 1998. Statistical and Data Analysis for Biological Sciences. Available at http://www.isds.duke.edu/courses/Fall98/sta210b/ terms.html. l R. Aris. 1995. Mathematical Modelling Techniques, New York: Dover. m E.J. Henley and H. Kunmamoto. 1981. Reliability Engineering and Risk Assessment. Upper Saddle River, N.J.: Prentice-Hall. n J.H. Friedman. 1991. Multivariate Adaptive Regression Splines. The Annals of Statistics 19(1):1-67. o M.S. Meyer and J.M. Booker. 1998. Eliciting and Analyzing Expert Judgment. LA-UR-99-1659. Los Alamos, N.Mex.: Los Alamos National Laboratory. p American Institute for Aeronautics and Astronautics. 1998. Guide for the Verification and Validation of Computational Fluid Dynamics Simu- lations. Reston, Va.: American Institute for Aeronautics and Astronautics.