Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 109
Appendix A
Glossary
TABLE A.1 Glossary of Terms Related to Verification, Validation, and Uncertainty Quantification
Term, with Synonyms
and Cross-References Definition Notes and Comments
accuracy See note under precision.
A measure of agreement between the
See also precision. estimated value of some quantity and its
true value. (Adapted from Society for
Risk Analysis [SRA] Glossary.a)
adjoint map Given a map (i.e., forward model) from The adjoint map is important for
an input vector space to an output vector determining properties of the original
space, the adjoint is an associated map map when the input and output vectors
between the vector space of linear real- cannot be observed directly. It plays a
valued functions on the output space to fundamental role in the theory of maps,
the vector space of linear real-valued e.g., for determining solvability of inverse
functions on the input space. Given a problems, stability and sensitivity, Green’s
linear real function on the output space, functions, and derivatives of the output
a linear real function on the input space of a map with respect to the input. The
is obtained by first applying the original concrete formulation and evaluation of an
map to any specified vector in the input adjoint depend heavily on the properties
space and then applying the given linear of the original map (i.e., forward model)
real function on the output space. and the input and output spaces, with extra
care needed for nonlinear maps.
aleatoric uncertainty A measure of the uncertainty of an See epistemic uncertainty.
Synonyms: aleatoric probability, aleatoric unknown event whose occurrence is
uncertainty, systematic error governed by some random physical
See also probability, epistemic phenomena that are either (1) predictable,
uncertainty. in principle, with sufficient information
(e.g., tossing a die), or (2) essentially
unpredictable (radioactive decay).b
109
OCR for page 110
110 ASSESSING THE RELIABILITY OF COMPLEX MODELS
Term, with Synonyms
and Cross-References Definition Notes and Comments
algorithm A finite list of well-defined instructions The instructions and executions are not
that, when executed, proceed through a necessarily deterministic; some algorithms
incorporate random input (see Monte
finite number of well-defined successive
Carlo simulation).
states, eventually terminating and
producing an output.
approximation The result of a computation or assessment
See also estimation (of parameters in that may not be exactly correct but that is
probability models). adequate for a particular purpose.c
average The sum of n numbers divided by n.d,e,f The average is a simple arithmetic
Synonyms: arithmetic mean, sample mean operation requiring a set of n numbers.
See also mean. It is often confused with the mean (or
expected value), which is a property of
a probability distribution. One reason
for this confusion is that the average of a
set of realizations of a random variable is
often a good estimator of the mean of the
random variable’s distribution.
Bayesian approach An approach that uses observations (data) In most problems the Bayesian
See also prior probability. to constrain uncertain parameters in a approach produces a high-dimensional
probability distribution describing
probabilistic model. The constrained
uncertainty is described by a posterior the joint uncertainty in all of the model
probability distribution, produced parameters. Functionals or integrals of this
using Bayes’s theorem to combine the posterior distribution are typically used
prior probability distribution with the to summarize the posterior uncertainty.
probabilistic model of the observations. These summaries are typically produced
by means of numerical approximation or
sampling methods such as Markov chain
Monte Carlo.
code verification The process of determining and
See also verification, solution verification. documenting the extent to which a
computer program (“code”) correctly
solves the equations of the mathematical
model.
computational model Computer code that (approximately) In physically based applications the
Synonym: computer model solves the equations of the mathematical computational model might encode
See also model (simulation). model. physical rules such as conservation of mass
or momentum. In other applications the
computational model might also produce a
Monte Carlo or a discrete-event realization.
conditional probability The probability of an event supposing In the Bayesian approach the posterior
See also probability. (i.e., “conditioned on”) the occurrence of distribution is a conditional probability
other specified events. distribution, conditioned on the physical
observations. It is important to note that
subjectively assessed probabilities are
based on the state of knowledge that holds
at the time of the probability assessment.
OCR for page 111
111
APPENDIX A
Term, with Synonyms
and Cross-References Definition Notes and Comments
confidence interval A range of values [a, b] determined from Confidence intervals should not be
Synonym: interval a sample, using a predetermined rule interpreted as implying that the parameter
chosen such that, in repeated random itself has a range of values; it has only one
samples from the same population, the value. For any given sample the confidence
fraction α of computed ranges will limits a and b define a random range
include the true value of an unknown within which the parameter of interest
parameter. The values a and b are will lie with probability a (provided that
called confidence limits; α is called the the actual population satisfies the initial
confidence coefficient (commonly chosen hypothesis).
to be .95 or .99); and 1 − α is called the
confidence level. (Adapted from SRA
Glossary.)a
constrained uncertainty Uncertainty about a parameter, For most of the examples in this report,
See also Bayesian approach. prediction, or other entity that has been uncertainty is constrained using the
reduced by incorporating additional Bayesian approach, conditioning on
information, such as new physical physical observations, producing a
observations. posterior distribution for parameters and
predictions.
continuous random variable A random variable, X, is continuous if it
See also cumulative distribution function, has an absolutely continuous cumulative
probability density function. distribution function.d
cumulative distribution function The probability that a random variable The cdf always exists for any random
Synonyms: cumulative distribution, cdf, X will be less than or equal to a value x; variable; it is monotonic nondecreasing in
x, and (being a probability 0 ≤ P{X ≤ x}
distribution function written as
P{X ≤ x}.f,g ≤ 1. If P{X ≤ x} is absolutely continuous
See also probability density function,
probability distribution. in x, then X is called a continuous
random variable; if it is discontinuous
at a finite or countably infinite number of
values of x, and constant otherwise, X is
called a discrete random variable.
data assimilation A recursive process for producing The combination method is usually based
predictions with uncertainty regarding on Bayesian inference. The Kalman filter,
some process, commonly used in weather the ensemble Kalman filter, and particle
forecasting and other fields of geoscience. filters are examples of approaches with
At a given iteration, new physical which data assimilation is carried out.
observations are combined with model-
based predictions to produce updated
predictions and updated estimates of the
current state of the system.
data verification and validation The process of verifying the internal
consistency and correctness of data
and validating that they represent real-
world entities appropriate for their
intended purpose or an expected range of
purposes.h
discrete random variable A random variable that has a nonzero
See also cumulative distribution function. probability for only a finite, or countably
infinite, set of values.b
OCR for page 112
112 ASSESSING THE RELIABILITY OF COMPLEX MODELS
Term, with Synonyms
and Cross-References Definition Notes and Comments
epistemic uncertainty A representation of uncertainty Some examples of epistemic uncertainty
are (1) a probability density function
Synonym: epistemic probability about propositions due to incomplete
See also aleatoric uncertainty. knowledge. Such propositions may be describing uncertainty regarding the
about either past or future events.b acceleration due to gravity at Earth’s
surface; (2) determination of the
probability that a required maintenance
procedure will, in fact, be carried out.
estimation (of parameters in probability A procedure by which sample data are Estimation procedures are usually based
models) used to assess the value of an unknown on statistical analyses that address
See also approximation. quantity.e their efficiency, effectiveness, limiting
behaviors, degrees of bias, etc. The most
common methods of parameter estimation
are “maximum likelihood” and the
method of moments. Under the Bayesian
approach estimates can be produced
by taking the mean, median, or most
likely value determined by the posterior
distribution.
expected value The first moment of the probability
Synonym: expectation distribution of a random variable X; often
denoted as E(X) and defined as ∑ xip(xi)
See also mean.
if X is a discrete random variable and
∫ xf(x)dx if X is a continuous random
variable.d, f
extrapolative prediction The use of a model to make statements
See also interpolative prediction. about quantities of interest (QOIs)
in settings (initial conditions, physical
regimes, parameter values, etc.) that
are outside the conditions for which the
model validation effort occurred.
face validation A nonquantitative “sanity check” on a Face validation should not be used by
See also validation. model that requires both its structural itself as a formal validation process.
content and outputs to be consistent with Instead, it should be used to guide
well-understood and agreed-on forms, model development, design of sensitivity
ranges, etc. analyses, etc.
forward problem The use of a model, given the values of
See also inverse problem. all necessary inputs (initial conditions,
parameters, etc.), to produce potentially
observable QOIs.
forward propagation Quantifying the uncertainty of a model’s
Synonym: uncertainty propagation (UP) responses that results from uncertainty
See also forward problem. in the model’s inputs being propagated
through the model.
OCR for page 113
113
APPENDIX A
Term, with Synonyms
and Cross-References Definition Notes and Comments
global statistical sensitivity analysis The study of how the uncertainty in the Global statistical sensitivity analysis is
See also sensitivity analysis. output or QOI of a model (numerical distinguished from local, or one-at-a-time,
or otherwise) can be apportioned to sensitivity analyses in that interactions and
different sources of uncertainty in the nonlinearities are considered.
model input. The term global ensures
that the analysis considers more than
just local or one-factor-at-a-time effects.
Hence interactions and nonlinearities
are important components of a global
statistical sensitivity analysis.
input verification The process of determining that the
See also verification. data entered into a model or simulation
accurately represent what the developer
intends. (Adapted from DOD, 2009.h)
interpolative prediction The use of a model to make statements In practice, it may be difficult to determine
See also extrapolative prediction. about QOIs in regimes within which the if a particular prediction is an interpolation
model has been validated. or not.
intrusive methods Approaches to exploring a computational
See also nonintrusive methods model that require a recoding of the
(black box methods). model. Such a recoding might be done
in order to efficiently produce derivative
information using the adjoint equation to
facilitate a sensitivity analysis.
inverse problem An estimation of a model’s uncertain An inverse problem is often formulated as
See also forward problem. parameters by using data, measurements, an optimization problem that minimizes an
or observations. appropriate measure of the “differences”
between observed and model-predicted
outputs (with constraints—or penalty
costs—on the values of some of the
parameters).
level of fidelity The amount of detail with which a model A high level of fidelity does not
See also validation. describes an actual process. Relevant necessarily imply that the model will give
features might include the descriptions highly accurate predictions for the system.
of geometry, model symmetries,
dimensionality, or physical processes in
the model. High-fidelity models attempt
to capture more of these features than do
low-fidelity models.
likelihood The likelihood, L(A | D), of an event, A, In informal usage, “likelihood” is often a
See also probability, uncertainty. qualitative description of probability or
given the data, D, and a specific model,
is often taken to be proportional to frequency. However, equally often these
P(D | A), the constant of proportionality descriptions do not satisfy the axioms of
being arbitrary.i probability.
linear regression Regression when the function to be fit is
Synonym: regression linear in the independent variables.
See also nonlinear regression.
OCR for page 114
114 ASSESSING THE RELIABILITY OF COMPLEX MODELS
Term, with Synonyms
and Cross-References Definition Notes and Comments
Markov chain Monte Carlo (MCMC) A sampling technique that constructs a MCMC typically requires many fewer
Markov chain to produce Monte Carlo points than grid-based sampling methods.
samples from a typically complicated, MCMC approaches become intractable as
multivariate distribution. The resulting the complexity of the forward problem and
sample is then used to estimate the dimensions of the parameter spaces
functionals of the distribution. increase.
mathematical model A model that uses mathematical language
Synonym: conceptual model (sets of equations, inequalities, etc.) to
See also model (simulation). describe the behavior of a system.
mean The first moment of a probability
See also expected value, average. distribution, with the same mathematical
definition as that of expected value. The
mean is a parameter that represents the
central tendency of a distribution.d,e,g,j
measurement error The discrepancy between a measurement Measurement error is often decomposed
and the quantity that the measurement into two components: replicate variation
instrument is intended to measure.k and bias.
model (simulation) A representation of some portion of the Mathematical models are used to aid our
See also simulation. world in a readily manipulated form. A understanding of some aspects of the
mathematical model is an abstraction real world and to aid in decision making.
that uses mathematical language to They are also valuable rhetorical tools for
describe the behavior of a system.l presenting the rationale supporting various
decisions, since they arguably allow
for transparency and the reproduction
of results by others. However, models
are only as good as their (validated)
relationship to the real world and within
the context for which they are designed.
model discrepancy A term accounting for or describing the In some cases, model discrepancy is the
difference between a model of the system dominant source of uncertainty in model-
Synonyms: model inadequacy, structural
error and the true physical system. based predictions. When relevant physical
data are available, model discrepancy can
be estimated. Estimating this term when
relevant physical observations are not
available is difficult.
Monte Carlo simulation Each set of “runs” of a simulation
A model constructed so that the input of
See also model (simulation). a large number of random draws from inherently represents the outcomes of a
defined probability distributions will series of experiments. The analysis of
generate outputs that are representative simulation output data therefore requires
of the random behavior of a particular a proper experimental design, followed by
system, phenomenon, consequences, etc., the use of statistical techniques to estimate
of a series of events.m parameters, test hypotheses, etc.
multiscale phenomena Equations representing the dynamics The analysis of multiscale phenomena
of a nonlinear system that combine the presents many challenges to numerical
behavior at many scales of physical analysis and associated software, so that
dimension and/or time. the coupling of results from one scale to
those of another may lead to instability in
the model output that might not represent
physical reality.
OCR for page 115
115
APPENDIX A
Term, with Synonyms
and Cross-References Definition Notes and Comments
multivariate adaptive regression splines A form of nonparametric regression
(MARS) analysis (usually presented as an
See also regression. extension of linear regression) that
automatically represents nonlinearities
and interactions in terms of splines (e.g.,
functions having smooth first and second
derivatives).n
nonintrusive methods Methods to carry out sensitivity analysis
(black box methods) or forward propagation or to solve
the inverse problem that only require
forward runs of the computational
model, effectively treating the model as a
black box.
nonlinear regression Regression when the function to be fit is
See also regression, linear regression. nonlinear in the independent variables.
parameter Terms in a mathematical function that Often parameters are fixed at assumed
remain fixed during any computational values, or they can be estimated using
procedure. These may include initial physical observations. Alternatively,
conditions, physical constants, boundary uncertainty regarding parameters may be
values, etc. constrained with physical data.
polynomial chaos A parameterization of random variables The coefficients in these representations
Synonym: PC, Wiener chaos expansion and processes that lends itself to the can be estimated in a number of ways,
See also Monte Carlo simulation. characterization of transformations including Galerkin projections, least
between input and output quantities. squares, perturbation expansions, statistical
The resulting representations are akin sampling, and numerical quadrature.
to a response surface with respect to
normalized random variables and can be
readily evaluated, yielding very efficient
procedures for sampling the output
variables.
posterior probability Probability distribution describing The Bayesian approach updates the prior
See also Bayesian approach, prior probability distribution by conditioning
uncertainty in parameters (and possibly
probability. other random quantities) of interest in a on the data (often physical observations),
statistical model after data are observed producing a posterior distribution for the
and conditioned on. same parameters. Often of interest is the
posterior predictive distribution for a QOI,
describing uncertainty about the QOI for
the physical system.
precision The implied degree of certainty with Consider two statements assessing
See also accuracy. which a value is stated, as reflected in Bill Gates’s net worth, W. A precise
the number of significant digits used but inaccurate assessment is “W =
to express the value—the more digits, $123,472.89.” An imprecise but accurate
the more precision. (Adapted from SRA assessment is “W > $6 billion.”
Glossary.a)
OCR for page 116
116 ASSESSING THE RELIABILITY OF COMPLEX MODELS
Term, with Synonyms
and Cross-References Definition Notes and Comments
prediction uncertainty The uncertainty associated with a This is a statement about reality, given
prediction about a QOI for the real-world information from an analysis typically
involving a computational model,
process. The prediction uncertainty could
be described by a posterior distribution physical observations, and possibly other
for the QOI, a predictive distribution, information sources.
a confidence interval, or possibly some
other representation.
prior probability Bayesian approach updates this prior
Probability distribution assigned to
Synonym: a priori probability parameters (and possibly other random probability distribution by conditioning
See also Bayesian approach, posterior quantities) of interest in a statistical on the physical observations, producing
probability. model before physical observations are a posterior distribution for the same
available. parameters. Obtaining the prior distribution
may be done using expert judgment or
previous data, or it may be specified to be
“neutral” to the analysis.
probability One of a set of numerical values between This definition holds for all quantification
See also likelihood, conditional of uncertainty: subjective or frequentist.
0 and 1 assigned to a collection of
probability, aleatoric uncertainty, random events (which are subsets of a
subjective probability. sample space) in such a way that the
assigned numbers obey two axioms:
(1) 0 ≤ P{A} ≤ 1 for any A and
(2) P{A} + P{B} = P{A B} for two
mutually exclusive events A and B.j
probability density function (pdf) The pdf is the common way to represent
The derivative of an absolutely
the probability distribution of a
continuous cumulative distribution
continuous random variable, because
function. j
its shape often displays the central
tendency (mean) and variability (standard
For a scalar random variable X, a
deviation). From its definition, P{a < X
function f such that, for any two
numbers, a and b, with a ≤ b, ≤ b} is the integral of the pdf between a
P{a ≤ X ≤ b} = ∫ab f(x)dx. and b.
probability distribution See cumulative distribution function.
probability elicitation A process of gathering, structuring, There are many approaches for probability
Synonyms: probability assessment, and encoding expert judgment (about elicitation, the most common of which
subjective probability uncertain events or quantities) in the are those used for obtaining a priori
form of probability statements about subjective probabilities. Note that the
future events.o results of probability elicitations are
sometimes called probability assessments
or assignments.
quantity of interest (QOI) A numerical characteristic of the system
being modeled, the value of which is of
interest to stakeholders, typically because
it informs a decision. To be useful the
model must be able to provide, as output,
values of or probability statements about
QOIs.
OCR for page 117
117
APPENDIX A
Term, with Synonyms
and Cross-References Definition Notes and Comments
reduced model A low-fidelity model developed to A reduced model is particularly useful for
Synonym: emulator replace (or augment) a computationally carrying out computationally demanding
analysis (e.g., sensitivity analysis,
demanding, high-fidelity model.
forward propagation of uncertainty,
solving the inverse problem) that would
be infeasible with the original model.
Sometimes a reduced model “collapses”
aspects of a “physics-based” model so
as to be referred to as a “physics-blind”
model.
regression A form of statistical analysis in which
See also: linear regression, nonlinear observational data are used to statistically
regression. fit a mathematical function that presents
the data (i.e., dependent variables) as a
function of a set of parameters and one or
more independent variables.
response surface A function that predicts outputs from a A response surface can be used
See also sensitivity analysis. like a reduced model to carry out
model as a function of the model inputs.
A response surface is typically estimated computationally demanding analyses
(e.g., sensitivity analysis, forward
from an ensemble of model runs using a
propagation, solving the inverse
regression, Gaussian process modeling,
problem). Since the response surface does
or some other estimation or interpolation
not exactly reproduce the computational
procedure.
model, there is typically additional error
in results produced by response surface
approaches.
robustness analysis For a prescriptive model, a procedure that
See also sensitivity analysis. analyzes the degree to which deviations
from a “best” decision provide suboptimal
values of the desired criterion. These
deviations can be due to uncertainty in
model formulation, assumed parameter
values, etc.
sensitivity analysis An exploration, often by numerical
See also robustness analysis. (rather than analytical) means, of how
model outputs (particularly QOIs)
are affected by changes in the inputs
(parameter values, assumptions, etc.).
simulation Many uncertainty quantification (UQ)
The execution of a computer code to
Synonym: model mimic an actual system. methods use an ensemble of simulations,
See also Monte Carlo simulation. or model runs, to construct emulators,
carry out sensitivity analysis, etc.
solution verification The process of determining as completely
See also verification, code verification. as possible the accuracy with which the
algorithms solve the mathematical-model
equations for a specified QOI.
standard deviation The square root of the variance of a
See also variance. distribution. j
OCR for page 118
118 ASSESSING THE RELIABILITY OF COMPLEX MODELS
Term, with Synonyms
and Cross-References Definition Notes and Comments
stochastic Pertaining to a sequence of observations, Often informally used as a synonym of
See also probability. each of which can be considered to be a “probabilistic.”
sample from a probability distribution.
subjective probability Expert judgment about uncertain events
See also probability elicitation. or quantities, in the form of probability
statements about future events. It is not
based on any precise computation but
is often a reasonable assessment by a
knowledgeable person.
uncertainty The condition of being unsure about For the purpose of this report, uncertainty
See also probability, aleatoric probability, something; a lack of assurance or is often described regarding a QOI of the
epistemic uncertainty. conviction.c true, physical system. This uncertainty
depends on a model-based prediction, as
well as on other information included in
the VVUQ assessment. This uncertainty
can be described using probability.
uncertainty quantification (UQ) The process of quantifying uncertainties More broadly, UQ can be thought of as the
in a computed QOI, with the goals of field of research that uses and develops
accounting for all sources of uncertainty theory, methodology, and approaches
and quantifying the contributions of for carrying out inference, with the aid
of computational models, on complex
specific sources to the overall uncertainty.
systems.
validation The process of determining the degree
to which a model is an accurate
representation of the real world from the
perspective of the intended uses of the
model.p
variance The second moment of a probability The variance is a common measure
See also standard deviation. distribution, defined as E(X – µ)2, of variability around the mean of a
distribution. Its square root, the standard
where µ is the first moment of the
deviation, having dimensional units of
random variable X.
the random variable, is a more intuitively
meaningful measure of dispersion from the
mean.
verification The process of determining whether a
See also code verification, solution computer program (“code”) correctly
verification. solves the mathematical-model equations.
This includes code verification
(determining whether the code correctly
implements the intended algorithms) and
solution verification (determining the
accuracy with which the algorithms solve
the mathematical-model equations for
specified QOIs).
a Society for Risk Analysis (SRA), Glossary of Risk Analysis Terms. Available at sra.org/resources_glossary.php.
b Cornell LCS Statistics Laboratory. See http://instruct1.cit.cornell.edu:8000/courses/statslab/Stuff/indes.php.
c American Heritage Dictionary. 2000. Boston: Houghton, Mifflin.
d Glossary of Statistics Terms. Available at http://www.stat.berkeley.edu/ users/stark/SticiGui/Text/gloss.htm.
e Statistical Education Through Problem Solving [STEP] Consortium. Available at http://www.stats.gla.ac.uk/steps/index.html.
f W. Feller. 1968. An Introduction to Probability Theory and Its Applications. New York, N.Y.: Wiley.
g J.L. Devore. 2000. Probability and Statistics for Engineering and the Sciences. Pacific Grove, Calif.: Duxbury Press.
OCR for page 119
119
APPENDIX A
h DOD (Department of Defense). 2009. Instruction 5000.61. December 9. Washington, D.C.
i A.W.F. Edwards. 1992. Likelihood. Baltimore, Md.: Johns Hopkins University Press.
j S.M. Ross. 2000. Introduction to Probability Models. New York: Academic Press.
k Duke University. 1998. Statistical and Data Analysis for Biological Sciences. Available at http://www.isds.duke.edu/courses/Fall98/sta210b/
terms.html.
l R. Aris. 1995. Mathematical Modelling Techniques, New York: Dover.
m E.J. Henley and H. Kunmamoto. 1981. Reliability Engineering and Risk Assessment. Upper Saddle River, N.J.: Prentice-Hall.
n J.H. Friedman. 1991. Multivariate Adaptive Regression Splines. The Annals of Statistics 19(1):1-67.
o M.S. Meyer and J.M. Booker. 1998. Eliciting and Analyzing Expert Judgment. LA-UR-99-1659. Los Alamos, N.Mex.: Los Alamos National
Laboratory.
p American Institute for Aeronautics and Astronautics. 1998. Guide for the Verification and Validation of Computational Fluid Dynamics Simu-
lations. Reston, Va.: American Institute for Aeronautics and Astronautics.
