physics, the method of manufactured solutions, and so on), among other issues. Similarly, software quality assurance involves decisions about the different types and the extent of software testing, coverage analysis, and so on. Validation studies also involve many activities, ranging from the choice of input space, to the design and fielding of experiments, to the selection of emulators, to the analysis of output data, and so on. Again, there are many important choices to be made within each of these activities and many important decisions to be made on how to trade off resources and time among them.

The results of a UQ study can help to inform the decision maker on the relative priorities among a broad set of choices. These choices can be viewed as a large set of possible trade-offs through which the uncertainty is managed (an uncertainty management trade space). Components of this trade space include the following:

•  Fundamental improvements to physics models,

•  Improvements to the integrated simulation and modeling capability,

•  The design and conduct of computer experiments,

•  The design and conduct of relevant constraining physical experiments, and

•  The engineering and design of the system to tolerate the predicted uncertainty.

The first four of these activities are typically (although not exclusively) considered as decisions within VVUQ. The final activity is typically considered to occur after the completion of the VVUQ study (treated in more detail in Section 6.3). All of these activities require resources, which include employing domain experts, accessing computational and experimental facilities, and influencing engineering design decisions. Decision makers must allocate resources throughout the VVUQ process, keeping in mind the goal of the study. For example, decision makers must weigh the relative benefit of investing in improvements to the fidelity of a given physics model against the benefit of conducting relevant physical experiments for calibration. Computing resources must be allocated across studies investigating detailed convergence, model fidelity, and completeness of UQ ensembles. Physical experiments must be selected from choices ranging from experiments involving components to fully integrated experiments. In industrial contexts, it is not uncommon for there to be a single budget for the entire process of modeling, simulator development, and UQ analysis—and the trade-offs are then even more critical. Ideally, the VVUQ framework helps to inform decisions on the relative impact of these activities and can be used to prioritize the allocation of resources.

Regardless of how carefully and efficiently the activities are carried out, difficult decisions will have to be made during the course of VVUQ. These decisions will have to withstand subsequent scrutiny and review by independent third parties.

Adequate documentation and transparency about the VVUQ process will facilitate peer review and provide archival information for future studies. It is important that peer reviewers be given access to all relevant information, data, and computational models (including codes, where appropriate) used in the VVUQ process.

Finding: It is important to include in any presentation of VVUQ results the assumptions as well as the sources of uncertainty that were considered. Appropriate documentation and transparency about the process and body of knowledge that were used to assess and quantify uncertainties in the relevant quantities of interest are also crucial for a complete understanding of the results of the VVUQ analysis.


Ultimately, decision makers are faced with a set of choices, each one of which will have certain advantages and disadvantages. Within this framework, decision makers must make trade-offs based on the analyses and the probabilities of the various scenarios. For example, someone in environmental management may have to choose between two remediation strategies for cleaning up a contaminated site. The decision maker could choose an option for monitored natural attenuation—in other words, leaving the site as is but closely monitoring it to make sure that the contamination does not spread to high-risk areas. Or the decision maker could choose a more active, but also more costly, procedure that might clean up the site. The choice of option will be based on several underlying computational models, each with its own set of uncertainties that need to be compared against one another.

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement