Societal Experts Action Network


SEAN is a network of experts in the social, behavioral, and economic sciences poised to assist decision makers at all levels as they respond to COVID-19.


What is SEAN?


The Societal Experts Action Network (SEAN) links decision makers with social, behavioral, and economic science researchers who can provide evidence-based expert guidance that supports local, state, and federal policies and responses related to COVID-19. The network, an activity of the National Academies of Sciences, Engineering, and Medicine that is sponsored by the National Science Foundation, responds to the most pressing questions and provides rapid, actionable responses. To learn more about SEAN, visit nationalacademies.org/SEAN

SEAN’s first rapid expert consultation provides leaders with insight into the strengths and weaknesses of the COVID-19 data they have available to make decisions about their communities. By understanding these characteristics, decision makers can work with the data type best-suited to the question at hand, and use the data available to inform effective decision making. Download the full rapid expert consultation now.

 

Criteria for Assessing the Reliability and Validity of Data Types


Decision makers must use the data that are available while understanding their limitations. There are five criteria against which the reliability and validity of these data types can be assessed:

  • Does the reporting population represent the population of interest?
  • Does each person in the population have an equal chance of being measured?
  • Are there systematic factors that could cause the values reported to be overestimates or underestimates of the actual values?
  • Is there a difference between what is reported and what one wants to measure?
  • Is there uncertainty due to small sample sizes; that is, do small sample sizes cause unstable numbers?
  • Have people been measured twice?
  • Do tests produce accurate results?
  • What is the time lag in reporting the numbers?
  • Are the numbers consistently updated, or are there time gaps in delivery of the data?
  • Do time lags differ across sources?
  • Has the nature of measurement changed over time in a way that impacts reported estimates?
  • Are events recorded on the day they occurred or the day they were reported?
  • Do the numbers cover all geographic areas of interest?
  • Are areas of particular interest covered?
  • Do all areas use the same measurement and classification system?
  • Do the indicators count persons outside the given jurisdiction?

Assessment of Data Types by Criteria for Reliability and Validity

Number of Confirmed Cases


Key Implication for Decision Making: This measure is readily available, but is likely to be a substantial underestimate of the prevalence of the disease in a population given that most people with COVID-19 are asymptomatic, and even among those who are symptomatic, not all are tested. As the volume of testing expands to include populations with less severe symptoms and asymptomatic individuals, this measure will be increasingly useful for determining the prevalence of COVID-19.

 

Representativeness

Bias

Uncertainty, Measurement & Sampling Error

Time

Space

Number of Confirmed Cases
READ MORE

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

Data source usually meets this criterion

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

table-key

Assessment of Data Types by Criteria for Reliability and Validity

Hospitalizations


Key Implication for Decision Making: Data on hospitalizations are typically available quickly at the local level, although the completeness of reporting may vary from day to day. These data reflect only the most severe cases of infection, but changes in the number of hospitalizations likely reflect similar changes in the total number of infections within a community. Note that patients requiring hospitalization were exposed several weeks previously.

 

Representativeness

Bias

Uncertainty, Measurement & Sampling Error

Time

Space

Hospitalizations
READ MORE

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

Data source usually meets this criterion

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

table-key

Assessment of Data Types by Criteria for Reliability and Validity

Emergency Department Visits


Key Implication for Decision Making: In some jurisdictions, emergency department (ED) visits are available at the local level in close to real time. The reason for the visit can be reported either as a syndrome (e.g., “influenza-like illness”) or as a specific diagnosis (e.g., “COVID-19”). These data are most useful in the early stages of an outbreak or to assess resurgence, though it should be noted that patients with symptoms were exposed up to 2 weeks earlier.

 

Representativeness

Bias

Uncertainty, Measurement & Sampling Error

Time

Space

Emergency Department Visits
READ MORE

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

Data source usually meets this criterion

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

table-key

Assessment of Data Types by Criteria for Reliability and Validity

Reported Deaths


Key Implication for Decision Making: Reported COVID-19 deaths are affected by the accuracy of cause-of-death determinations and reflect the state of the outbreak several weeks previously because of the long course of COVID-19 infection. Sometimes lags in reporting of data also occur.

 

Representativeness

Bias

Uncertainty, Measurement & Sampling Error

Time

Space

Reported Deaths
READ MORE

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

table-key

Assessment of Data Types by Criteria for Reliability and Validity

Excess Deaths


Key Implication for Decision Making: Compared with the other data reviewed here, excess deaths are the best indicator of the mortality impacts of the pandemic. However, because of the possibility of death misclassification, these data represent a mix of confirmed COVID-19 deaths and deaths from other causes.

 

Representativeness

Bias

Uncertainty, Measurement & Sampling Error

Time

Space

Excess Deaths
READ MORE

Data source usually meets this criterion

data source may or may not meet criterion, and questions related to that criterion should be asked

Data source usually meets this criterion

Data source usually meets this criterion

Data source usually meets this criterion

table-key

Assessment of Data Types by Criteria for Reliability and Validity

Fraction of Viral Tests that are Positive


Key Implication for Decision Making: These data may not be an adequate measure of prevalence, depending on testing criteria. If mainly symptomatic people are tested, this figure is expected to overestimate the true community prevalence. The proportion is expected to decline as testing expands to include mildly symptomatic and asymptomatic people.

 

Representativeness

Bias

Uncertainty, Measurement & Sampling Error

Time

Space

FRACTION OF VIRAL TESTS THAT ARE POSITIVE
READ MORE

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

data source may or may not meet criterion, and questions related to that criterion should be asked

table-key

Assessment of Data Types by Criteria for Reliability and Validity

Representative Prevalence Surveys


Key Implication for Decision Making: Representative prevalence surveys are the best strategy for understanding the prevalence of a disease in any given population at a specific point in time. Such surveys can be undertaken for specific populations (e.g., workplace, nursing home, jails and prisons). Although they require undertaking a special study rather than using routinely collected data, many public health agencies have this capacity. There will be some time lag involved, however, in mounting and interpreting such a survey. While prevalence surveys in general, such as surveys of healthcare workers or convenience samples of grocery shoppers, may be useful if replicated over time to measure trends, they are not necessarily representative.

 

Representativeness

Bias

Uncertainty, Measurement & Sampling Error

Time

Space

Representative Prevalence Surveys
READ MORE

Data source usually meets this criterion

Data source usually meets this criterion

Data source usually meets this criterion

data source may or may not meet criterion, and questions related to that criterion should be asked

Data source usually meets this criterion

table-key

How can SEAN help?


Are you a policy maker? Do you have a question you need answered? SEAN will consider the most pressing questions and engage the nation’s experts to focus on your challenges. Contact us at SEAN@nas.edu or 202-334-3440.

SEAN is a network of experts in the social, behavioral, and economic sciences poised to assist decision makers at all levels as they respond to COVID-19. The network appreciates any and all feedback on its work. Please send comments to SEAN@nas.edu