Assessing Morbidity and Mortality Associated with the COVID-19 Pandemic: A Case Study Illustrating the Need for the Recommendations in This Report
Authored by M. A. Stoto and M. K. Wynia, July 22, 2020
INTRODUCTION AND SUMMARY
In a crisis, the public wants to know what is happening and policy makers want good data for decisions. The coronavirus disease 2019 (COVID-19) pandemic has generated a sea of numbers: cumulative totals and daily numbers of new cases, individuals hospitalized and recovered, deaths and death rates, numbers tested, testing capacity, and undiagnosed and asymptomatic cases. Policy makers seek to use these and other data to know whether we have “flattened the curve,” to assess which non-pharmaceutical interventions—social distancing, mask wearing, restrictions on gatherings, and so on—are most effective, and to serve as metrics for relaxing or tightening restrictions. The pandemic has dramatically illustrated the high level of public interest in the total numbers of reported cases and deaths, though it is widely acknowledged by experts (and for reasons described in this report) that these early estimates typically underestimate the full effects of the pandemic.
The pandemic has also illustrated how numbers of deaths and illnesses associated with a disaster can inform critical decisions with public health, economic, and political implications, which can make the accuracy of these estimates a source of vigorous debate. In this regard, the pandemic has shone a spotlight on the need for accurate information to both guide disaster response and improve preparedness for the future. Yet, the plethora
of COVID-19 numbers also reminds us that the consequences of any major disaster are far more complex than can be represented by a single number, the “death toll.” Moreover, it shows that even the “death toll” cannot be fully captured in a single number except at a single point in time.
As described in the body of the committee’s report, there are two basic approaches to assessing morbidity and mortality from disasters. Both approaches produce valuable information about disaster-related mortality and morbidity, though each has different strengths, weaknesses, and appropriate uses. One is to try to count casualties individually (whether deaths, injuries, or cases of illness) and determine whether each is directly or indirectly caused by, or associated with, the disaster. The second approach is to estimate the number of casualties using statistical means, either through sampling methods or by comparing observed deaths with deaths observed in the previous year or a comparison population.
The first approach (case counting) typically generates an underestimate of total morbidity and mortality, and the COVID-19 pandemic has shown that these numbers may be skewed even further downward when testing for the presence or absence of the condition in question is unavailable or unreliable. Yet, the counting approach is essential for contact tracing and for getting relief to specific individuals affected by the disaster. Case counts are also often the first set of numbers available to decision makers.
Meanwhile, the population estimation approach typically takes more time, and suffers from the additional weakness of not being useful for determining which specific individuals might have been affected and which would have died or become ill even in the absence of the disaster. However, population estimation methods provide a much more complete picture of the entire population affected by the disaster and are preferred (when available) for making policy decisions where this more comprehensive view is important, such as about reopening strategies and targeting aid to areas and populations most affected.
To reiterate: both approaches are scientifically legitimate, though they differ in their assumptions and data requirements, and as suggested in Recommendation 2-2, both should be used and reported. More important, the availability of alternative approaches does not mean that “anything goes”—the COVID-19 pandemic has shown the value of carefully considering each method’s unique strengths, weaknesses, and appropriate uses (see Table C-1). Although we note below where some methods might be more appropriate for specific uses than others, we do not regard any given method as always better than the others; they are different.
TABLE C-1 Strengths, Weaknesses, and Intended Use of Different Methods for Assessing COVID-19 Mortality and Morbidity
|Assessment Method||Strengths||Weaknesses||Most Useful for|
|Case counts, by public health and emergency operations||
|Vital statistics, by death certificates||
|Excess morbidity and mortality estimates||
We present this analysis as a case study that illustrates the issues raised in the committee’s report1 and provides real-time context and support for the its recommendations. This analysis draws from a rapid expert consultation from the National Academies of Sciences, Engineering, and Medicine, which describes the strengths and weaknesses of different types of COVID-19 data.2 The COVID-19 situation is changing rapidly, so we note that this analysis reflects the situation as of July 22, 2020.
Counting Individual COVID-19 Cases and Deaths
The report notes that during and shortly after a natural disaster the focus is typically on trying to count the individuals injured or killed. This counting requires (1) a process to determine whether a particular death is directly or indirectly caused by the disaster and (2) a mechanism to report these deaths to a central office so they can be tabulated and analyzed. Individual deaths are classified as directly or indirectly caused by the disaster by physicians, medical examiners, or coroners. Some of these decisions are obvious. For instance, people who drown in a flood or are hit by flying debris in a tornado should be reported as “direct” deaths, while a person killed while clearing trees after a tornado should be counted as an indirect death. Yet, current methods for individual counting generally would not count as an indirect death someone who suffered a fatal heart attack triggered by the stress of the disaster. And some cases are even more challenging, such as the person who survived a wildfire but then returned to his burned-out home and committed suicide.3
With regard to reporting, as in normal times, deaths during disasters are reported to state health departments through vital registration systems, which can take days or even weeks. For this reason, deaths and injuries in disasters are also often reported to and rapidly tabulated by public health emergency operations centers. This becomes especially important in infectious disease outbreaks.
1 The Statement of Task directed the committee to focus on non-infectious disease–related disasters as defined in the Robert T. Stafford Disaster Relief and Emergency Assistance Act (Stafford Act). However, the triggering of a Stafford Act Declaration in March 2020 in response to the COVID-19 pandemic brought the consideration of this disaster within the committee’s scope. The study sponsor, the Federal Emergency Management Agency, gave its approval for the committee’s consideration of COVID-19 in this report.
2 National Academies of Sciences, Engineering, and Medicine. 2020. Evaluating data types: A guide for decision makers using data to understand the extent and spread of COVID-19. Washington, DC: The National Academies Press. https://doi.org/10.17226/25826.
3 See https://sanfrancisco.cbslocal.com/2017/11/14/man-kills-himself-in-ruins-of-his-burned-out-santa-rosa-home (accessed September 1, 2020).
Using Public Health Case Counts
There are several differences between how morbidity and mortality are classified and reported during a pandemic versus a natural disaster such as a fire, blizzard, or hurricane. In many natural disasters the focus is on immediate deaths, while in a pandemic, public health epidemiologists often focus on live cases, since they pose an infectious risk to others. Still, the COVID-19 pandemic has shown the intensity of public and policy-maker interest in the numbers of deaths based on counts, even as experts acknowledge that early case counts typically provide underestimates (see below). For pandemics, individuals are included in the epidemiological “case count” if they meet an established case definition, which typically includes characteristic symptoms caused by the pathogen and a laboratory test confirming infection, if one exists. These definitions can evolve as more is learned, and they often include options for naming someone as a “probable,” “presumptive,” or “confirmed” case, which can be critical for carrying out effective contact tracing. Importantly, contact tracing is a primary purpose of public health surveillance: clinicians are required to report cases to local or state health departments, which use this information to trace case contacts and help stem the outbreak. With regard to mortality estimates, however, the total number of deaths from the outbreak derived from case counts is based on tracking the survival of only those cases known to the health department, and one must acknowledge that errors in diagnosis and reporting can arise at every step of this process.
With regard to morbidity estimates, the COVID-19 pandemic illustrates several problems related to diagnosis. For instance, with no preexisting diagnostic test for the virus, early COVID-19 case definitions in Wuhan, China, were based on symptoms and a characteristic pattern in a computed tomography (CT) scan.4 Setting aside any questions about intentional misreporting, respiratory symptoms are common in the winter, some patients do not present for care, and CT scans are expensive and not always available in outpatient settings, so it is entirely predictable that many cases in Wuhan were not included in early official case counts. Similarly, at the peak of the outbreak in New York City, tests were not available in sufficient numbers to test everyone with symptoms, and many individuals with symptoms of COVID-19 were regarded as “presumptive cases.” Adding these presumptive cases into the official case count was appropriate, though it resulted in what seemed like a sudden jump in COVID-19 cases
4 Wu, P., X. Hao, E. H. Y. Lau, J. Y. Wong, K. S. M. Leung, J. T. Wu, B. J. Cowling, and G. M. Leung. 2020. Real-time tentative assessment of the epidemiological characteristics of novel coronavirus infections in Wuhan, China, as at 22 January 2020. Euro Surveillance 25(3):2000044.
and deaths in New York.5 Thus, the pandemic provides several examples of uncertainties that can arise in case counting due to diagnostic uncertainty.
The COVID-19 pandemic also illustrates some common challenges related to accurate and timely reporting of known cases. While physicians have a responsibility to report COVID cases, and health departments publish guidance about what should be reported and how, the changing nature of this guidance has made it difficult for busy practitioners to know what to do. To mitigate the risk of physician under-reporting, laboratories are also required to report positive test results, so that health departments can reach out to their physicians to gather additional data. Still, for COVID and for other reportable conditions, cases can be missed. Although some testing locations no longer require a referral, in many cases the patient to be tested must first seek health care, then the physician must decide to order a diagnostic test, the test must be available, and in many situations the patient must then go somewhere else to obtain the test. Further fueling the inaccuracy of national case counts for COVID-19 have been that testing processes, and even case definitions, have varied substantially from state to state.6
For many reportable infections, provider reporting is far from complete7 and sudden increases in the proportion of cases reported, as can arise during a local outbreak, can distort surveillance statistics.8 For many infectious diseases, such as influenza, physicians often make treatment decisions “empirically,” based on symptoms alone, so test samples are not collected or sent to a lab. As a result, these cases are not included in initial public health case counts. Because COVID-19 can also be treated empirically, and tests have been scarce and some health systems overwhelmed, relatively healthy people with compatible symptoms have until recently been encouraged to stay home without testing.9 As a result, the number of
5 Goodman, J. D., and W. K. Rashbaum. 2020. N.Y.C. death toll soars past 10,000 in revised virus count. The New York Times, April 14, 2020. www.nytimes.com/2020/04/14/nyregion/new-york-coronavirus-deaths.html?smid=em-share (accessed September 1, 2020).
6 Brown, E., and B. Reinhard. 2020. Which deaths count toward the COVID-19 death toll? It depends on the state. The Washington Post, April 16. www.washingtonpost.com/investigations/which-deaths-count-toward-the-covid-19-death-toll-it-depends-on-the-state/2020/04/16/bca84ae0-7991-11ea-a130-df573469f094_story.html (accessed September 1, 2020).
7 Fill, M. A., R. Murphree, and A. C. Pettit. 2017. Health care provider knowledge and attitudes regarding reporting diseases and events to public health authorities in Tennessee. Journal of Public Health Management and Practice 23(6):581–588.
8 Piltch-Loeb, R., J. Kraemer, K. W. Lin, and M. A. Stoto. 2018. Public health surveillance for Zika virus: Data interpretation and report validity. American Journal of Public Health 108(10):1358–1362.
9 Begley, S. 2020. States have a long way to go on testing, review shows. The Boston Globe, April 28. www.bostonglobe.com/2020/04/27/nation/states-have-long-way-go-testing-review-show (accessed September 1, 2020).
officially recorded COVID-19 cases in the United States almost certainly underestimates the true number of infections, perhaps dramatically.
Epidemiologists refer to the fact that only a proportion of infected individuals seek care, are diagnosed, and are reported as the “iceberg effect.” This has occurred with COVID-19 and it is a natural phenomenon; it should not be regarded or portrayed as an attempt to hide the full extent of the pandemic. For example, Holtgrave et al. demonstrated this effect during the height of the COVID-19 outbreak in New York State, and they also found that the proportion of those tested and diagnosed varied widely by race and ethnicity. They estimated that only 6.5 percent of infected Hispanic adults were diagnosed compared to 11.7 percent and 10.1 percent of non-Hispanic Whites and Blacks, respectively. Hispanics and Blacks who were infected, on the other hand, were more than twice as likely to be hospitalized compared to Whites.10 This study from the COVID-19 pandemic illustrates another reason for cautious interpretation of case count data; certain population subgroups may be disproportionately represented (or unrepresented) in counting mechanisms.
The pandemic also illustrates how changes in case counting methods over time can affect reported numbers. As testing capacity grew in April, May, and June, so did the number of positive results, possibly “catching up” with actual cases and perhaps not reflecting a true rising incidence of new infections. Similarly, reducing the amount of testing would be expected to reduce the reported case count, which led public health officials to adjust recommendations for reopening according to testing numbers. For instance, one of the criteria for relaxing social distancing in the White House plan was a “downward trajectory of positive tests as a percent of total tests within a 14-day period,” but the volume of tests must be flat or increasing11 to avoid this bias. During the surge in cases in the South and West in June and July, however, shortages of testing resources12 and delays in obtaining results13 emerged, perhaps re-creating undercounts of cases. Notably, a state-by-state analysis by Stat News showed that in 26 of the 33 states that saw cases increase between mid-May and mid-July, the case
10 Holtgrave, D. R., M. A. Barranco, J. M. Tesoriero, D. S. Blog, and E. S. Rosenberg. 2020. Assessing racial and ethnic disparities using a COVID-19 outcomes continuum for New York State. Annals of Epidemiology 48:9–14. https://doi.org/10.1016/j.annepidem.2020.06.010.
12 Mervosh, S., and M. Fernandez. 2020. Months into virus crisis, U.S. cities still lack testing capacity. The New York Times, July 7. www.nytimes.com/2020/07/06/us/coronavirus-testshortage.html?smid=em-share (accessed September 1, 2020).
13 Wu, K. J. 2020. Testing backlogs may cloud the true spread of the coronavirus. The New York Times, July 19. www.nytimes.com/2020/07/19/health/coronavirus-testing-viral-spread.html?smid=em-share (accessed September 1, 2020).
count rose because there was actually more disease, not because there was more testing.14
Finally, there have been multiple reports of U.S. patients diagnosed retrospectively with COVID-19 who had not been previously included in official case counts. Some individuals were thought to have the infection but tests were not available;15 in others COVID-19 was not initially suspected.16 Such reports are sometimes characterized as surprising, or even to suggest that cases had been intentionally hidden, but they are an expected occurrence in disease outbreaks, especially with a novel pathogen.
The differences in the way COVID-19 cases are classified and reported—among jurisdictions and over time—described in this section create challenges for local, state, and national public health experts monitoring the pandemic. Developing a set of national standards as called for in Recommendation 3-3 would help policy makers at all levels choose appropriate control strategies.
Using Vital Statistics Data to Count COVID-19 Mortality
COVID-19 has also illustrated both opportunities and challenges associated with using vital statistics data to count mortality from the pandemic. Because COVID-19 cases are likely to be undercounted by public health, so are COVID-19 associated deaths based on case counts.17 An alternative source of individual-level mortality data is vital statistics, which uses different definitions and processes than public health case counting and is essentially complete (i.e., nearly every person who dies in the United States is accounted for on a death certificate). In early April 2020, the National Center for Health Statistics (NCHS) issued guidance18 indicating that if COVID-19 played a role in a death, this condition should be specified on the death certificate
14 Begley, S. 2020. Trump said more COVID-19 testing “creates more cases.” We did the math. STAT News, July 20. https://www.statnews.com/2020/07/20/trump-said-more-covid19-testing-creates-more-cases-we-did-the-math (accessed September 1, 2020).
15 Duhigg, C. 2020. Seattle’s leaders let scientists take the lead. New York’s did not. The New Yorker, May 4. www.newyorker.com/magazine/2020/05/04/seattles-leaders-let-scientists-take-the-lead-new-yorks-did-not (accessed October 1, 2020).
16 Fuller, T., M. Baker, S. Hubler, and S. Fink. 2020. A coronavirus death in early February was “probably the tip of an iceberg.” The New York Times, April 22. www.nytimes.com/2020/04/22/us/santa-clara-county-coronavirus-death.html?smid=em-share (accessed September 1, 2020).
17 On the other hand, serious cases are more likely to seek health care and be tested, so the degree of undercounting is probably less. Thus, the Case Fatality Rate (the proportion of cases with a condition who die) calculated from these data is likely to be an overestimate, and the same is true for the proportion of infected who suffer severe symptoms and need to be hospitalized.
18 NVSS (National Vital Statistics System). 2020. Guidance for certifying deaths due to coronavirus disease 2019 (COVID-19). Vital Statistics Reporting Guidance. https://www.cdc.gov/nchs/data/nvss/vsrg/vsrg03-508.pdf (accessed September 1, 2020).
either as the underlying cause of death where warranted or as “probable” or “presumed” if the circumstances were compelling within a reasonable degree of certainty, even if testing had not been done (as it often was not, due to lack of testing capacity). Consequently, vital statistics data, which are compiled from death certificate data, will include some deaths not in the public health case counts. But some COVID-19 deaths will be missed in both public health case counts and on death certificates, and other deaths might be inaccurately attributed to COVID-19 on a death certificate.
For instance, in April 2020, vital statistics reports indicated a large increase in individuals dying at home, rather than in the hospital,19 especially in New York City.20 One might infer that many of these in-home deaths were caused, directly or indirectly, by COVID-19, but most were never tested or reported and hence are not included in health department case counts. Some of these deaths might eventually appear in vital statistics reports, if physicians follow the NCHS guidance, but because death certificate data are not used for contact tracing, most of these cases will never appear in public health case counts.
The COVID-19 pandemic provides multiple examples of the political tensions that can arise around these numbers. For instance, there have been claims offered without evidence that doctors are being “coached” to mark COVID-19 as the cause of death on death certificates even when it is not, to inflate the pandemic’s death toll for political purposes.21 In response, some states elected to not include deaths without a mention of COVID-19 on the death certificate in their official counts, even if the person had tested positive and was included in the public health surveillance database. In Colorado, this corresponded to a 24 percent reduction in deaths from COVID-19, as only 878 of 1,150 deaths (as of May 15, 2020) had COVID-19 specifically noted on the death certificate.22 Moreover, NCHS added an extra step in which humans instead of computers must verify the information on the
19 Gillum, J., L. Song, and J. Kao. 2020. There’s been a spike in people dying at home in several cities. That suggests coronavirus deaths are higher than reported. ProPublica. www.propublica.org/article/theres-been-a-spike-in-people-dying-at-home-in-several-cities-thatsuggests-coronavirus-deaths-are-higher-than-reported (accessed September 1, 2020).
20 Hogan, G. 2020. Staggering surge of NYers dying in their homes suggests city is undercounting coronavirus fatalities. Gothamist, April 7. www.gothamist.com/news/surge-number-new-yorkers-dying-home-officials-suspect-undercount-covid-19-related-deaths (accessed September 1, 2020).
21 Rosenberg, M., and J. Rutenberg. 2020. Fight over virus’s death toll opens grim new front in election battle. The New York Times, May 9. www.nytimes.com/2020/05/09/us/politics/coronavirus-death-toll-presidential-campaign.html?referringSource=articleShare (accessed September 1, 2020).
22 Ingold, J., and J. Paul. 2020. Nearly a quarter of the people Colorado said died from coronavirus don’t have COVID-19 on their death certificate. The Colorado Sun, May 15. www.coloradosun.com/2020/05/15/colorado-coronavirus-death-certificate/?mc_cid=7ed16a0b8b (accessed September 1, 2020).
death certificate before it is added to the tally, adding around 1 week of delay in formally recording COVID-19 deaths.23
COVID-19 also shows how mortality data based on counting methods, while valuable for some purposes, can also create inaccurate impressions of overall pandemic control efforts.24 For instance, as the number of new cases started to increase throughout the South and West of the United States in June and July 2020, some took comfort that mortality remained low in those states. But deaths typically occur 2–3 weeks or longer after an individual is infected, and there is further delay until they are reported, so the increase in mortality appeared in July.25 Thus, COVID-19 shows how deaths can be a “lagging indicator.” In addition, recent COVID-19 trends demonstrate two other ways in which deaths can provide an incomplete picture. First, because COVID-19 cases tended to be younger in June and July than they had been in April and May,26 the early case fatality rate in this “second wave” in the United States was lower. At the time of this writing however, the virus was spreading to older and more vulnerable individuals, and the case fatality rate will likely go up. Second, recent case fatality rates have probably also been lower because the medical care provided to infected individuals has improved, though as hospitals in the South and West become as overwhelmed with cases as those in the New York City area were in April and May, that effect too may diminish.27,28
As with case counts, differences in the way COVID-19 deaths are recorded and tabulated described in this section create challenges for monitoring the pandemic. Standardizing mortality data and reporting (Recommendation 3-2) and strengthening systems to improve the quality of these
23 Arnold, C. 2020. How scientists know COVID-19 is way deadlier than the flu. National Geographic, July 3. www.nationalgeographic.com/science/2020/07/coronavirus-deadlier-than-many-believed-infection-fatality-rate-cvd (accessed September 1, 2020).
24 Thompson, D. 2020. COVID-19 cases are rising, so why are deaths flatlining? The Atlantic, July 10. www.theatlantic.com/ideas/archive/2020/07/why-covid-death-rate-down/613945 (accessed September 1, 2020).
25 Stockman, F., M. Smith, and G. McDonnell Nieto del Rio. 2020. Daily virus death toll rises in some states. The New York Times, July 10. www.nytimes.com/2020/07/10/us/daily-virus-death-toll-rises-in-some-states.html?referringSource=articleShare (accessed September 1, 2020).
26 Bosman, J., and S. Mervosh. 2020. As virus surges, younger people account for “disturbing” number of cases. The New York Times, June 26. www.nytimes.com/2020/06/25/us/coronavirus-cases-young-people.html?action=click (accessed September 1, 2020).
27 Bump, P. 2020. The White House’s favorite new coronavirus metric—mortality rate—probably won’t be a favorite for long. The Washington Post, July 13. www.washingtonpost.com/politics/2020/07/13/white-houses-favorite-new-coronavirus-metric-mortality-rate-probablywont-be-favorite-long/?arc404=true (accessed September 1, 2020).
data (Recommendation 3-1) would provide policy makers at all levels with the ability to monitor the pandemic and make better decisions about controlling it.
In sum, experiences with COVID-19 to date have provided a number of valuable examples of ways in which individual-level counting methods can underestimate total mortality and morbidity in disasters. Experiences with the pandemic also demonstrate that even though case counts and death counts can be (and typically are) presented as precise numbers down to the single case—and even though individual counting methods are critical when it comes to certain tasks such as contact tracing and assigning death benefits to individuals—these statistics are in fact estimates of the true total mortality or morbidity. To illustrate this, consider that even the two primary sources of data for individual counts of deaths—public health case counts and vital statistics—can be expected to generate different totals.29 The COVID-19 pandemic further demonstrates that when case counts are the product of evolving case definitions, testing procedures, and reporting processes, the estimates generated through case counting can change (and hopefully improve) over time, but they remain estimates; typically, they will produce underestimates of the total disaster impact.
Population Estimation Approaches
The COVID-19 pandemic has also provided valuable illustrations of how a complementary set of methods can provide useful data and better assessments of the total impact of the pandemic on morbidity and mortality. These methods include (a) surveys using representative or complex sampling of affected populations (i.e., surveillance data) and (b) estimates derived by comparing observed deaths or illness to expected numbers based on prior years or comparison populations. Because these methods include a broader range of associated illness and death, they often produce numbers larger than those determined using individual case counts.
While statistical estimation might sound less precise than case counting, recall that case count methods provide imprecise results that are also estimates of the true effect of a disaster. Statistical estimation methods also have both strengths and weaknesses. While they can provide a more complete picture of the total impact of the disaster on health outcomes, they are not useful for determining whether any given dead or ill person became dead or ill as a direct or indirect result of the disaster. As an extreme example of these trade-offs, following Hurricane Maria a group of analysts based
29 Reinhard, B., and E. Brown. 2020. Which deaths count toward the COVID-19 death toll? It depends on the state. The Washington Post, April 16. www.washingtonpost.com/investigations/which-deaths-count-toward-the-covid-19-death-toll-it-depends-on-the-state/2020/04/16/bca84ae0-7991-11ea-a130-df573469f094_story.html (accessed September 1, 2020).
at Harvard University used survey sampling methods to estimate 4,645 (95% confidence interval [CI] 793–8,498) excess deaths among Puerto Ricans.30 Another group based at The George Washington University used excess mortality methods to estimate 2,975 (95% CI 2,658–3,290) excess deaths for the period of September 2017 through February 2018.31 These numbers are now widely agreed to be much more accurate estimations of the total impact of the hurricane than the case count method, which suggested only 64 people died as a result of the hurricane. While Hurricane Maria represents a particularly egregious case of inadequacy of counting methods, it is illustrative of the dramatic undercounting that is possible following disasters. Experiences to date with using statistical estimation methods to assess COVID-19 morbidity and mortality have shown smaller, but still very significant estimation differences between case counting and statistical estimation methods, as summarized below.
Using Survey Sampling Methods to Assess Total COVID-19 Morbidity and Mortality
Efforts are under way to use survey methods to assess total morbidity and mortality of COVID-19. These methods use information collected from samples of individuals, extrapolating this information to a population to estimate the total impact of COVID-19. These methods are being used to inform a number of important policy questions. For instance, random sample survey methods can determine whether a COVID-19 outbreak in a city or a state is getting better or worse, and how fast, more accurately than counting only positive test results from those presenting for care. Specifically, sero-prevalence surveys from randomly chosen individuals in the population can determine the real percentage of people in a community recently infected with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), assuming an accurate serologic assay is available.32,33 For example, Rosenberg et al. analyzed a statewide convenience sample of New
30 Kishore, N. D., D. Marqués, A. Mahmud, M. V. Kiang, I. Rodriguez, A. Fuller, P. Ebner, C. Sorensen, F. Racy, J. Lemery, L. Maas, J. Leaning, R. A. Irizarry, S. Balsari, and C. O. Buckee. 2018. Mortality in Puerto Rico after Hurricane María. New England Journal of Medicine 379(2):162–170.
31 Milken Institute School of Public Health. 2018. Ascertainment of the estimated excess mortality from Hurricane María in Puerto Rico. Washington, DC: Milken Institute School of Public Health, The George Washington University.
32 Mostashari, F., and E. J. Emanuel. 2020. We need smart coronavirus testing, not just more testing. STAT, March 24. www.statnews.com/2020/03/24/we-need-smart-coronavirus-testing-not-just-more-testing (accessed September 1, 2020).
33 Lipsitch, M. 2020. Opinion. “Serology” is the new coronavirus buzzword. Here’s why it matters. The Washington Post, May 4. www.washingtonpost.com/opinions/2020/05/04/serology-is-new-coronavirus-buzzword-heres-why-it-matters (accessed September 1, 2020).
York grocery store customers and estimated that the cumulative incidence of COVID-19 through March 29, 2020, was 14 percent. This rate varied substantially by geographic area (reaching 24 percent in New York City) as well as race and ethnicity. They also estimated that only 8.9 percent of individuals infected during this period were diagnosed, and that this fraction varied from 6.1 percent of individuals 18–34 years of age to 11.3 percent of those 55 years of age or older.34 Sero-prevalence studies have been conducted in California35,36 and other countries (e.g., Geneva, Switzerland37). The World Health Organization is coordinating sero-prevalence studies in at least six countries.38
Experience with COVID-19 also shows how data from survey methods can inform clinical decision making. For example, “sentinel testing” on samples of individuals at high risk of infection, such as health care workers or contacts of known cases,39 have helped improve understanding of viral transmission risk and risk factors for more severe disease.40
COVID-19 has demonstrated the value of ongoing surveillance efforts, such as the Centers for Disease Control and Prevention’s (CDC’s) Outpatient Influenza-like Illness Surveillance Network (ILINet), which provides data on visits for influenza-like illness (ILI) (fever and cough and/or sore throat) from
34 Rosenberg, E. S., J. M. Tesoriero, E. M. Rosenthal, R. Chung, M. A. Barranco, L. M. Styer, M. M. Parker, S.-Y.J. Leung, J. E. Morne, D. Greene, R. Holtgrave, D. Hoefer, J. Kumar, T. Udo, B. Hutton, and H. A. Zucker. 2020. Cumulative incidence and diagnosis of SARS-CoV-2 infection in New York. Annals of Epidemiology 48:23–29. doi: 10.1016/j. annepidem.2020.06.004.
35 Bendavid, E., B. Mulaney, N. Neeraj Sood, S. Shah, E. Ling, R. Bromley-Dulfano, C. Lai, Z. Weissberg, R. Saavedra-Walker, J. Tedrow, D. Tversky, A. Bogan, T. Kupiec, D. Eichner, R. Gupta, J. Ioannidis, and J. Bhattacharya. 2020. COVID-19 antibody seroprevalence in Santa Clara County, California. medRxiv preprint. https://doi.org/10.1101/2020.04.14.20062463.
36 Sood, N., P. Simon, P. Ebner, D. Eichner, J. Reynolds, E. Bendavid, and J. Bhattacharya. 2020. Seroprevalence of SARS-CoV-2-specific antibodies among adults in Los Angeles County, California, on April 10–11, 2020. JAMA 323(23):2425–2427. doi: 10.1001/jama.2020.8279.
37 Stringhini, S., A. Wisniak, G. Piumatti, A. S. Azman, S. A. Lauer, H. Baysson, D. De Ridder, D. Petrovic, S. Schrempft, K. Marcus, S. Yerly, I. Arm Vernez, O. Keiser, S. Hurst, K. M. Posfay-Barbe, D. Trono, D. Pittet, L. Gétaz, F. Chappuis, I. Eckerle, N. Vuilleumier, B. Meyer, A. Flahault, L. Kaiser, and I. Guessous. 2020. Seroprevalence of anti-SARS-CoV-2 IgG antibodies in Geneva, Switzerland (SEROCoV-POP): A population-based study. The Lancet 396(10247):313–319. doi: 10.1016/S0140-6736(20)31304-0.
38 Vogel, G. 2020. “These are answers we need.” WHO plans global study to discover true extent of coronavirus infections. Science, April 2. www.sciencemag.org/news/2020/04/these-are-answers-we-need-who-plans-global-study-discover-true-extent-coronavirus (accessed September 1, 2020).
39 Mostashari, F., and E. J. Emanuel. 2020. We need smart coronavirus testing, not just more testing. STAT, March 24. www.statnews.com/2020/03/24/we-need-smart-coronavirus-testing-not-just-more-testing (accessed September 1, 2020).
40 Lipsitch, M., D. L. Swerdlow, and L. Finelli. 2020. Defining the epidemiology of COVID-19—studies needed. New England Journal of Medicine 382:1194–1196.
approximately 2,600 primary care providers, emergency departments, and urgent care centers throughout the United States. Because COVID-19 illness often presents with ILI symptoms, ILINet is being used to track trends and allows for comparison with prior influenza seasons. Also, the National Syndromic Surveillance Program (NSSP), which tracks emergency department visits in 47 states, has been extended to include COVID-19-like illness (fever and cough or shortness of breath or difficulty breathing). Figure C-1 displays the NSSP data through July 11, 2020,41 suggesting a peak number of cases in early April and a re-emergence in June and July.
Several survey studies of COVID-19 have used data from quasi or non-random samples, which generally raise questions of bias and nongeneralizability, but the pandemic has provided examples of how such studies can still provide useful information. For instance, SARS-CoV-2 testing was performed on 214 pregnant women who delivered infants at the NewYork-Presbyterian Allen Hospital and Columbia University Irving Medical Center during the height of New York’s outbreak (between March 22 and April 4, 2020) and 33 tested positive (i.e., a prevalence rate of about 15 percent), but only 4 of these infected women (12 percent) had symptoms of COVID-19, suggesting very high rates of asymptomatic infection among pregnant women.42
COVID-19 also shows how population survey data can be used in combination with case count data to generate insights. For example, CDC is partnering with commercial laboratories to conduct sero-prevalence surveys using de-identified clinical blood specimens from people with blood drawn for reasons unrelated to COVID-19, aiming to test about 1,800 samples from 10 areas around the United States approximately every 3–4 weeks.43 Initial results show the proportion of persons with COVID-19 antibodies ranged from 1.0 percent in the San Francisco Bay area (collected April 23–27) to 6.9 percent of persons in New York City (collected March 23–April 1). When compared to case count estimates, these data suggest that the number of total infections ranged from 6 to 24 times the number of
42 Sutton, D., K. Fuchs, M. D’Alton, and D. Goffman. 2020. Universal screening for SARSCoV-2 in women admitted for delivery. New England Journal of Medicine 382:2163–2164.
43 Centers for Disease Control and Prevention. 2020. Commercial laboratory seroprevalence survey data. https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/commercial-labsurveys.html (accessed September 1, 2020).
41 Centers for Disease Control and Prevention. 2020. COVIDView—A weekly surveillance summary of U.S. COVID-19 activity: Key updates for week 28, ending July 11, 2020. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/pdf/covidview-07-17-2020.pdf (accessed August 3, 2020).
reported cases; for seven sites the total number of infections was estimated to be more than 10 times the number of reported cases.44
COVID-19 has also shown how surveys can be used to measure and track morbidity, such as mental health and other problems, by adding a disaster-specific module to an ongoing survey. For example, the weekly “pulse surveys” fielded by the U.S. Census Bureau were modified in response to the COVID-19 epidemic and are based on a representative sample of more than 1 million households.45 Early results in May 2020, found that 34 percent of respondents showed symptoms of anxiety, depression, or both.46 The results also show that many households have experienced a loss in employment income, are concerned about food security, and have deferred decisions to access health care.47
Despite the promise of these approaches, it must be noted that most of the survey-based methods described in this section were developed quickly to meet emergent needs. Methodological research to improve these approaches (Recommendation 4-1) and efforts to enhance the nation’s capacity to conduct such research (Recommendation 4-2) would enhance the validity of survey results and facilitate their use and utility in future disasters.
Using Excess Mortality and Morbidity Methods in the COVID-19 Pandemic
Calculation of excess mortality and morbidity may provide the most complete, albeit often imprecise, estimates of the total impacts of disasters, including for infectious diseases. This has been illustrated by the experience with COVID-19, and it has been long-recognized for other infectious diseases. For instance, excess mortality is the standard way to determine the overall death toll for influenza each year. Because pneumonia is often
44 Havers, F. P., C. Reed, T. Lim, J. M. Montgomery, J. D. Klena, A. J. Hall, A. M. Fry, D. L. Cannon, C.-F. Chiang, A. Gibbons, I. Krapiunaya, M. Morales-Betoulle, K. Roguski, M. Ata Ur Rasheed, B. Freeman, S. Lester, L. Mills, D. S. Carroll, S. M. Owen, J. A. Johnson, V. Semenova, C. Blackmore, D. Blog, S. J. Chai, A. Dunn, J. Hand, S. Jain, S. Lindquist, R. Lynfield, S. Pritchard, T. Sokol, L. Sosa, G. Turabelidze, S. M. Watkins, J. Wiesman, R. W. Williams, S. Yendell, J. Schiffer, and N. J. Thornburg. 2020. Seroprevalence of antibodies to SARS-CoV-2 in 10 sites in the United States, March 23–May 12, 2020. JAMA Internal Medicine. doi: 10.1001/jamainternmed.2020.4130.
45 U.S. Census Bureau. 2020. Household pulse survey technical definition. https://www.census.gov/programs-surveys/household-pulse-survey/technical-documentation.html (accessed September 1, 2020).
46 Fowers, A., and W. Wan. 2020. A third of Americans now show signs of clinical anxiety or depression, Census Bureau finds amid coronavirus pandemic. The Washington Post, May 26. www.washingtonpost.com/health/2020/05/26/americans-with-depression-anxietypandemic/?arc404=true (accessed September 1, 2020).
47 U.S. Census Bureau. 2020. New household pulse survey shows concern over food security, loss of income. https://www.census.gov/library/stories/2020/05/new-household-pulse-surveyshows-concern-over-food-security-loss-of-income.html (accessed August 3, 2020).
the proximate cause of death for individuals with influenza, and laboratory testing for influenza is often not performed, CDC regularly tracks the number of deaths from either pneumonia or influenza as a proportion of all deaths recorded each week. These data are then compared to typical seasonal patterns and departures above this pattern, as in 2018, which indicate higher total mortality from flu (see Figure C-2). Recently, CDC added COVID-19 deaths to this analysis and found that almost 25 percent of all deaths occurring during the week ending April 11, 2020, were due to pneumonia, influenza, or COVID-19. This is far above the traditional epidemic threshold of 7.0 percent, with sharp weekly increases from the end of February through mid-April.48 CDC also uses statistical modeling
48 Centers for Disease Control and Prevention. 2020. COVIDView—A weekly surveillance summary of U.S. COVID-19 activity. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/covidview/index.html (accessed August 3, 2020).
COVID-19 also illustrates that excess mortality methods are especially useful for assessing total deaths during an outbreak, including both direct and indirect causes. The initial efforts to describe excess mortality for COVID-19 were published in the media. For instance, The Economist found sharp increases in cardiac arrest 911 calls and deaths as well as confirmed COVID-19 deaths in March and early April 2020 in New York City (see Figure C-3).51
Similarly, based on data compiled by NCHS, The New York Times estimated that there had been 23,000 excess deaths in New York City between March 15 and May 2, 2020, leading to a total number of deaths that was more than three times the normal amount.52 A more comprehensive analysis published by The New York Times estimated that by May 13, more than 100,000 Americans had already died as a result of the pandemic, rather than the 83,000 whose deaths had been directly attributed to the disease by that date.53The Economist maintains a comparison of excess deaths across countries.54
In May 2020 the New York City Department of Health and Mental Hygiene published a preliminary estimate of excess mortality in New York City from March 11 through May 2. As illustrated in Figure C-4, they estimated that out of a total of 32,107 reported deaths, 24,172 were in excess of the seasonal expected baseline. Of the excess, 13,831 (57 percent) were laboratory-confirmed COVID-19-associated deaths and 5,048 (21 percent) were probable COVID-19-associated deaths, leaving 5,293 (22 percent)
49 Rolfes, M. A., I. M. Foppa, S. Garg, B. Flannery, L. Brammer, J. A. Singleton, E. Burns, D. Jernigan, S. J. Olsen, J. Bresee, and C. Reed. 2018. Annual estimates of the burden of seasonal influenza in the United States: A tool for strengthening influenza surveillance and preparedness. Influenza and Other Respiratory Viruses 12:132–137.
50 CDC. 2020. COVIDView—A weekly surveillance summary of U.S. COVID-19 activity: Key updates for week 28, ending July 11, 2020. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/pdf/covidview-07-17-2020.pdf (accessed September 1, 2020).
51The Economist. 2020. Deaths from cardiac arrests have surged in New York City. www.economist.com/graphic-detail/2020/04/13/deaths-from-cardiac-arrests-have-surged-in-new-york-city (accessed September 1, 2020).
52 Katz, J., D. Lu, and M. Sanger-Katz. 2020. Tracking the real coronavirus death toll in the United States. The New York Times, May 6. www.nytimes.com/interactive/2020/05/05/us/coronavirus-death-toll-us.html?smid=em-share (accessed September 1, 2020).
53 Kristof, N. 2020. America’s true COVID toll already exceeds 100,000. The New York Times, May 14. www.nytimes.com/2020/05/13/opinion/coronavirus-us-deaths.html?smid=em-share (accessed September 1, 2020).
54The Economist. 2020. Tracking COVID-19 excess deaths across countries. www.economist.com/graphic-detail/2020/07/15/tracking-covid-19-excess-deaths-across-countries (accessed September 1, 2020).
excess deaths that were not identified as either laboratory-confirmed or probable COVID-19-associated deaths.55
In an analysis originally published in The Washington Post,56 Weinberger et al. conducted a similar analysis for the entire United States from March 1 through May 30, 2020. They estimated that there were 122,300 more deaths than would typically be expected at that time of year, 28 percent higher than the official tally of COVID-19-reported deaths during that period based on case counts. The patterns varied substantially across geographical areas; Figure C-5 illustrates the results from New York City and State, the hardest hit areas during this period.57
The COVID-19 pandemic can be used to demonstrate that many of the deaths missed by case counting but captured using excess mortality methods are indirect deaths. For example, Woolf et al. analyzed mortality between March 1 and April 25, 2020, and estimated 87,001 excess deaths nationally, of which 65 percent were attributed directly to COVID-19. But the authors also identified substantial increases in mortality from heart disease, diabetes, and other causes, and few from non-COVID pneumonias or influenza as underlying causes.58
Corroborating evidence of indirect health effects has also been obtained during the COVID-19 pandemic. For example, 29 percent of adults in a recent survey said they have avoided medical care, fearing contracting the coronavirus,59 and there has been a dramatic drop in the number of vaccines provided to children since the national emergency was declared on
55 New York City Department of Health and Mental Hygiene COVID-19 Response Team. 2020. Preliminary estimate of excess mortality during the COVID-19 outbreak—New York City, March 11–May 2, 2020. Morbidity and Mortality Weekly Report 69:603–605. http://dx.doi.org/10.15585/mmwr.mm6919e5.
56 Brown E., A. B. Tran, B. Reinhard, and M. Ulmanu. 2020. U.S. deaths soared in early weeks of pandemic, far exceeding number attributed to COVID-19. The Washington Post. https://www.washingtonpost.com/investigations/2020/04/27/covid-19-death-toll-undercounted/?p9w22b2p=b2p22p9w00098 (accessed September 1, 2020).
57 Weinberger, D. M., J. Chen, T. Cohen, F. W. Crawford, F. Mostashari, D. Olson, V. E. Pitzer, N. G. Reich, M. Russi, L. Simonsen, A. Watkins, and C. Viboud. 2020. Estimation of excess deaths associated with the COVID-19 pandemic in the United States, March to May 2020. JAMA Internal Medicine 180(10):1336–1344. doi: 10.1001/jamainternmed.2020.3391.
58 Woolf, S. H., D. A. Chapman, R. T. Sabo, D. M. Weinberger, and L. Hill. 2020. Excess deaths from COVID-19 and other causes, March–April 2020. JAMA. doi: 10.1001/jama.2020.11787.
59 Kacik, A. 2020. Nearly a third of Americans have put off healthcare during COVID-19. Modern Healthcare. https://www.modernhealthcare.com/patient-care/nearly-third-americans-have-put-healthcare-during-covid-19 (accessed August 3, 2020).
March 13, 2020.60 Researchers at the Well Being Trust and the Robert Graham Center have estimated that COVID-related unemployment, social isolation, and uncertainty could result in as many as 75,000 “deaths of despair” from suicide or alcohol and other substance abuse.61 On the other hand, social distancing might also result in fewer traffic accidents and deaths caused by pollution. All of these indirect effects of the pandemic will be best assessed using population estimations.
As with individual counts, population estimation methods require a number of judgments about definition, statistical methods, data, and other assumptions. For instance, increases in cardiac mortality are common following natural disasters,62 so the increases being documented now are plausibly related to stress caused by the pandemic. Whether these excess deaths should be regarded as “caused by” the pandemic is a matter of definitional dispute, and an illustration of how population estimation methods require judgments to interpret as well as judgments to carry out. The methods research described under Recommendation 4-1 would address these issues and help to ensure the validity and utility of excess mortality estimates in future disasters.
Lessons from Interpreting Data on Mortality and Morbidity in the COVID-19 Pandemic
In a crisis, the public wants to know what is happening and policy makers want good data for decisions. However, the COVID-19 pandemic shows how the availability of different approaches for assessing morbidity and mortality can create confusion. In particular, the availability of different methods generating widely differing estimates creates opportunities for manipulation or the appearance of manipulation. In addition, COVID-19 has shown that the lack of standards for who gathers and analyzes the data, definitions and processes used, and how they are reported can create further confusion and opportunities for intentional or inadvertent selective use of data to support a point of view. The pandemic also illustrates that the precision of the “death toll” based on case counting methods holds enormous
60 Santoli, J. M., M. C. Lindley, M. B. DeSilva, E. O. Kharbanda, M. F. Daley, L. Galloway, J. Gee, M. Glover, B. Herring, Y. Kang, P. Lucas, C. Noblit, J. Tropper, T. Vogt, and E. Weintraub. 2020. Effects of the COVID-19 pandemic on routine pediatric vaccine ordering and administration—United States, 2020. Morbidity and Mortality Weekly Report 69:591–593. http://dx.doi.org/10.15585/mmwr.mm6919e2.
61 Well Being Trust. 2020. The COVID pandemic could lead to 75,000 additional deaths from alcohol and drug misuse and suicide. https://wellbeingtrust.org/areas-of-focus/policyand-advocacy/reports/projected-deaths-of-despair-during-covid-19 (accessed August 3, 2020).
62 Hayman, K. G., D. Sharma, R. D. Wardlow, and S. Singh. 2020. Burden of cardiovascular morbidity and mortality following humanitarian emergencies: A systematic literature review. Prehospital and Disaster Medicine 30(1):80–88. doi: 10.1017/S1049023X14001356.
appeal for policy makers and the public, despite agreement among experts that these are an underestimate of the full impacts of the pandemic.
Case counting and statistical estimation methods have different strengths and weaknesses and generally produce different but complimentary information (see Table C-1), but the COVID-19 pandemic also shows that there are special risks of generating misleading numbers when using these methods in combination. In particular, case count and population estimation methods predictably lead to lower and higher numbers, respectively. So, use of population estimation methods such as serological surveys to assess infection rates will predictably generate higher estimates than case counting methods, while using individual case counts to assess mortality will predictably generate low estimates. Doing so in combination will therefore suggest large numbers of non-fatal infection (i.e., generating an artificially low case fatality rate). Using data obtained through different methods to obtain a falsely low case fatality rate is inappropriate, but such calculations could easily be carried out from naiveté rather than malice. In the end, data obtained from these different methods should be regarded as pieces of a puzzle that, when used in appropriate combinations, can help create a clearer picture of how the disease is spreading and its severity.63 Thus, as suggested in Recommendation 2-2, both counts and estimates should be used and reported.
The major difference between pandemics and other natural disasters is the matter of temporality; hurricanes and wildfires occur over a period of days or weeks (although recovery can take much longer), while the COVID-19 pandemic has already stretched into months and could last years. In all disasters, attributed mortality and morbidity counts and estimates change over time for two reasons: some long-term consequences take time to occur and all data systems have lags, which vary over time. In pandemics, both of these factors apply, but there is an additional dynamic: the continued infection of new cases. Indeed, questions about the evolution of the pandemic itself—where the number of new cases is growing or shrinking in response to control efforts—are critical. For instance, real-time estimates of incidence are used as triggers to decide whether certain non-pharmaceutical measures—such as social distancing or mask wearing—can be relaxed. For this purpose, it is critical to know whether a decrease in the number of new reported cases reflects decreased incidence or simply less testing being done, as noted earlier.
For many disasters, case-based death counts are the focus of attention during the response phase, often because nothing else is available in the short term. While case counts have some lag associated with them,
63 NASEM. 2020. Evaluating data types: A guide for decision makers using data to understand the extent and spread of COVID-19. Washington, DC: The National Academies Press. https://doi.org/10.17226/25826.
survey-based estimates and excess mortality calculations often take much longer (though one could imagine better data infrastructure alleviating some of these delays). But with COVID-19, the time frame is extended, meaning that statistical estimation methods have the potential to provide a more complete and accurate characterization of the COVID-19 pandemic in time to inform policy and practice decisions. The COVID-19 pandemic also reminds us that “death toll” estimates based on case counts are often misleading. To avoid both confusion and manipulation, statistics derived from case counts should be referred to as “reported infections” and “reported deaths” from COVID-19 rather than as “total infections” or the “death toll.” These counts should include suspected and probable cases, though these should also be reported separately from confirmed cases.
Total mortality, or the “death toll” from COVID-19, should only be reported using population estimation approaches, preferably using the same methods as are used for seasonal influenza. These methods produce a more complete picture of the consequences of the pandemic and are preferable for guiding policy decisions, such as about reopening strategies and targeting aid to areas and populations most affected.
Finally, though disease surveillance is primarily a state responsibility, CDC should not only issue standard case definitions, but also recommend common processes for reporting cases and deaths and metrics that state and local health departments report to help ensure that comparisons among states and other population groups are more meaningful. In this spirit, a group of public health experts recently published a list of 15 key metrics with standardized definitions that states and communities can use so that health departments, decision makers, and the public can get a clearer picture of how the response to the pandemic is working in their area.64,65 Standardizing mortality data and reporting (Recommendation 3-2) and strengthening systems to improve the quality of these data (Recommendation 3-1) would address these issues and improve future policy makers’ ability to manage pandemics and other disasters.
64 Arnold, C. 2020. How scientists know COVID-19 is way deadlier than the flu. National Geographic. https://www.nationalgeographic.com/science/2020/07/coronavirus-deadlier-than-many-believed-infection-fatality-rate-cvd (accessed September 1, 2020).
65 Tracking COVID-19 in the United States. 2020. Prevent epidemics. https://preventepidemics.org/covid19/resources/indicators (accessed August 3, 2020).
This page intentionally left blank.