3

Data and Methods for the SHAD II Study

The Shipboard Hazard and Defense (SHAD) II study was conducted to respond to continued concerns that Project SHAD veterans may be experiencing increased long-term health effects or mortality that are associated with their participation in SHAD tests during the 1960s. This chapter describes the design of the SHAD II study and reviews the details of the study population, the measures of exposure and health outcomes, and the approach to the analysis. It also notes some of the challenges in trying to study the experience of these veterans and limitations in the data available for the study and describes important differences between the SHAD I and SHAD II studies. A discussion of the technical details of the analytic methods chosen for the SHAD II study and their application is provided in Appendix D.

NEW APPROACHES IN SHAD II

The SHAD I study (IOM, 2007) constituted a major effort to identify veterans who participated in the SHAD tests, select an appropriate comparison population, define indicators of exposure, and assess a range of health outcomes. The SHAD II study presented an opportunity to build on this previous work. For SHAD II, the committee adopted new approaches in three areas: (1) proposing a set of literature-based a priori hypotheses about health outcomes that might be expected to be associated with exposure to certain substances used in the SHAD tests; (2) attempting to refine the assessment of individuals’ potential exposures to specific substances; and (3) obtaining information about health outcomes for members of the study population from records of health care services received between 1999 and 2011, as reflected in Medicare databases of the Centers for Medicare & Medicaid Services (CMS) and from 1997 through 2011 from the Department of Veterans Affairs (VA) health services.

Hypotheses on Health Outcomes

The SHAD I study commissioned a set of papers that summarized the literature on the physical, chemical, and toxicological properties of the test substances and on the presence or

absence of health effects of exposures of various types (e.g., inhalation, ingestion).¹ These papers provided essential background information for the analyses that compared a broad range of health outcomes among SHAD participants with those among the comparison population.

For SHAD II, the committee also sought to frame hypotheses it could test regarding SHAD exposures and outcomes. To that end, committee members reviewed the commissioned papers as well as reports published between January 2000 and January 2012 that considered the potential health effects of exposure to the substances used in the SHAD tests. For each of the substances, the Institute of Medicine (IOM) research librarian carried out targeted searches of Medline, Toxline, Embase, the Science Citation Index within the Web of Science, Chemical Carcinogenesis Research Information System (CCRIS), Hazardous Substances Data Bank (HSDB), and Integrated Risk Information System (IRIS). These searches used the substance’s typical name(s) and its Chemical Abstracts Service (CAS) registry number, if appropriate. The searches were designed to identify literature evaluating the safety or potential long-term effects of exposure to the substances from studies in vitro, in animals, or in humans. Papers reporting use of the substances in laboratory procedures without evaluation of health effects were not considered relevant, nor were papers in which the substance was not the focus of the study.

On the basis of these reviews, the committee formulated hypotheses that certain adverse outcomes might be more likely for SHAD participants in tests that used any of six particular substances (see Table 3-1): Coxiella burnetii (C. burnetii), Escherichia coli (E. coli), Staphylococcal enterotoxin type B (SEB), sarin, betapropiolactone, and zinc cadmium sulfide. These hypotheses were seen as an opportunity to focus a portion of the analysis in a way that was not attempted in the IOM’s previous SHAD study, and were not intended as a weight-of-evidence review to conclusively attribute them to the exposures. The analysis of health outcomes considered morbidity, represented by record of an inpatient or outpatient diagnosis for any of the potentially associated conditions. For those conditions that might be an underlying cause of death, the analysis also included consideration of mortality. A summary of the reviews and a list of the papers that were examined appear in Appendix C.

Refining Exposure Metrics

Ideally, a study of the potential health effects of an exposure would be able to make use of a quantitative measurement of the dose received by each member of the study population to look for evidence of increasing risks for given adverse health outcomes with increasing doses of a given agent. Often, however, levels of exposure must be assessed less directly. When that is necessary, levels may be defined qualitatively (e.g., high, medium, low) and may be based on indirect markers of exposure such as time spent in a particular job assignment.

Because the SHAD tests were generally designed to assess physical penetration of ship spaces by the test substances, or the effectiveness of protective gear, information on human exposure was not systematically collected. For the SHAD I study, an individual’s exposure was defined most broadly as any participation in any SHAD test. Additional analyses grouped individuals based on the substances used in the tests in which they had participated. Four of these

______________

¹ These papers were prepared in 2004 and are available at http://iom.nationalacademies.org/reports/2007/long-termhealth-effects-of-participation-in-project-shad-shipboard-hazard-and-defense.aspx.

TABLE 3-1 Hypotheses to Be Tested Concerning Certain Substances Used in SHAD Testing and Adverse Health Outcomes

^a Because the individuals who served on the vessels involved in testing staphylococcal enterotoxin type B were not identified, the committee could not test the hypotheses generated for this agent.

exposure groups were defined: Group A for simulants Bacillus globigii (BG) or methylacetoacetate (MAA) only; Group B for simulant trioctyl phosphate (TOF) only; Group C for any active nerve or biological agent (with or without simulant exposure); and Group D for other substances or combinations of substances, excluding any active agents. These groups are described further in the SHAD I report (IOM, 2007).

For the SHAD II study, the committee sought to refine the representation of exposure within the limits of the information available. Although the more desirable forms of exposure data were still not available, the committee took advantage of other information to characterize potential variations in exposure among the SHAD veterans in ways that were not attempted in the SHAD I study. One step was to make use of information about the number of tests, and the number of trials within each test, in which a SHAD veteran was a participant. This information made it possible to estimate the number of potential exposures to a test substance each participant may have received. The committee also chose to group exposures to biological agents separately from exposures to chemicals.

The formulation and application of these exposure indicators in the SHAD II analysis are discussed further in Appendix D.

Obtaining Information About Health Outcomes

Identifying health outcomes, which include deaths and illnesses, among the SHAD participants and the comparison population of military veterans was a crucial part of both the SHAD I and SHAD II studies.

Deaths (Mortality)

Both studies relied on the National Death Index (NDI) as a source of information on causes of death for men who died in 1979 or subsequently. The NDI, which is operated by the National Center for Health Statistics, maintains a compilation of information from states’ death certificates issued from 1979 forward (NCHS, 2015), and can provide data on cause of death. Records for deaths that occurred before 1979 are held by individual states. For the SHAD II study, an effort was made to obtain information on the causes of as many of these pre-1979 deaths as possible, a step that was not feasible for the SHAD I study.

Illnesses (Morbidity)

The SHAD I and SHAD II studies took fundamentally different approaches to determining the health status of the surviving members of the study population. For SHAD I, an extensive survey was used to collect self-reported information on physical and mental health status, health history, and related information such as smoking history. SHAD II used records of from Medicare and VA of health care received by members of the study population. The alternate approach was used to obtain data that was more representative of the entire study population than that used in SHAD I. The response rate for the SHAD I survey was 61 percent for SHAD participants and 47 percent for the comparison population.

While surveys have the advantages of using a single framework for collecting information and of obtaining information about the respondents’ behaviors, experiences, and perceptions that are not consistently captured in medical records, they are vulnerable to biases related to self-reports (e.g., flawed recall and nonresponse). Records generated in conjunction with the delivery of health care services offer diagnostic information produced over many years and with the benefit of clinical expertise. The records of health care services used in the SHAD II study already exist in electronic form and do not vary by exposure status (i.e., participation in SHAD or not) or depend on the accuracy or completeness of a respondent’s recall or response. This approach was not suitable at the time of the SHAD I study, when only 18 percent of the study population were age 65 or older and age-eligible for Medicare services. At the time the SHAD II study follow up concluded, 83 percent of the surviving members of the study population were age-eligible for Medicare services

STUDY DESIGN

As in SHAD I, the SHAD II study was a retrospective cohort design (Hulley et al., 2013). With this approach, the exposure(s) of interest and the recording of health outcomes all occurred in the past. The data were collected for purposes other than the study, such as in electronic clinical or administrative databases. The study was conducted with the oversight and approval of the National Academy of Sciences, Engineering, and Medicine Committee to Review Studies on Human Subjects.

All cohort studies, including the SHAD studies, face the challenge of ensuring the comparability of the exposed group to the comparison population and controlling for factors that could lead to a spurious presence or absence of an association between the exposure and health effect. The SHAD I study sought to maximize the comparability of the comparison group by selecting military personnel from comparable ships or Marine Corps units that were serving at the same time as the units that participated in the SHAD tests. Use of this comparison population counters the concerns that might

be anticipated if the SHAD participants were compared to a group of civilians or military personnel who were not deployed. The SHAD II study used the same comparison population.

A major assumption in the selection of the comparison group is that the two groups are at similar risk for other potentially health-altering exposures, including combat duty in Vietnam, behaviors known to influence health outcomes (e.g., alcohol and tobacco use), and post-SHAD employment. The SHAD I health survey found that similar percentages of the SHAD and the comparison respondents had ever smoked cigarettes (79.1 percent and 82.5 percent, respectively) or currently drank alcohol (58.2 percent and 57.1 percent, respectively). Similarly, data from the Veterans Benefits Administration (VBA) showed that 17.4 percent of SHAD participants and 18.4 percent of the comparison group had been identified as having service in Vietnam.²

STUDY POPULATION

Identifying SHAD Participants

For the first SHAD study, the Department of Defense (DoD) assembled an initial roster of Project SHAD participants and updated the roster as new information was gathered. The IOM staff independently reviewed the quarterly unit rosters (BuPers Report 1080-14) as well as the daily personnel diaries for each ship in each test to compile lists of personnel present on the vessels involved in Project SHAD tests. When the IOM identified personnel who were not on the DoD lists, these names were provided to DoD for review and validation. The IOM kept in its study only those men that DoD could validate as SHAD participants and excluded those that DoD could not validate (IOM, 2007, p. 16). On the advice of the SHAD I advisory committee, the IOM staff undertook additional outreach efforts to inform the veteran community of the study and encourage those who might have been SHAD participants to contact DoD to be included in the study. No more than two study participants were added through these efforts.

The statement of task for the SHAD II study stipulated that the same population of SHAD participants and their comparison group that had been collected and validated for the SHAD I study was to be used in the SHAD II study. The IOM therefore did not repeat its independent review of personnel diaries and quarterly reports. However, the IOM did request and receive from DoD an updated listing of SHAD participants (as of December 2012) and reviewed it to ensure that any SHAD participants who had been newly identified since the completion of the SHAD I study were not missed. The IOM’s review of the updated DoD listing resulted in the addition of nine men to the population of SHAD participants. These men were participants in nine different SHAD tests.

For an individual to be considered eligible for inclusion in the SHAD I study, it was necessary to have information sufficient to uniquely identify him for purposes of contact for the survey. For the SHAD II study, 19 men were excluded because the available information (name or service number) was insufficient to determine their social security numbers, the identifier by which information was extracted from records on vital status and health care services received.

______________

² These data are for veterans who have filed a disability claim and were identified by VBA as having had Vietnam service. Vietnam service is defined as duty or visitation within the country of South Vietnam, including Navy veterans who entered inland waterways on ships, went ashore from offshore ships, or served only aboard offshore ships and never went ashore or entered inland waterways (Personal communication, J. Sampsel, Veterans Benefits Administration, March 12, 2014).

An additional 42 men were found to be ineligible because they were participants in (or from comparison units for) Project 112 tests that were not part of SHAD testing. Also, despite the data validation efforts by DoD and the IOM, three men were excluded because they had dates of death that preceded the dates of the SHAD tests in which they were supposed to have participated. An additional six men whose records indicated that they were dead but for whom dates of death were missing were included in the study population, but they could not be included in the survival analysis described in Chapter 4.

SHAD Comparison Group

As described in greater detail in the report on the SHAD I study (IOM, 2007), the SHAD comparison population was drawn from two sources. Each ship in a SHAD test was matched, to the extent possible, with a ship of a similar class, crew complement, operating area, and home port at the time of the SHAD test. For Marine Corps units, the comparison groups were selected to be similar units in operation at the same time as the SHAD test; where possible the identical unit in a parallel battalion or division was selected. The list of comparison units is provided in Appendix E.

Once the comparison ships or units were selected, the individuals serving on the ships or with those Marine Corps units were identified using a process similar to that used for the participant ships. Quarterly reports were obtained for the time period of the test and the name, military service number, and rate or job title were recorded for each person in the unit at the corresponding time. In this manner, 6,753 individuals (including 6,137 Navy, 614 Marines) were identified to serve as the comparison population for the study of the 5,868 service members (including 5,066 Navy, and 750 Marines) identified as participants in the SHAD testing.

One exception to this process was for the Navy service members who served on the five light tug boats that participated in the SHAD testing. These tugs were used to set up sampling arrays in the open ocean for the tests of active biological agents and simulants, as well as in one known instance, they were used as couriers of samples. The tugs were originally built for the Army, but were specially modified with small laboratory spaces for use as sampling stations for the SHAD testing. No comparable vessels were part of the regular Navy fleet, and as a result no comparison ships and crews could be identified. Moreover, crew records for the tugs were not maintained in the same fashion as for the other ships. The only test for which a list that identifies men who served on the tugs (9-12 men per vessel) is available is Shady Grove. The list includes members of the Project SHAD technical staff who were detailed as crew members of the five light tugs and to the laboratory ship, the USS Granville S. Hall. The choice of a comparison group for the personnel from the tugs and Project SHAD technical staff is discussed in Appendix D.

ASSESSING EXPOSURE

The SHAD tests involved at least 19 different vessels and 7 Marine Corps units in 21 tests that used at least 16 different test substances (alone or in combination) over a period of 7 years. Each test included multiple trials, ranging from as few as five trials (Test 69-10) to as many as 31 (Flower Drum I). The units that participated and the substances that were used could vary from trial to trial within a test. Some ships were involved in only one test whereas others participated in multiple tests. Ships that participated in multiple tests had changes in crew members from test to test. Approximately 69 percent of the SHAD participants (4,050 men) were

in only one test. The remainder participated in from two to seven tests. Because the nature of the tests varied widely, the nature of SHAD veterans’ potential exposures also vary.

Information about the substances used in the SHAD tests—active agents, simulants, tracers, and decontaminants—was available to the committee from three sources. The fact sheets for each Project SHAD test that DoD prepared and posted online³ give summary information about the substances used in the tests, the units involved, and the range of test dates. Additional detail about many of the tests and the substances used was available to the committee from declassified portions of the final reports and other documents prepared by the Deseret Test Center at the time the tests were being conducted. In addition, information on the dates of each test’s trials and test substances was compiled by DoD in 2004 during its efforts to identify and declassify information sought for the IOM SHAD I study.

Environmental concentrations of test substances are frequently used as alternative indicators of potential human doses. Although environmental readings were collected in many SHAD tests, the data remain classified and were not available to the committee. To characterize exposures that SHAD participants may have received to the substances used in the various tests, the committee used multiple approaches. First, a strictly qualitative approach grouped SHAD participants on the basis of the substances used in the tests in which they participated. Second, combining information on the substances used in SHAD tests with the number of exposure opportunities represented by number of trials within each test also made it possible to group participants on the basis of higher and lower levels of presumed exposure to certain substances.

Qualitative Exposure Groups Related to Test Substances

Some SHAD tests made use of a single substance, but many used a combination of substances (e.g., a simulant and a tracer). As a result many of the test participants were exposed to multiple substances. For the SHAD II study, the committee created substance-specific exposure groups for the six substances for which hypotheses regarding health outcomes were formulated, The committee also created three broad exposure groups based on the substances used in the SHAD tests: (1) any biological substance; (2) any chemical substance, except trioctyl phosphate (TOF); and (3) any decontaminant. This approach was used to examine the potential for a broad category of test substances to be associated with adverse health outcomes within a group with a variety of specific exposure experiences. Exposure groups 1, 2, and 3 are not mutually exclusive; a SHAD test participant might be included in any or all of them.

The committee also carried out analyses of three groups with distinctive aspects of their exposures: (1) the crew of the USS George Eastman, a specially outfitted ship involved in multiple tests including some with live agents; (2) members of the Project SHAD technical staff who were identified in a roster from January 1965, with additional persons with service on the light tugs; and (3) participants in Test 69-10 who received unique exposure to TOF. The membership of these special groups was largely mutually exclusive. The potential exposures to these groups are listed in Box 3-1.

______________

³ See http://www.health.mil/Military-Health-Topics/Health-Readiness/Environmental-Exposures/Project-112SHAD/Fact-Sheets (accessed October 15, 2015).

The comparison group for each of these six exposure groups was determined on the basis of the tests in which the exposures occurred, the units in the test that received the specified exposure, and the designated comparison unit for that test. The selection of these comparison groups is described further in Appendix D.

Special consideration was also given to the crew of the USS Granville S. Hall, which served as the laboratory ship for many of the SHAD tests. Because of its role, care was taken to keep the ship upwind from the tests specifically to avoid exposure to the test substances. The ship’s regular crew was supplemented by identified members of the Project SHAD technical staff. Regular crew other than the laboratory staff were not permitted in the laboratory area or to handle samples (Personal communication, J. Alderson, U.S. Navy Reserve (Ret.), January 2012).

For overall analyses, the committee included all Granville S. Hall crew members as part of the Project SHAD study population. For analyses that addressed specific forms of exposure, the Project SHAD technical staff was considered as potentially exposed and the members of the regular crew were excluded as unexposed.

Accounting for Trials as Well as Tests

Through a detailed review of the available documentation on each test, the study staff compiled information on the dates of the trials in each test, the test substances used for each trial, the equipment and patterns used to disperse test substances, and the ships or other units participating in the trial. Summaries of this information are provided in the annex to Chapter 2. The data compiled by the study staff were tabulated to arrive at cumulative numbers of exposure events for each of the test substances for each individual identified as being present during the test period. Presence during a trial was established from information the IOM staff collected from the Navy quarterly unit rosters and a review of the daily personnel diaries for each ship, which noted individuals who came on or left the ship each day. Similarly, information from monthly personnel rosters for Marines was augmented with company diaries regarding the movement of individuals into and out of units during the test trials (IOM, 2007). Committee members reviewed the source materials and the compiled information on trial exposures as well as the procedures used to tabulate the exposure hits received by each member of the group of test participants.

“Exposure” for each SHAD participant was represented by the number of trials within each test and the number of potential exposure opportunities for a given test substance within each trial.⁴ Each exposure opportunity was counted. For example, if a sailor were present on the USS Navarro for all of the nine trials of the Autumn Gold test and for the three trials the Navarro participated in during the Eager Belle II test, he would be considered to have 12 exposures (9 + 3 = 12) to BG. If he was on the ship for the three Eager Belle II trials and for only five trials during the Autumn Gold test, he would have a total of 8 exposures (5 + 3 = 8), and so on. Because some trials included release of more than one sprayed “line” of a test substance, each line was considered to constitute an exposure opportunity and counted as an additional exposure. This approach was used to generate a count of potential exposures to each separate test substance for each individual.

In addition to tallying the number of exposure opportunities individuals had for a test substance, the committee considered their distribution among the SHAD participants. For some substances, the exposed participants had all similar numbers of exposure opportunities, but for others the number of potential exposures varied widely. Where the size of the exposed group and the variation in the number of potential exposures were considered sufficient, the committee established subgroups to reflect the range of exposure experience (e.g., low, medium, high) among the SHAD participants. Appendix D includes a description of the process used to evaluate the distribution of hits and determine the exposure subgroups.

______________

⁴ For the purposes of this report, “test substance” refers to any of the active agents, simulants, tracers, and decontaminants reported by DoD to have been used in each given test.

Classified Data Concerning Exposure

As previously noted, information from many documents (e.g., test plans, final reports) from the Project SHAD tests remains classified and was not available to the committee for its work. Although many documents are automatically declassified after 25 years, the records on SHAD concern testing of chemical and biological agents and the automatic declassification rules do not apply (Personal communication, M. Kilpatrick, Department of Defense, January 19, 2012). In its 2000-2003 investigation of Project 112/SHAD, DoD committed to declassifying “medically relevant” information, but it did not include information such as nozzle size or concentrations of test agents sprayed out of concern that the information on delivery concentration could be calculated. Many of the SHAD-related DoD documents, including the declassified materials, that the committee had access to are identified in the Chapter 2 annex. All of the materials available to the committee are also publicly available.⁵

HEALTH OUTCOMES

Health outcomes were evaluated for this study using data on mortality (timing and underlying cause of death) and morbidity (diagnoses received from inpatient and outpatient care).

Mortality

Mortality follow up for SHAD participants began at the date of their first test exposure, and ended on the date of death or December 31, 2011, whichever came first. For the members of the comparison group, mortality follow up began on the first day of the test for which their vessel or unit was selected as a comparison unit. As of the end of follow up for the SHAD I study (December 31, 2004), 22 percent of SHAD veterans and 21 percent of the comparison population had been identified as deceased (IOM, 2007). The vital status of the remaining members of the study population—whether they were known to be dead or presumed alive on December 31, 2011—was established for the current study using VBA records and information available through Lexis/Nexis.

VBA gathers reports of veterans’ deaths through notifications from Veterans Health Administration (VHA) facilities, family members applying for death benefits, or VA’s National Cemetery Administration. It also obtains information through an automated match with the Social Security Administration (SSA) Death Master File. Historically, ascertainment of deaths through the VA benefits system has been found to be nearly complete. Lexis/Nexis was another source of vital status information. A comparison of the SHAD study population with the VBA records and Lexis/Nexis led to ascertainment of 1,347 new deaths since the final ascertainment for the SHAD I study.

______________

⁵ The declassified portions of the SHAD II reports are publicly available from the DoD Freedom of Information Act (FOIA) Reading Room at http://www.dod.mil/pubs/foi/readingRoom.html (accessed November 16, 2015). Additional information is available through the Public Access Records Office of the National Academies of Sciences, Engineering, and Medicine.

Cause of Death

Information on underlying cause of death for members of the study population was obtained in four ways: (1) from the search of the National Death Index (NDI) Plus database that was conducted for SHAD I; (2) from a search of the VA/DoD records on cause of death for veterans, which also draws on NDI Plus; (3) from a direct search of NDI Plus requested by the IOM staff; and (4) from death certificates obtained from state vital records offices for men who died before 1979. Overall, a cause of death was ascertained for 91 percent of the men determined to be deceased by the end of study follow up, December 31, 2011. The cause of death was not successfully obtained for the remaining 9 percent of deceased men. In some cases the cause of death was not obtained because there was not a good match with the NDI Plus database.

For the 162 SHAD participants and 206 members of the comparison population who died before 1979, identifying causes of death required determining the state in which the person died and submitting a request to the state’s vital records office for a copy of the death certificate. Each state has its own procedures and requirements for processing such requests. Because place of death is not reliably recorded in the records on the members of the study population from VA and DoD, the IOM staff supplemented information on residence and claims processing from VA and DoD records with searches of electronic death indexes maintained by some states and of other online genealogic resources. Requests for death certificates were submitted to 34 states, the District of Columbia, and the Philippines, resulting in the receipt of 184 death certificates from 28 states. Other online sources produced information on an additional 14 early deaths among the SHAD study population.

For deaths that occurred before January 1, 1979, cause of death may be missing because place of death could not be identified and the death certificate could not be requested; a requested death certificate was not located by the state vital records office; or it was not possible to successfully meet a state’s requirements for access to vital records. Additional details regarding steps used to ascertain cause of death are provided in Appendix D.

Morbidity

The SHAD II study assessed morbidity through records collected into automated databases as a byproduct of medical care received through VA (1997-2011) or under the auspices of Medicare billing (1999-2011). After leaving military service, some veterans receive medical care from VA facilities, many receive care from clinicians and facilities in the community, and some may receive care from both sources. On reaching age 65, almost all veterans become eligible for Medicare, but some may continue to receive some or all of their care through the VA or private insurance. The majority of US veterans have private health insurance or are eligible for Medicare and typically receive their care from sources outside VA (Shen et al., 2003; VA, n.d.-a; Westat, 2010).

VA Inpatient and Outpatient Visits

For veterans who use the VA health care system, diagnostic data are available from automated records for inpatient and outpatient medical encounters from October 1999 to the present (VA, n.d.-b). Records through December 31, 2011, were used for the study. In some cases, records of VA health care provided to those in the study population were available from late 1997, and these were used where available. Eligibility for VA health services is generally

based on a combination of criteria for minimum length of active duty service and the terms of discharge from military service. Additional criteria concerning a veteran’s health status, military service, and income level are used to determine eligibility for one of eight priority groups, which are used to manage demand for VA health care services. Veterans classified as at least 50 percent disabled for a service-connected condition qualify for the highest priority.

For this study, the committee drew on information available from multiple VHA datasets, including Inpatient Encounters (IE), Bed section acute care dataset (PB), Main acute care dataset (PM), Procedure acute care dataset (PP), Surgery acute care dataset (PS), and Event dataset (SE). The files captured overlapping information on inpatient and outpatient encounters among those who use VHA health care services.

Medicare Data

Medicare is a federal health insurance program for people age 65 and older and certain persons who are younger and meet Social Security Administration requirements for disability. Once people are enrolled in Medicare, electronic records (i.e., claims) for each of their medical encounters that are processed by the Centers for Medicare & Medicaid Services (CMS) are retained. These records include information about the date of care, the reasons (diagnosis), and the type of care received. Records of these medical care claims are potentially available for use in research upon approval of investigators’ ability to protect the privacy of study subjects. Medicare data are widely used for studies of health care including studies of effectiveness, access to care, disease incidence, and outcomes of care. Estimates suggest that approximately 96 percent of those who are eligible on the basis of age are covered by Medicare (EBRI, 2003).

Medicare enrollment information is provided in the Master Beneficiary Summary File (formerly the Denominator File). This file contains on an annual basis information about Medicare enrollment status and eligible benefits as well as demographic data and information about mortality. Medicare files on health care use are divided by the type of service provided. The Medicare Provider Analysis and Review (MedPAR) file contains information about hospital stays. The Outpatient and Carrier files contain information about services provided in non-inpatient facilities and care by physicians.

There are important limitations of Medicare data that must be acknowledged. First, the Medicare data do not include information about health care that is provided outside of the Medicare program. Second, the data do not include information about care provided by Medicare Managed Care enrollees. Medicare data contain information only about conditions for which diagnoses have been recorded as part of a health care claim. Some conditions that could have been (or should have been) diagnosed are not recorded. For example, dementia has consistently been shown to be under-recognized in clinical settings (e.g., Mitchell et al., 2011), and therefore is also missing from the Medicare data. Finally, Medicare records inconsistently capture information about health behaviors or risk factors such as tobacco use and obesity. Many of these limitations also apply to VHA data.

DATA PROCESSING AND ANALYSIS

To meet VA requirements for the protection of the personally identifiable data that were the foundation for the study, it was necessary for all the data to be stored and used within the VA’s information technology system, known as VINCI (VA Informatics and Computing

Infrastructure).⁶ Once data from separate sources were matched, analysis files were created using case numbers without name and social security number. Data management was handled through Structure Query Language. All analyses were performed using Statistical Analysis System (SAS) or R software. The principal analyses were Kaplan-Meier survival analyses and Cox proportional hazards regression analyses for mortality, and logistic regression, Fisher’s exact tests, and Wilcoxon rank-sum tests for indicators of morbidity, including median hospital days.

The data were cleaned and prepared by members of the study staff, and analyses were carried out by a consultant to the study. For each analysis undertaken, counts and simpler analyses such as percentages were confirmed by study staff using Microsoft Access. The committee determined the analyses to be performed and reviewed the results; committee members did not have access to the individual level data on health outcomes or the VINCI system where the analyses were run. Some statistical analyses that did not require access to the individual level data were carried out by a committee member. At several steps in the planning and performance of the analyses, a committee member reviewed elements of the SAS code used to ensure their appropriateness. The entire committee regularly reviewed results of the analyses and checked for consistency and interpretability. When questions arose, the committee proposed validation and alternative analyses.

Under the requirements governing use of data from CMS and in order to apply a consistent approach to all health outcome data, cell counts of 10 or less were reported only as ≤10. In situations where a simple calculation could permit reconstruction of a small cell size, the table shows NR, for not reported. The committee reported confidence intervals without the point estimates for results of analyses for which it was feasible to reconstruct a small cell size. Additional detail about analytical approaches is provided in Chapters 4 and 5 and Appendix D.

REFERENCES

EBRI (Employee Benefit Research Institute). 2003. Health insurance and the elderly. Facts from EBRI. Washington, DC: EBRI. http://www.ebri.org/pdf/publications/facts/0803fact.pdf (accessed October 27, 2015).

Hulley, S. B., S. R. Cummings, W. S. Browner, D. G. Grady, and T. B. Newman. 2013. Designing clinical research. Philadelphia, PA: Lippincott Williams & Wilkins.

IOM (Institute of Medicine). 2007. Long-term health effects of participation in Project SHAD (Shipboard Hazard and Defense). Washington, DC: The National Academies Press.

Mitchell, A. J., N. Meader, and M. Pentzek. 2011. Clinical recognition of dementia and cognitive impairment in primary care: A meta-analysis of physician accuracy. Acta Psychiatrica Scandinavica 124:165-183.

NCHS (National Center for Health Statistics). 2015. National Death Index. http://www.cdc.gov/nchs/ndi.htm (accessed October 13, 2015).

Rothman, K. J., S. Greenland, and T. L. Lash. 2008. Modern epidemiology, 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins.

Shen, Y., A. Hendricks, S. Zhang, and L. E. Kazis. 2003. VHA enrollees’ health care coverage and use of care. Medical Care Research Review 60:253-267.

______________

⁶ Data files and programs used in this analysis will be stored on VINCI following completion of this study to the extent permitted by the respective data use agreements under which the IOM was given access to the data and subject to the conditions of the agreement with VA regarding use of its VINCI system. It is anticipated VA will retain the data and programs for at least 1 year.

VA (Department of Veterans Affairs). n.d.-a. 2001 national survey of veterans: Final report. http://www.va.gov/VETDATA/docs/SurveysAndStudies/NSV_Final_Report.pdf (accessed October 27, 2015).

VA. n.d.-b. Corporate data warehouse: VA informatics and computing infrastructure. http://www.hsrd.research.va.gov/for_researchers/vinci/cdw.cfm (accessed October 27, 2015).

Westat. 2010. National survey of veterans, active duty service members, demobilized National Guard and reserve members, family members, and surviving spouses. Final Report. Rockville, MD: Westat. http://www.va.gov/SURVIVORS/docs/NVSSurveyFinalWeightedReport.pdf (accessed September 24, 2015).