E
Verification of Completeness and Accuracy of the Participant Roster
Introduction
Verification of the completeness and accuracy of the CROSSROADS participant file is particularly important to this study. In 1989, the Medical Follow-up Agency (MFUA) learned from the Nuclear Test Personnel Review (NTPR) that the participant roster provided for its 1985 "Five-Series" study of atomic veterans (NRC 1985) contained classification errors: approximately 15,000 nonparticipants were included as participants, while 28,000 actual participants were not included in the roster. These classification errors were discovered by NTPR in the process of updating its participant database after consolidating the individual service database into a single database in 1987. Subsequently, the General Accounting Office and the Office of the Technology Assessment (OTA) reported the origin and extent of the classification errors (GAO 1992; OTA 1992). The OTA report indicated that the number of misclassified individuals was smaller than originally indicated, but still significant, and recommended redoing that study.
NTPR delivered the first participant roster for CROSSROADS in 1986, well after the consolidated NTPR program began cleaning up the participant files. That means that the initial participant data should have been nearly complete and free of the classification errors experienced with the earlier study.
Nevertheless, we have taken additional steps to assure the quality of the participant identification process. This process included a comparison between the current (1994) data set and the 1986 data set, as well as comparisons with two other sources of participation information—a roster of CROSSROADS participants from the National Association of Atomic Veterans (NAAV) and a roster compiled from direct solicitation of information from veterans by MFUA.
Methodology
Comparison to Previous Versions of the Participant Roster
By comparing the current participant data set to the 1986 version and seeking verification of participation for sampled individuals, we were able to estimate a crude false positive rate and better understand changes that have occurred in the participant cohort over time. We drew a sample of 50 participants from each of the following categories:
- Participants who were found in both the 1986 participant list and in the current, 1994, participant list (matched, n = 39,844). Matched participants were defined by a corresponding first and last name and other confirmatory information such as military service number, date of birth, and unit of assignment. Allowances were made for obvious typographical errors (for example, "Johsnson, John" was accepted as a match to "Johnson, John" if there was also a match on service number or other confirming evidence).
- Participants who are currently in the study but could not be matched to a 1986 participant (n = 2,713; new-only's).
- Participants who were in the study in 1986 but could not be matched to a participant in the 1994 file (n = 667; old-only's).
MFUA requested documentation from NTPR to verify the status of each of the selected individuals. We then categorized every putative participant as a verified participant, a verified nonparticipant, or a person whose participation could not be verified using written documentary sources. To get a false positive rate we then calculated the fraction of inappropriately classified participants in the matched and new-only samples.
Comparison of Participant Roster to National Association of Atomic Veterans (NAAV) Mortality Study List
Estimating the number of persons erroneously left out of the study was more difficult than verifying the participation of those whose names were already known to be on the participant list. To estimate the false negative rate—
that is, the proportion of actual CROSSROADS participants who have been incorrectly excluded from the study—we needed to find participants who were unknown. We used two sources to find these additional participants.
NAAV provided us with a list of veterans (n = 1,251) who reported service in Operation CROSSROADS. Using this list as a benchmark, we estimated a false negative rate by matching the NAAV participants against those in our current data set, using the criteria presented above. NAAV participants were classified as either "matches" or "insufficient data."
The NAAV database was compiled by Mr. Boley Caldwell, director, NAAV Medical History Survey, from a number of medical surveys NAAV conducted on its members. The latest questionnaire was circulated in 1992 and is included in Appendix A along with Mr. Caldwell's comments on the data collection methods. Because there was insufficient time to obtain approval for follow-up, we have not attempted to contact individual veterans to verify or obtain additional identifying information. We have accepted the NAAV database as it was presented to us, editing only as necessary to ensure consistency of format in fields such as data of birth and to eliminate obvious duplicate records.
The NAAV benchmark represents a highly selective population, since it is based upon health surveys that were intended to determine potentially radiogenic mortality and morbidity among the atomic veterans. It is conceivable that veterans in the database may have been more likely to have contacted the NTPR program or the VA and, consequently, are more likely to be on our list of participants. To avoid this possible bias, we also sought participants through sources that were not connected with NAAV.
Comparison of DNA Participant Roster to a List of Participants Solicited through Veterans' Journals
In order to obtain a group of veterans for comparison who were not associated with NAAV, we placed announcements of the CROSSROADS study in several veterans' publications.28 We included the NAAV Newsletter for completeness and to see if NAAV responses would be different from the others. The publications that published our announcement (in some form) included:
- Journal of the Veterans of Foreign Wars,
- Journal of the American Legion,
- Journal of the Retired Enlisted Association, and
- The NAAV Newsletter.
With the exception of the NAAV Newsletter, we were limited to a few lines of text inviting a response from CROSSROADS veterans. The magazines edited the announcement to suit their needs for format and availability of space. The NAAV accommodated us with a half-page form for their readers to fill out and send in. That enabled us to distinguish between respondents who were newsletter recipients, and most likely members of NAAV, and those who were not.
We asked veterans of CROSSROADS to provide us with personal identification information and details of their participation. We refer to this as the write-in verification sample. This group (n = 477) was treated like the NAAV sample, but hopefully constitutes a less selective (and potentially less biased) comparison group. Because more data were available for individuals in the write-in group, we were able to classify them in more detail when we matched them to the 1994 participant file:
- "Matches" corresponded to individuals in the NTPR participant file as defined above.
- "Probable participants" included those who provided sufficient documentation (orders, "participation cards," narrative that indicated participation in the test, etc.).
- "Not-CROSSROADS" included individuals who mentioned the CROSSROADS study in their correspondence, but provided documentation of participation that definitely placed them at a different time and place—most often in another atomic test in the Pacific such as CASTLE (1954).
- "Unknown," which included:
-
- "Possible post-CROSSROADS." These individuals indicated participation during dates that were outside the operational period of CROSSROADS, but they may have been included as participants in the posttest period. Posttest participants are not included in this study as noted in Chapter 6.
- "Possible Kwajalein/Enewetak." These individuals told us of their assignment on Kwajalein or Enewetak Atolls during the operational period of CROSSROADS. Kwajalein was about 300 miles southeast of ground zero for the test and was both a staging area for test personnel and a garrison for military personnel not connected with CROSSROADS. Participation for military personnel assigned to Kwajalein depended upon whether they were assigned to a CROSSROADS unit or to a nonparticipating garrison unit. Enewetak is about 200 miles west of ground zero; the same participation rules apply.
- "Insufficient information." These individuals did not provide enough information to classify them into one of the above categories. Typically, these responders provided only last name and initials or a nickname, with no other identifying information.
Individual Follow-Up on Putative Participants of Uncertain Status
As a final quality assurance measure, we sought additional participation information on the NAAV (n = 90)29 and write-in (n = 17) individuals whom we could not positively identify as either participants or nonparticipants. We sent identifying information for these 107 individuals to DNA, and asked for verification of participant status through a search of the NTPR database and personnel records.
Results
Comparison with an Earlier Version of the Participant Roster
Participants Found in Both 1986 and Current Data Sets
Among the 50 individuals sampled from participants that were in both the 1986 and current data sets, 49 were found to be in the study. One was found to be an error; he had been transferred to a non-CROSSROADS unit on 30 June 1946, the day before the first detonation.
Participants Found Only in the Current Data Set—New-Only's
Among the sample of 50 new-only participants whose names were found in the present study, but could not be found in the 1986 data, 45 were confirmed as new participants. For one, documentation found during the validation process indicated that the individual left active duty with the Navy in November 1945 and would not have been at CROSSROADS. For another participant, the record search revealed no data that could absolutely confirm or deny participation. Three (3) individuals were not assigned duties in the CROSSROADS test but were passengers in transit aboard the PANAMINT, a CROSSROADS-participating ship. These "in transit" personnel will be discussed later in detail.
In summary, the review of the newly added participants resulted in 48 being confirmed (includes the 3 PANAMINT passengers), 1 erroneously included, and 1 unverifiable.
Participants Found Only in the 1986 Data Set—Old-Only's
Of the sample of 50 participants found only in the 1986 list, and not matched to the current participant list, 26 (52 percent) were still participants. They had not matched until their identification information (spelling of name or service number) had been corrected during clean-up. Another comparably sized group of 23 (46 percent) were confirmed deletions for the following reasons:
- one was an "in transit" case, as described above.
- thirteen were deleted during the clean-up of the CROSSROADS data when information indicated they were elsewhere during the test.
- nine were deleted during clean-up when their units were found not to have met eligibility criteria.
The remaining individual was identified only by last name and initials of first and middle name. Although he appears on both old and new CROSSROADS rosters as a participant, the lack of definitive identification prevents absolute confirmation of his status.
NTPR estimated that there should have been only 100 participants eliminated from the study between 1986 and 1994. Hence, NTPR was justifiably concerned that we found 667 individuals in the old data set who could not be matched, with certainty, to the current one. They also agreed to investigate the validity of the remaining 617, in addition to providing documentation on the 50 sampled old-only's. The final status of the remaining 617 are shown in Table E-1.
TABLE E-1. Resolution of 617 Apparent Participant Deletions Between 1986 and 1994
No. of Apparent Deletions |
% |
Resolution of Participant Status |
333 |
54 |
In the participant list with corrected identification data |
273 |
44 |
Validated deletions—new data contradicted participation |
7 |
1 |
Erroneous deletions |
4 |
1 |
Records not available at time of writing or insufficient information available for positive identification |
617 |
100 |
Total personnel researched |
Of the seven erroneous deletions, three were Marines who were assigned to CROSSROADS but stationed on Kwajalein Atoll (see discussion above of "Possible Kwajalein/Enewetak" participant). Three were found to have been assigned to participating Navy ships. One was noted as a validated deletion without further annotation.
Comparison with the NAAV Medical Survey
Of the 1,251 veterans in the NAAV Medical Survey who indicated participation in CROSSROADS, we were able to match all but 89 to our current participant list. For these 89 individuals we had insufficient information to declare them a match.
Comparison with the Write-in List
The amount of information provided by those who responded to our publication inquiry varied widely. Some veterans provided detailed documentation of their participation, including both official government documents and their own narrative description of events they witnessed. Others provided only their name and a statement that they were present at CROSSROADS.
In all, we received 477 responses that mentioned CROSSROADS in one way or another. When we matched the respondents to our participant list, we obtained the results shown in Table E-2. Respondents who used the NAAV form are tabulated separately from those who responded by letter or other written means.
TABLE E-2. Summary of Completeness of the NTPR Participant List as Indicated by Veteran Responses to MFUA Solicitations Published in Veterans' Publications
Match Status |
NAAV* Form Respondents |
Other Respondents |
Total No. of Respondents |
Matched to Participant File |
174 |
271 |
445 |
Not in CROSSROADS |
5 |
10 |
15 |
Probable Participant, not listed on the file |
0 |
7 |
7 |
Unknown—Possible Enewetak Participant |
1 |
0 |
1 |
Unknown—Possible Kwajalein Participant |
0 |
3 |
3 |
Unknown—Possible Post Participant |
1 |
2 |
3 |
Unknown—Insufficient Information |
0 |
3 |
3 |
Total Number of Respondents |
181 |
296 |
477 |
* Those respondents who replied by means of a form published by the NAAV in their newsletter. |
Follow-Up on Putative Participants of Uncertain Status
Of the 107 individuals we sent to DNA for individual verification of status, 1 was found to be a duplicate and 3 were found on both the NAAV and write-in lists. Of the 103 unique records, DNA provided participation information summarized in Table E-3.
TABLE E-3. Disposition of 103 Records of Putative Participants in CROSSROADS Whose Status was Uncertain
No. of Individuals |
Disposition |
1 |
Newly identified navy participant who went to the CROSSROADS operational area but was not on the muster roll of the ship to which he was assigned. Participation was ascertained by reference to individual's personnel record. |
1 |
Newly identified Army Air Corps participant assigned to Kwajalein. Not assigned to operation CROSSROADS per se, but his personnel jacket noted he flew missions in support of the operation. He therefore qualifies as a participant. |
1 |
Newly identified Navy participant assigned to Naval Air Station, Kwajalein. He had administrative duties on Kwajalein in support of CROSSROADS and therefore qualifies as a participant. |
1 |
New Navy participant whose service record shows he was assigned to Naval Air Base Kwajalein in support of CROSSROADS. |
65 |
Non-participants who had personnel records showing duties with non-CROSSROADS units during the operational period. Many of these were identified as participants in other nuclear tests, most frequently SANDSTONE (1948). |
19 |
Unknown. Personnel records were not available, because they were burned or there was insufficient information available to retrieve a record. |
15 |
Confirmed as participants and found on the MFUA study roster based on name match. These individuals are considered by DNA to be participants, and are in the MFUA study file. They were not initially matched because of unrecognized typographical errors in the source list (ex. Vonname should have been Von Name). |
103 |
Total records considered |
The final disposition of the 89 discrepancies from the NAAV Medical Survey list are summarized in Table E-4.
TABLE E-4. Disposition of 89 Discrepant Records from the NAAV Medical Survey List
Disposition |
Count |
Matched to participant file |
10 |
Newly identified CROSSROADS participants |
3 |
Civilians in CROSSROADS but not in the study |
2 |
Post-CROSSROADS participant |
1 |
On MFUA study roster but found ineligible |
1 |
Not in CROSSROADSa |
57 |
Unknownb |
15 |
Total |
89 |
a These individuals were assigned duties elsewhere during the CROSSROADS operational period. Several were found at tests other than CROSSROADS; SANDSTONE, 1948, was common. b Personnel records could not be found for the se individuals to verify or disallow participation. |
With the additional data provided by DNA on the 17 unresolved cases in Table E-2, we were able to collapse the write-in data as shown in Table E-5.
TABLE E-5. Summary of Completeness of the NTPR Participant List According to Write-in Data (includes verification of 17 discrepancies by personnel record searches at DNA)
Match Status |
NAAV Respondents |
Write-in Respondents |
Total No. of Respondents |
Matched to participant roster |
174 |
272 |
446 |
Newly identified participant |
0 |
1 |
1 |
Not in CROSSROADSa |
6 |
20 |
26 |
Unknownb |
1 |
3 |
4 |
Total number of respondents |
181 |
296 |
477 |
a These individuals were assigned duties elsewhere during the CROSSROADS operational period. Several were found at tests other than CROSSROADS; SANDSTONE, 1948, was common. b Personnel records could not be found for these individuals to verify or disallow participation. |
Discussion
The vast majority (93.6 percent) of participants in the old CROSSROADS data set (1986) matched with the new (1994). Based upon our evaluation of the
validation data, the primary reasons for mismatches were clean-up of identification information (service number, name spelling, etc.) and deletion of individuals (or units) whose participation was contradicted by new information. To determine the approximate numbers of participants who have been deleted and added to the study since 1986, we needed to correct for the participants who were currently in both the old-only and new-only categories because of corrections in identification information that prevented their matching. Using estimated proportions of verified and deleted participants, available in Table E-2, we estimated the actual numbers of participants added to and deleted from the study to be approximately 2,250 (5.3 percent) and 375 (0.9 percent) respectively. The remaining 0.2 percent is accounted for by erroneous deletions and by individuals for whom insufficient information exists to make a definitive determination of participant status.
Among the sample of 50 matched and 50 new-only participants, one individual in each group was found to be erroneously classified as a participant. One new-only had no records available and was classified as unknown. Based on the combined data for matched and new-only data, we estimate the false inclusion rate to be 2 percent or 3 percent, depending upon the status of the unknown participant.
We have included as confirmed participants, three new-only individuals who were ''in transit'' aboard the PANAMINT (AGC-13). These warrant special attention since they illustrate a point about the inclusion criteria for participation in CROSSROADS. Inclusion in the CROSSROADS participant list is primarily an administrative decision intended for compensation purposes and is not based on documented exposure to radiation (see note 10). The "in transit" individuals were not assigned duties in the CROSSROADS test, but were given passage aboard a CROSSROADS-participating ship.
The VA addresses the question of passengers as follows (VA 1993:
Persons whose only potential for exposure arose from passage on contaminated vessels would be deemed to have 'onsite participation' ... if they were passengers during an official operational period [emphasis added].
Initially, lacking a definition of "contaminated," NTPR considered these PANAMINT passengers to be participants. This appears reasonable, because the ship did not receive its official radiological clearance until 22 November 1946, well after the operational period of the test. However, the PANAMINT had an assigned dose of 0.00 Sv (0.00 rem) (i.e., those aboard received no radiation dose as a result of their presence on the ship). In July 1995, based on this dose, NTPR determined that PANAMINT did not appear to satisfy the "contaminated" ship criteria. Therefore, its passengers were designated
nonparticipants. For our purposes, we have left these (approximately 50)30 "in transit" personnel in the study, maintaining their classification as participants as of the freeze date of our data set (28 February 1995).31
We derived the missing participant rate (false negatives) by comparing the NTPR participant list to individuals on the write-in list compiled from publication advertisements and to those on the NAAV Medical Survey list. Because many of the write-in veterans provided detailed data in response to our solicitation, the process of matching them to the participant roster may have been somewhat more reliable than it was for the NAAV Medical Survey. As a result of this, and perhaps other factors, the fraction of exact matches was slightly higher for the write-in analysis—93.3 percent for the write-in vs. 92.8 percent for the NAAV Medical Survey.
Several individuals in the write-in sample (n = 477) provided evidence that they were actually not in CROSSROADS. Removing those respondents, we were left with a denominator of 462 potential participants. If we assume the worst, that all of the unknown and probable cases were true participants, then our false negative rate is 3.7 percent (17/462). On the other hand, if we assume the best case, that only the probable cases (n = 7) were actually left out, and that the other unknowns (n = 10) are not participants, we have a false negative rate of 1.5 percent (7/452). Thus, based on this data, the actual completeness of the 1994 participant file probably lies between 96 and 98 percent.
Of the total write-in responses we received, 38 percent were on forms from the NAAV Newsletter, while the remaining 62 percent were letters. Interestingly, none of those who may have been inadvertently left out of the study (probable participants, n = 7) were NAAV respondents. It is also interesting that the completeness as indicated by the NAAV responses is higher (99.4 percent) than completeness indicated by the non-NAAV responses (94.8 percent).
In evaluating the inclusion and exclusion figures cited above, it is important to consider the assumptions we have made in deriving them. Where there was doubt, we have erred to the side that would maximize the appearance of error in the completeness of the data set.
Unless otherwise stated, if an individual's participation could not be verified, his inclusion was considered an error.
If an individual did not match with name and other confirmatory information, he was considered a nonmatch. There are individuals in this category that have first and last name matches to the current participant list, but
who have no other confirming data. These could be valid matches, but are not counted as such.
We have assumed that all the information we have been given by the veterans and the NAAV are completely accurate. In fact, there may be misspellings of names or errors in transcription of identification numbers that prevented individuals from matching our participant list.
In summary, there are two important points to be made. First, the use of participants from the 1994 file is thoroughly justified. Overwhelming proportions of both matched (to the 1986 file) and new-only participants in the random-sample verification were shown by documentary evidence to have been at CROSSROADS. A significant proportion of old-only subjects were confirmed nonparticipants, and most of the remainder were actually found in the 1994 file, only with modified (presumably corrected) identifying information.
Second, a check of the completeness of the CROSSROADS file using information independent of the administrative systems that provided our participant data showed a high level of completeness of the participant roster. Around 93–99 percent of subjects who made independent claims of participation in CROSSROADS were found on the study roster. The proportion is even higher for the respondents to the advertisement if one excludes the few confused cases in which the subjects were probably in the area at the time of the tests but do not fit the precise administrative criteria for official CROSSROADS participation (see Chapter 6).
Above, we established the completeness of the study roster independent of the DNA-NTPR program. If we now consider the new information provided by DNA on the 103 unresolved discrepancies (Table E-3), our estimates of missing participants change somewhat. The estimate from the NAAV Medical Survey list (counting all nonmatches as missing participants) suggested a 7 percent missing participant rate. From the more detailed data available in Table E-4, we can count 10 individuals as being matched to study participants; 61, non-CROSSROADS; and 18, missed participants (3 newly identified plus 15 still unknown). That yields, based on the NAAV Medical Survey List, a missing participant rate of 1.5 percent (18/[ 1251-61 ]).
The write-in data yield similar results when adjusted for the additional follow-up information in Table E-5. DNA-NTPR found one new (i.e., missing) participant and was unable to find any records on four others. Twenty-six were described as not in CROSSROADS. Based upon those data, the missing participant rate becomes 1.1 percent (5/[47–26]).
In these revised calculations, we have removed the non-CROSSROADS individuals (10, NAAV; 26, write-ins) from consideration. We do that with some caution. Although it is unlikely, information that could have indicated a temporary assignment to CROSSROADS may not have been entered in, or may be missing from, personnel records. If some of these individuals had
undocumented CROSSROADS participation, our estimates of completeness will be toward the low end of the range cited above. But, we have retained all of the remaining unknown cases as missing participants, which tends to inflate the missing rate. In summary, the additional follow-up by DNA-NTPR suggests that the participant capture rate may be toward the high end of the 93–99 percent range.
Conclusion
We estimate that fewer than 2–4 percent of the current participants included in the NTPR data set may not actually be CROSSROADS participants (i.e., they are false positives). We are also confident that the data captures between 93 and 99 percent of those who were actually CROSSROADS participants. Since we were conservative in matching purported participants to the NTPR participant list, erring on the side of exclusion, the inclusion rate could actually be higher. If additional follow-up information provided by DNA-NTPR is considered, the capture rate would be toward the upper end of the range. In short, there is no evidence from any of the above sources that the roster of CROSSROADS participants studied by MFUA is deficient in a substantial manner.