- Using historical data, simulate enough test data to satisfy the protocol with the largest sample size. This can be done either nonparametrically (by randomly drawing with replacement from an appropriate pool of historical test data) or parametrically (by using summary statistics from previous tests).
- Note that the simulations may require several summary statistics (or appropriate data to resample), including probability of first and second shot complete and partial penetrations by shot order and first and second shot BFD by shot order. Depending on the criteria for each protocol, data may be simulated (for example, data on partial penetrations) that will be used to assess only one protocol.
- If the two protocols require a different sample size, randomly select the smaller sample size from the simulated test data.
- Compute whether or not the test would have been passed or failed under each protocol.
- Repeat the simulation as necessary and appropriate to estimate the four pairs of probabilities: (1) pass under protocol 1/pass under protocol 2, (2) pass under protocol 1/fail under protocol 2, (3) fail under protocol 1/pass under protocol 2, or (4) fail under protocol 1/fail under protocol 2.
- Repeat the simulation for a range of representative manufacturer and design results.
This type of analysis may point to specific design characteristics that are advantaged or disadvantaged by particular protocols.