deaths averted per year, x2 = primary benefit to infants and children; x3 = –$30,000/QALY gained; and x4 = requires refrigeration. The final SMART Score is then numerically calculated as follows:
The output indicates that the candidate vaccine is approximately 40 percent of the distance “up” from a vaccine that is scored 0 toward a vaccine that is scored 100.
Given the intrinsic variability in SMART Scores from user to user, how can the ranks then be compared in a user group? While some comparisons can be made, nothing guarantees the ability to map user A’s scores to those of users B, C, and D. The committee has used the analogy of Fahrenheit and Celsius thermometers to assist users in understanding what the SMART Scores mean (and what they do not mean), but the choices of parameter settings in the software may create further complexity in multi-user group settings.
All multi-attribute utility models—including SMART Vaccines—have the characteristic that a difference of, say, 10 points for user A has the same meaning all along user A’s scale, so it is correct to say that the difference between 20 and 10 has the same meaning as the difference between 80 and 70. But one cannot say that “20 is twice as good as 10” any more than one can say that “20°F is twice as warm as 10°F.” It is also correct to say that a 10-point difference on user A’s scale is not the same as a difference of 10 points on user B’s scale, just as with the thermometer analogy: A difference of 10 degrees is not the same in Fahrenheit and Celsius scales.
Unfortunately, the analogy becomes less useful when users A and B have employed a different set of attributes for their valuation. To the extent that they have commonly chosen attributes (e.g., premature deaths averted per year, cost-effectiveness, or potential to improve delivery methods), then the weights they have placed on these attributes lead to predictable changes in each user’s SMART Scores. If they have no common attributes in their respective value models, then it is not possible to compare one user’s SMART Scores (and hence rankings) to those of another user.