agencies may compare their alternative with only one other option (e.g., a “do-nothing,” or status quo, option). The challenge for OMB, therefore, is to persuade agencies to perform more serious analyses of a next-best alternative or a variant of the preferred alternative. When OMB pushes for these additional analyses, it often experiences pushback from the agencies that express concerns about limited time or resources. Dr. Graham remarked that the FDA probably experiences a similar problem. While manufacturers may present data comparing their proposed therapy to a placebo, the more important clinical questions may relate to whether the proposed therapy is superior to other treatments already on the market. The solution to this problem is not straightforward. If the FDA were to compel manufacturers to generate data on a broader range of alternatives, the cost and delays associated with FDA approval would increase significantly.
A better solution, Dr. Graham argued, would be to stimulate more research and analysis during the postapproval period. Echoing what other presenters had suggested, this postmarketing research should be conducted by a variety of sources. Dr. Graham suggested multiple organizations that compete with each other for the reputation of doing quality work. An expansion of university-based programs in pharmacoepidemiology and pharmacoeconomics, with a mix of government and industry funding, would be a useful step in the right direction.
Dr. Graham then discussed the validation of benefit and risk data after a regulatory decision has been made. In its most recent report to Congress Validating Regulatory Analysis, OMB assembled all 47 published case studies (out of more than 20,000 new regulations since 1981) in which benefit and cost estimates had been validated after the rule was promulgated (OMB 2005). Such a limited sample allows only limited insights, but it is nonetheless interesting to note that federal regulators exaggerated both benefits and costs in most cases. They exaggerated benefit because they wanted their product to look good. They exaggerated cost because they underestimated the creativity of the industry in finding ways to meet regulatory requirements at lower cost. The report highlights the need for a broader literature to allow us to validate preapproval benefit–risk estimates.
With respect to the FDA, are there numeric projections that are falsifiable? Could we perform validation analyses on this process? While it is not obvious that this can be done, the advantage of doing it would be a track record of better estimations of risk and benefit. By documenting systematic errors, it becomes feasible to improve future benefit–risk analyses and identify situations where adjustments need to be made. Ultimately, Dr. Graham’s presentation raised concern about the lack of resources and incentives for following up on regulatory decisions.