The elements of this multi-step process are detailed below:

  1. Agree on initial guidelines. These initial guidelines are based on the legislation, the Memorandum of Understanding, and contracts.

  2. Clarify objectives. What central questions must the study answer? What other interesting but optional questions should be addressed? What questions will specifically not be considered? This is discussed further in Section 3 of this chapter.

  3. Develop operational definitions: For example, while Congress has mandated that the study address the extent to which SBIR supports the agencies’ missions, the Committee needs to develop operational definitions of “support” and “agency mission,” in collaboration with agency managers responsible for program operations. This is a necessary step before developing the relevant metrics. This is discussed further in Section 4 of this chapter.

  4. Identify metrics for addressing study objectives. The Committee will determine extent of commercialization fostered by SBIR—measured in terms of products procured by agencies, commercial sales, licensing revenue, or other metrics. This is discussed further in Section 5 of this chapter.

  5. Identify data sources. Implementation of agreed metrics requires data. A wide mix of data sources will be used, so the availability of existing data and the feasibility of collecting needed data by different methods will also condition the selection of metrics, and the choice of study methods. The existence or absence of specific methodologies and data sets will undoubtedly lead to the modification, adoption, or elimination of specific metrics and methods. This is discussed further in Section 6 of this chapter.

  6. Develop primary research methodologies. The study’s primary research components will include interviews, surveys, and case studies to supplement existing data. Control groups and counterfactual approaches will be used where feasible and appropriate to isolate the effects of the SBIR program. Other evaluation methods may also be used on a limited basis as needed to address questions not effectively addressed by the principal methods. This is discussed further in Section 7 of this report.

  7. Complete Phase I. Phase I of the NRC study will be formally completed once a set of methodologies is developed and documented, is approved by the Committee, and passes successfully through the Academy’s peer review process.

  8. Implement the research program (NRC Study Phase II). The variety of tasks involved in implementing the research program is previewed in Annex I of this report.

  9. Prepare agency-specific reports. Results from the research program will be presented in five agency-specific reports—one for each of the agencies. Where appropriate, agency-specific findings and recommendations will be formulated by the relevant study subcommittee for review and approval by the full Committee.

  10. Prepare overview report. A separate summary report, buttressed by the relevant commissioned work and bringing together the findings of the individual agency reports, along with general recommendations, will be produced for distribution. This final report will also draw out, as appropriate, the contrasts and similarities among the agencies in the way they administer SBIR. It will follow the approval procedure outlined above.

  11. Organize public meetings to review and discuss findings. Following report review, findings and recommendations will be presented publicly for information, review, and comment.

  12. Submit reports to Congress.

  13. Disseminate findings broadly.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement