National Academies Press: OpenBook

SBIR/STTR at the National Institutes of Health (2015)

Chapter: Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools

« Previous: Appendixes
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

Appendix A

Overview of Methodological Approaches, Data Sources, and Survey Tools

This series of reports on the Small Business Innovation Research (SBIR) and the Small Business Technology Transfer (STTR) programs at the Department of Defense (DoD), National Institutes of Health (NIH), National Aeronautics and Space Administration (NASA), Department of Energy (DoE), and National Science Foundation (NSF) represents a second-round assessment of the program undertaken by the National Academies of Sciences, Engineering, and Medicine.1 The first-round assessment, conducted under a separate ad hoc committee, resulted in a series of reports released from 2004 to 2009, including a framework methodology for that study and on which the current methodology builds.2

The current study is focused on the twin objectives of assessing outcomes from the programs and of providing recommendations for improvement.3 Sec-

_______________

1Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council or NRC are used in an historic context identifying programs prior to July 1.

2National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004.

3The methodology developed as part of the first-round assessment of the SBIR program also identifies two areas that are excluded from the purview of the study: “The objective of the study is not to consider if SBIR should exist or not—Congress has already decided affirmatively on this question. Rather, we are charged with providing assessment-based findings of the benefits and costs of SBIR . . . to improve public understanding of the program, as well as recommendations to improve the program’s effectiveness. It is also important to note that, in accordance with the Memorandum of Understanding and the Congressional mandate, the study will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress and the Administration acting through the agencies. Instead, the study is concerned with the effective review of each area.” National Research Council, Assessment of the SBIR Program: Project Methodology, op. cit. In implementing this approach in the context of the current round of SBIR assessments, we have opted to focus more deeply on operational questions.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

tion 1c of the Small Business Administration (SBA) SBIR Directive states program objectives as follows:

“The statutory purpose of the SBIR Program is to strengthen the role of innovative small business concerns (SBCs) in federally-funded research or research and development (R/R&D). Specific program purposes are to:

  • (1) Stimulate technological innovation;
  • (2) use small business to meet federal R/R&D needs;
  • (3) foster and encourage participation by socially and economically disadvantaged small businesses (SDBs), and by women-owned small businesses (WOSBs), in technological innovation; and
  • (4) increase private sector commercialization of innovations derived from Federal R/R&D, thereby increasing competition, productivity and economic growth.”4

The parallel language from the SBA’s STTR Policy Directive is as follows:

“(c) The statutory purpose of the STTR Program is to stimulate a partnership of ideas and technologies between innovative small business concerns (SBCs) and Research Institutions through Federally-funded research or research and development (R/R&D). By providing awards to SBCs for cooperative R/R&D efforts with Research Institutions, the STTR Program assists the small business and research communities by commercializing innovative technologies.”5

The SBIR/STTR programs, on the basis of highly competitive solicitations, provide modest initial funding for selected Phase I projects (up to $150,000) and for feasibility testing and further Phase II funding (up to $1 million) for qualifying Phase I projects.6

From a methodology perspective, assessing this program presents formidable challenges. Among the more difficult are the following:

  • Lack of data. NIH has only limited ability to track outcomes data, both in scope (share of awards tracked) and depth (time tracked after the end of the award). There are no published or publicly available outcomes data.
  • Intervening variables. Small innovative businesses can be deflected from a development by a wide range of positive and negative variables. A single breakthrough contract—or technical delay—can make or break a company.
  • Lags. Not only do outcomes lag awards by a number of years, but also the lag itself is highly variable. Some companies commercialize within

_______________

4Ibid., 3.

5Small Business Administration, Office of Investment and Innovation, “Small Business Technology Transfer (STTR) Program— Policy Guidance,” updated February 24, 2014.

6These figures reflect standard sizes. NIH and other agencies have the flexibility to adjust the award sizes to accommodate the needs of particular projects and technologies.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

6 months of award conclusion; others take decades. And often, revenues from commercialization peak many years after products have reached markets.

ESTABLISHING A METHODOLOGY

The methodology utilized in this second-round study of the SBIR/STTR programs builds on the methodology established by the committee that completed the first-round study.

Publication of the 2004 Methodology

The committee that undertook the first-round study and the agencies under study acknowledged the difficulties involved in assessing SBIR/STTR programs. Accordingly, that study began with development of the formal volume on methodology, which was published in 2004 after undergoing the standard Academies peer-review process.7

The established methodology stressed the importance of adopting a varied range of tools based on prior work in this area, which meshes with the methodology originally defined by the first study committee. The first committee concluded that appropriate methodological approaches—

“build from the precedents established in several key studies already undertaken to evaluate various aspects of the SBIR/STTR. These studies have been successful because they identified the need for utilizing not just a single methodological approach, but rather a broad spectrum of approaches, in order to evaluate the SBIR/STTR from a number of different perspectives and criteria.

This diversity and flexibility in methodological approach are particularly appropriate given the heterogeneity of goals and procedures across the five agencies involved in the evaluation. Consequently, this document suggests a broad framework for methodological approaches that can serve to guide the research team when evaluating each particular agency in terms of the four criteria stated above. Table A-1 illustrates some key assessment parameters and related measures to be considered in this study.”8

The tools identified in Table A-1 include many of those used by the committee conducting the first-round study of the SBIR/STTR programs. Other tools have emerged since the initial methodology review.

_______________

7National Research Council, Assessment of the SBIR Program: Project Methodology, 2.

8National Research Council, Assessment of the SBIR Program: Project Methodology, 2.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

TABLE A-1 Overview of Approach to SBIR/STTR Programs Assessment

SBIR/STTR Assessment Parameters → Quality of Research Commercialization of SBIR/STTR Funded- Research/ Economic and Non-economic Benefits Small Business Innovation/Growth Use of Small Businesses to Advance Agency Missions
Questions How does the quality of SBIR/STTR-funded research compare with that of other government funded R&D? How effectively does SBIR/ STTR support the commercialization of innovative technologies? What non-economic benefits can be identified? How to broaden participation and expand the base of small innovative firms? How to increase agency support for commercializable technologies while continuing to support high-risk research
 
Measures Peer-review scores, publication counts, citation analysis Sales, follow-up funding, other commercial activities Patent counts and other IP/employment growth, number of new technology firms Innovative products resulting from SBIR/STTR work
 
Tools Case studies, agency program studies, study of repeat winners, bibliometric analysis Phase II surveys, program manager discussions, case studies, study of repeat winners Phase I and Phase II surveys, case studies, study of repeat winners Program manager surveys, case studies, agency program studies, study of repeat winners
 
Key Research Challenges Difficulty of measuring quality and of identifying proper reference group Skew of returns; significant interagency and inter-industry differences Measures of actual success and failure at the project and firm levels; relationship of federal and state programs in this context Major interagency differences in use of SBIR/STTR to meet agency missions

NOTE: Supplementary tools may be developed and used as needed. In addition, since publication of the methodology report, this committee has determined that data on outcomes from Phase I awards are of limited relevance.
SOURCE: National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004, Table 1, p. 3. The contents of the table have been adjusted to focus on the specific program at the NIH.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

Tools Utilized in the Current SBIR/STTR Study

Quantitative and qualitative tools being utilized in the current study of the SBIR/STTR programs include the following:

  • Surveys. An extensive survey of NIH SBIR/STTR award recipients was commissioned as part of the analysis. These are described in depth below.9
  • Case studies. In-depth case studies of 20 SBIR/STTR recipients of NIH awards were commissioned. These companies were geographically and demographically diverse and were at different stages of the company lifecycle.
  • Workshops. Several workshops were commissioned to allow stakeholders, agency staff, and academic experts to provide insights into the programs’ operations, as well as to identify questions that should be addressed.
  • Analysis of agency data. NIH provided the committee with a range of datasets covering various aspects of agency SBIR/STTR activities, which were analyzed and included as appropriate.
  • Open-ended responses from SBIR/STTR recipients. For the first time, the survey—the 2014 Survey—solicited textual survey responses. More than 500 recipients provided narrative comments.
  • Agency meetings. Discussions about program operations were held with NIH staff members. Most were helpful in providing information both about the program and the challenges that they faced.
  • Literature review. Since the start of the committee’s research in this area, a number of papers have been published addressing various aspects of the SBIR/STTR programs. In addition, other organizations—such as the Government Accountability Office (GAO)—have reviewed particular parts of the SBIR/STTR programs. These works were relevant and are referenced in the course of this analysis.

Taken together with committee deliberations and the expertise brought to bear by the individual committee members, these tools provide the primary inputs into the analysis.

For the first-round study and for the current study, multiple research methodologies feed into every finding and recommendation. No findings or recommendations rested solely on data and analysis from the survey; conversely, survey data were used to support analysis throughout the report.

_______________

9The survey conducted as part of the current, second-round assessment of the SBIR/STTR programs is referred to as the “2014 Survey.”

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

COMMERCIALIZATION METRICS AND DATA COLLECTION

Recent congressional interest in the SBIR/STTR programs has to a considerable extent focused on bringing innovative technologies to market. This enhanced attention to the economic return from public investments made in small business innovation is understandable. However, in contrast to the Department of Defense (DoD), which may procure selected SBIR/STTR technologies, commercialization at NIH takes place almost entirely in private-sector markets.

In its 2009 report on the NIH SBIR/STTR programs,10 the committee charged with the first-round assessment held that a binary metric of commercialization was insufficient. It noted that the scale of commercialization is also important and that there are other important milestones both before and after the first dollar in sales that should be included in an appropriate approach to measuring commercialization.

Challenges in Tracking Commercialization

Despite substantial efforts at NIH, described below, significant challenges remain in tracking commercialization outcomes for the NIH SBIR/STTR programs. These include the following:

  • Data limitations. NIH, like most other agencies, has not maintained a comprehensive electronic reporting system for post-award data. It also does not penalize companies for failing to report outcomes. Companies face few incentives to report their successes and failures in commercialization.
  • Linear linkages. Tracking efforts usually seek to link a specific project to a specific outcome. Separating the contributions of one project is difficult for many companies, given that multiple projects typically contribute to both anticipated and unanticipated outcomes.
  • Lags in commercialization. Data from the extensive DoD commercialization database suggest that most projects take at least 2 years to reach the market after the end of the Phase II award. They do not generate peak revenue for several years after this. Therefore, efforts to measure program productivity must account for these significant lags.
  • Attribution problems. Commercialization is often the result of several awards, not just one, as well as other factors, so attributing company-level success to specific awards is challenging at best.

Why New Data Sources Are Needed

Congress often seeks evidence about the effectiveness of programs or indeed about whether they work at all. This interest has in the past helped to drive the

_______________

10National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Washington, DC: The National Academies Press, 2009.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

development of tools such as the Company Commercialization Report (CCR) at DoD, which captures the quantitative commercialization results of companies’ prior Phase II projects. However, in the long-term the importance of tracking may rest more in its use to support program management. By carefully analyzing outcomes and CCR’s associated program variables, program managers will be able to manage their SBIR/STTR portfolios more successfully.

In this regard, the NIH SBIR/STTR programs can benefit from access to the survey data. The survey work provides quantitative data necessary to provide an evidence-driven assessment and, at the same time, allows management to focus on specific questions of interest. For example, at NIH the survey was tuned to include additional information on company efforts to meet U.S. Food and Drug Administration (FDA) requirements for clinical trials prior to commercialization.

SURVEY ANALYSIS

Traditional modes of assessing the NIH SBIR/STTR programs include case studies, interviews with program staff, review of documents, and other qualitative methods of assessment. These remain important components of the overall methodology, and a chapter in the current report is devoted to lessons drawn from case studies. However, qualitative assessment alone is insufficient.

2014 Survey

The 2014 Survey offers some significant advantages over other data sources. Specifically, it—

  • provides a rich source of textual information in response to openended questions;
  • probes more deeply into company demographics and agency processes;
  • for the first time addresses principal investigators (PIs), not just company business officials;
  • allows comparisons with previous datacollection exercises;
  • generates the first SBIR/STTRrelated data on clinical trials; and
  • addresses other Congressional objectives for the program beyond commercialization.

For these and other reasons, a survey was determined to be the most appropriate mechanism for developing quantitative approaches to the analysis of the SBIR/STTR programs. At the same time, however, there are limitations of survey research in this case. Box A-1 describes a number of areas where caution is required when reviewing results.

To take account of these limits, while retaining the utility and indeed explanatory power of survey-based methodology, the current report contextualizes

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

BOX A-1 Multiple Sources of Bias in Survey Responsea

Large innovation surveys involve multiple sources of potential bias that can skew the results in different directions. Some potential survey biases are noted below.

  • Successful and more recently funded companies are more likely to respond. Research by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and increases the greater the award amount.b About 60 percent of respondents to the 2014 Survey received NIH awards during fiscal years (FY) 2006-2010. Winners from more distant years are difficult to reach: small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR/STTR awards. While there is evidence of bias for project-performance count variables such as the number of publications or patents associated with a publicly-subsidized project, there is also evidence that there may not be a response bias for commercialization measures.c
  • Non-respondent bias. Very limited information is available about SBIR/STTR award recipients: company name, location, and contact information for the PI and the company point of contact, agency name, and date of award (data on woman and minority ownership are not considered reliable). No detailed data are available on applicants who did not win awards. It is therefore not feasible to undertake detailed analysis of non-respondents, but the possibility exists that they would present a different profile than would respondents.
  • Success is self-reported. Self-reporting can be a source of bias, although the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospective analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third-party observation.d In short, company founders and/or PIs are in many cases simply the best source of information available.
  • Survey sampled projects from PIs with multiple awards. Projects from PIs with large numbers of awards were underrepresented in the sample, because PIs could not be expected to complete a questionnaire for each of numerous awards over a 10-year time frame.
  • Failed companies are difficult to contact. Survey experts point to an “asymmetry” in the survey’s ability to include failed companies for follow-up surveys in cases where the companies no longer exist.e It is worth noting that one cannot necessarily infer that the SBIR/STTR project failed; what is known is only that the company no longer exists.
  • Not all successful projects are captured. For similar reasons, the survey could not include ongoing results from successful projects in companies that merged or were acquired before and/or after commercialization of the project’s technology. This is the outcome for many successful companies in this sector.
  • Some companies may not accurately acknowledge SBIR/STTR contribution to project success. Some companies may be unwilling to acknowledge that
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
  • they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts. Other companies may overstate the importance of SBIR/STTR.

  • Commercialization lags. Although the 2005 Survey broke new ground in data collection, the amount of sales made—and indeed the number of projects that generated sales—are inevitably undercounted in a snapshot survey taken at a single point in time. On the basis of successive data sets collected from NIH SBIR/STTR award recipients, it is clear that total sales from all responding projects will be considerably greater than can be captured in a single survey.f This underscores the importance of follow-on research based on the now-established survey methodology. Figure Box A-1 illustrates this impact in practice at DoD: projects from FY2006 onward have not yet completed commercialization as of August 2013.

images

FIGURE BOX A-1 The impact of commercialization lag.
SOURCE: DoD Company Commercialization Database.

_______________

a The limitations described here are drawn from the methodology outlined for the previous survey in National Research Council, An Assessment of the SBIR Program at the Department of Defense, Washington, DC: The National Academies Press, 2009.

b Albert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005.

c See, for example, Dora Gicheva and Albert N.Link, “Leveraging Entrepreneurship through

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

Private Investments: Does Gender Matter?,” Small Business Economics, 40:199-210, 2013 (finding that the probability of securing outside finance do not find any response bias); Alfred N. Link and John T. Scott, “Private Investor Participation and Commercialization Rates for Government-sponsored Research and Development: Would a Prediction Market Improve the Performance of the SBIR Programme?” Economica, 76:264-281, 2009 (finding that there is no response bias in the estimates for the probability of commercialization).

d Although economic theory is formulated on what is called “revealed preferences,” meaning that individuals and companies reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information, especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or company reveals its preferences and that all relevant markets for such preferences are operational. See Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments,” Administrative Science Quarterly, 29:52-73, 1984; Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions Norwell, MA: Kluwer Academic Publishers, 1998.

e Link and Scott, Evaluating Public Research Institutions.

f Data from the National Research Council assessment of the SBIR program at NIH indicate that a subsequent survey taken 2 years later would reveal substantial increases in both the percentage of companies reaching the market and the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Washington, DC: The National Academies Press, 2009.

the data by comparing results to those from the survey conducted as part of the first-round assessment of the SBIR/STTR programs (referred to below as the “2005 Survey”). This report also adds transparency by publishing the number of responses for each question and indeed each subgroup. As noted later in the discussion, the use of a control group was found infeasible for comparing Phase II and Phase I recipients, but feasible for comparing Phase IIB and Phase II recipients.

Grunwald Associates LLC was contracted to administer a survey to award recipients. The Academies’ 2014 Survey is built closely on the 2005 Survey, but it is also adapted to draw on lessons learned and includes some important changes discussed in detail below. A subgroup of this committee with particular expertise in survey methodology also reviewed the survey and incorporated current best practices. The 2014 survey covered NIH and the Department of Energy (DoE) simultaneously.11

_______________

11Delays at NIH and DoE in contracting with the Academies, combined with the need to complete work contracted with DoD, National Science Foundation (NSF), and the National Aeronautics and Space Administration (NASA) led us to proceed with the survey at the remaining three agencies first, in 2011, followed by the NIH-DoE survey in 2014.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

The primary objectives of the 2014 Survey (in combination with the 2010 Phase IIB Survey) were as follows:

  • Provide an update of the program “snapshot” taken in 2005, maximizing the opportunity to identify trends within the program;
  • Probe more deeply into program processes, with the help of expanded feedback from participants and better understanding of program demographics; and
  • Reduce costs and shrink the time required by combining three 2005 questionnaires—for the company, Phase I, and Phase II awards, respectively—into a single 2014 Survey questionnaire.

The survey was therefore designed to collect the maximum amount of data, consistent with the commitment to minimizing the burden on individual respondents.

In light of these competing considerations, the decision was made to administer the survey to PIs—the lead researcher on each project—rather than to the registered company point of contact (POC), who in many cases would be an administrator rather than a researcher. This decision was reinforced by difficulties in accessing current POC information. Key areas of overlap between the 2005 and 2014 surveys are captured in Table A-2.

Starting Date and Coverage

The 2014 Survey included awards made from FY2001 to FY2010 inclusive. This end date allowed for completion of Phase II-awarded projects (which nominally fund 2 years of research) and provided a further 2 years for commercialization. This time frame was consistent with the previous survey, administered in 2005, which surveyed awards from FY1992 to FY2001. It was also consistent with a previous GAO study, which in 1991 surveyed awards made through 1987.

The aim in setting the overall time frame at 10 years was to reduce the impact of difficulties in generating information about older awards, because some companies and PIs may no longer be in place and memories fade over time.

Determining the Survey Population

Following the precedent set by both the original GAO study and the first round of Academies analysis, we differentiate between the total population of SBIR/STTR recipients, the preliminary survey target population, and the effective population for this study, which is the population of respondents that were reachable.

The effective survey population is the denominator for the survey, used to determine response rates.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

TABLE A-2 Similarities and Differences: 2005 and 2014 Surveys

Item 2005 Survey 2014 Survey
Respondent selection
Focus on Phase II winners images images
All qualifying awards images
PIs images
POCs images
Max number of questionnaires per respondent <20 2
Distribution
Mail images No
Email images images
Telephone follow-up images images
Questionnaire
Company demographicsa Identical Identical
Commercialization outcomes Identical Identical
IP outcomes Identical Identical
Women and minority participation images images
Additional detail on minorities images
Additional detail on PIs images
New section on agency staff activities images
Information about technological categories images
New section on company recommendations for SBIR/STTR images
New section capturing open-ended responses images

a While company demographics in the two surveys appear to be identical, we note that the information collected about companies in the 2014 survey is not directly comparable to the surveys used in 2005. In addition, information about the company’s age was not included in the 2014 survey, but, as pointed out in reviewer comments, should be included in future evaluations of SBIR.

Initial Filters for Potential Recipients

Determining the effective study population required the following steps:

  • acquisition of data from the sponsoring agencies—NIH and DoE—covering record-level lists of award recipients;
  • elimination of records that did not fit the protocol agreed upon by the committee—namely, a maximum of two questionnaires per PI (in cases
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
  • where PIs received more than two awards), awards were selected first by program (STTR, then SBIR), then by agency (DoE and NIH, in that order), then by year (oldest first), and finally by random number; and

  • elimination of records for which there were significant missing data—in particular, where emails and/or contact telephone numbers were absent.

This process of excluding awards either because they did not fit the selection profile approved by the committee or because the agencies did not provide sufficient or current contact information reduced the total award list provided by NIH from 3,851 awards to a preliminary survey population of 3,375 awards.

Secondary Filters to Identify Recipients with Active Contact Information

This nominal population still included many potential respondents whose contact information was complete but who were no longer associated with the contact information provided and hence effectively unreachable. This is unsurprising given that small businesses experience considerable turnover in personnel and that the survey reaches back to awards made in FY2001. Recipients may have switched companies, the company may have ceased to exist or been acquired, or telephone and email contacts may have changed, for example. Consequently, we utilized two further filters to help identify the effective survey population.

  • First, contacts for which the email address bounced twice were eliminated. Because the survey was delivered via email, the absence of a working email address disqualified the recipient. This eliminated approximately 30 percent of the preliminary population.
  • Second, email addresses that did not officially “bounce” (i.e., return to sender) may still in fact not be active. Some email systems are configured to delete unrecognized email without sending a reply; in other cases, email addresses are inactive but not deleted. So a non-bouncing email address did not equal a contactable PI.

Deployment

The 2014 Survey opened on December 3, 2014, and was deployed by email, with voice follow-up support. Up to four emails were sent to the effective population (emails discontinued once responses were received). In addition, two voice mails were delivered to non-respondents between the second and third and between the third and fourth rounds of email. In total, up to six efforts were made to reach each questionnaire recipient. After the members of the data subgroup of our committee concluded that sufficient data for the purposes had been collected, the survey closed on April 7, 2015. It was open for a total of 18 weeks.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

Response Rates

Standard procedures were followed to conduct the survey. These data collection procedures were designed to increase response to the extent possible within the constraints of a voluntary survey and the survey budget. The population surveyed is a difficult one to contact and obtain responses from as evidence from the literature shows. Under these circumstances, the inability to contact and obtain responses always raises questions about potential bias of the estimates that cannot be quantified without substantial extra efforts that would require resources beyond those available for this work.

The lack of detailed applications data from the agency makes it impossible to estimate the possible impact of non-response bias. We, therefore, have no evidence either that non-response bias exists or that it does not.

Table A-3 shows the response rates at NIH by phase, based on both the preliminary study population prior to adjustment and the effective study population after all adjustments.

All subsequent references to the 2014 Survey in this report address only responses for awards made by NIH.

Effort at Comparison Group Analysis

Several readers of the reports in the first-round analysis of the SBIR/STTR programs suggested the inclusion of comparison groups in the analysis. We concurred that this should be attempted. There is no simple and easy way to acquire a comparison group for Phase II SBIR/STTR awardees. These are technology-based companies at an early stage of company development, which have the demonstrated capacity to undertake challenging technical research and to provide evidence that they are potentially successful commercializers. Given that the operations of the SBIR/STTR programs are defined in legislation and limited by

TABLE A-3 2014 Survey Response Rates at NIH

Preliminary population 3,375
Not contactable 1,723
Effective population 1,652
Responses 726
Surveys as Percentage of Awards Contacted 43.9
Surveys as Percentage of Sample 21.5

SOURCE: 2014 Survey.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×

TABLE A-4 SBIR/STTR Responses by Year of Award (Percent Distribution)

NIH Total SBIR STTR PHASE IIB
Fiscal Year of Award TOTAL
2001   7.5   8.2   3.6
2002   9.1   9.3   8.1
2003   7.8   9.1   0.9
2004   6.1   6.6   3.6
2005   9.3   8.4 13.5 10.3
2006 10.8 11.5   7.2 24.1
2007 10.5 10.2 11.7 20.7
2008 11.7 10.9 15.3 20.7
2009 11.4 10.8 14.4 13.8
2010 16 14.9 21.6 10.3
BASE: ALL RESPONDENTS 669 558 111 29

SOURCE: 2014 Survey.

the Small Business Administration (SBA) Policy Guidance, randomly assigned control groups were not a possible alternative. Efforts to identify a pool of SBIR/STTR-like companies were made by contacting the most likely sources—Dunn and Bradstreet and Hoovers—but these efforts were not successful, because sufficiently detailed and structured information about companies was not available.

In response, we sought to develop a comparison group from among Phase I awardees that had not received a Phase II award from the three surveyed agencies (DoD, NSF, and NASA) during the award period covered by the survey (FY20012010). After considerable review, however, the committee concluded that the Phase I-only group was not appropriate for use as a statistical comparison group.

NIH Responses and Respondents

Table A-4 shows NIH SBIR/STTR responses by year of award. The survey primarily reached companies that were still in business: overall, 94 percent of respondents indicated that the companies were still in business.12

_______________

122011 Survey, Question 4A.

Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 275
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 276
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 277
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 278
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 279
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 280
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 281
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 282
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 283
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 284
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 285
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 286
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 287
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 288
Suggested Citation:"Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools." National Academies of Sciences, Engineering, and Medicine. 2015. SBIR/STTR at the National Institutes of Health. Washington, DC: The National Academies Press. doi: 10.17226/21811.
×
Page 289
Next: Appendix B: Major Changes to the SBIR Program Resulting from the 2011 SBIR Reauthorization Act, P.L. 112-81, December 2011 »
SBIR/STTR at the National Institutes of Health Get This Book
×
Buy Paperback | $85.00 Buy Ebook | $69.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs provide federal research and development funding to small businesses. In 2008, the National Research Council completed a comprehensive assessment of the SBIR and STTR programs. The first-round study found that the programs were "sound in concept and effective in practice." Building on the outcomes from the Phase I study, this second phase examines both topics of general policy interest that emerged during the first phase and topics of specific interest to individual agencies, and provides a second snapshot to measure the program's progress against its legislative goals.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!