Overview of Methodological Approaches, Data Sources, and Survey Tools
This series of reports on the Small Business Innovation Research (SBIR) and the Small Business Technology Transfer (STTR) programs at the Department of Defense (DoD), National Institutes of Health (NIH), National Aeronautics and Space Administration (NASA), Department of Energy (DoE), and National Science Foundation (NSF) represents a second-round assessment of the program undertaken by the National Academies of Sciences, Engineering, and Medicine.1 The first-round assessment, conducted under a separate ad hoc committee, resulted in a series of reports released from 2004 to 2009, including a framework methodology for that study and on which the current methodology builds.2
The current study is focused on the twin objectives of assessing outcomes from the programs and of providing recommendations for improvement.3 Sec-
_______________
1Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council or NRC are used in an historic context identifying programs prior to July 1.
2National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004.
3The methodology developed as part of the first-round assessment of the SBIR program also identifies two areas that are excluded from the purview of the study: “The objective of the study is not to consider if SBIR should exist or not—Congress has already decided affirmatively on this question. Rather, we are charged with providing assessment-based findings of the benefits and costs of SBIR . . . to improve public understanding of the program, as well as recommendations to improve the program’s effectiveness. It is also important to note that, in accordance with the Memorandum of Understanding and the Congressional mandate, the study will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress and the Administration acting through the agencies. Instead, the study is concerned with the effective review of each area.” National Research Council, Assessment of the SBIR Program: Project Methodology, op. cit. In implementing this approach in the context of the current round of SBIR assessments, we have opted to focus more deeply on operational questions.
tion 1c of the Small Business Administration (SBA) SBIR Directive states program objectives as follows:
“The statutory purpose of the SBIR Program is to strengthen the role of innovative small business concerns (SBCs) in federally-funded research or research and development (R/R&D). Specific program purposes are to:
- (1) Stimulate technological innovation;
- (2) use small business to meet federal R/R&D needs;
- (3) foster and encourage participation by socially and economically disadvantaged small businesses (SDBs), and by women-owned small businesses (WOSBs), in technological innovation; and
- (4) increase private sector commercialization of innovations derived from Federal R/R&D, thereby increasing competition, productivity and economic growth.”4
The parallel language from the SBA’s STTR Policy Directive is as follows:
“(c) The statutory purpose of the STTR Program is to stimulate a partnership of ideas and technologies between innovative small business concerns (SBCs) and Research Institutions through Federally-funded research or research and development (R/R&D). By providing awards to SBCs for cooperative R/R&D efforts with Research Institutions, the STTR Program assists the small business and research communities by commercializing innovative technologies.”5
The SBIR/STTR programs, on the basis of highly competitive solicitations, provide modest initial funding for selected Phase I projects (up to $150,000) and for feasibility testing and further Phase II funding (up to $1 million) for qualifying Phase I projects.6
From a methodology perspective, assessing this program presents formidable challenges. Among the more difficult are the following:
- Lack of data. NIH has only limited ability to track outcomes data, both in scope (share of awards tracked) and depth (time tracked after the end of the award). There are no published or publicly available outcomes data.
- Intervening variables. Small innovative businesses can be deflected from a development by a wide range of positive and negative variables. A single breakthrough contract—or technical delay—can make or break a company.
- Lags. Not only do outcomes lag awards by a number of years, but also the lag itself is highly variable. Some companies commercialize within
_______________
4Ibid., 3.
5Small Business Administration, Office of Investment and Innovation, “Small Business Technology Transfer (STTR) Program— Policy Guidance,” updated February 24, 2014.
6These figures reflect standard sizes. NIH and other agencies have the flexibility to adjust the award sizes to accommodate the needs of particular projects and technologies.
6 months of award conclusion; others take decades. And often, revenues from commercialization peak many years after products have reached markets.
ESTABLISHING A METHODOLOGY
The methodology utilized in this second-round study of the SBIR/STTR programs builds on the methodology established by the committee that completed the first-round study.
Publication of the 2004 Methodology
The committee that undertook the first-round study and the agencies under study acknowledged the difficulties involved in assessing SBIR/STTR programs. Accordingly, that study began with development of the formal volume on methodology, which was published in 2004 after undergoing the standard Academies peer-review process.7
The established methodology stressed the importance of adopting a varied range of tools based on prior work in this area, which meshes with the methodology originally defined by the first study committee. The first committee concluded that appropriate methodological approaches—
“build from the precedents established in several key studies already undertaken to evaluate various aspects of the SBIR/STTR. These studies have been successful because they identified the need for utilizing not just a single methodological approach, but rather a broad spectrum of approaches, in order to evaluate the SBIR/STTR from a number of different perspectives and criteria.
This diversity and flexibility in methodological approach are particularly appropriate given the heterogeneity of goals and procedures across the five agencies involved in the evaluation. Consequently, this document suggests a broad framework for methodological approaches that can serve to guide the research team when evaluating each particular agency in terms of the four criteria stated above. Table A-1 illustrates some key assessment parameters and related measures to be considered in this study.”8
The tools identified in Table A-1 include many of those used by the committee conducting the first-round study of the SBIR/STTR programs. Other tools have emerged since the initial methodology review.
_______________
7National Research Council, Assessment of the SBIR Program: Project Methodology, 2.
8National Research Council, Assessment of the SBIR Program: Project Methodology, 2.
TABLE A-1 Overview of Approach to SBIR/STTR Programs Assessment
SBIR/STTR Assessment Parameters → | Quality of Research | Commercialization of SBIR/STTR Funded- Research/ Economic and Non-economic Benefits | Small Business Innovation/Growth | Use of Small Businesses to Advance Agency Missions |
Questions | How does the quality of SBIR/STTR-funded research compare with that of other government funded R&D? | How effectively does SBIR/ STTR support the commercialization of innovative technologies? What non-economic benefits can be identified? | How to broaden participation and expand the base of small innovative firms? | How to increase agency support for commercializable technologies while continuing to support high-risk research |
Measures | Peer-review scores, publication counts, citation analysis | Sales, follow-up funding, other commercial activities | Patent counts and other IP/employment growth, number of new technology firms | Innovative products resulting from SBIR/STTR work |
Tools | Case studies, agency program studies, study of repeat winners, bibliometric analysis | Phase II surveys, program manager discussions, case studies, study of repeat winners | Phase I and Phase II surveys, case studies, study of repeat winners | Program manager surveys, case studies, agency program studies, study of repeat winners |
Key Research Challenges | Difficulty of measuring quality and of identifying proper reference group | Skew of returns; significant interagency and inter-industry differences | Measures of actual success and failure at the project and firm levels; relationship of federal and state programs in this context | Major interagency differences in use of SBIR/STTR to meet agency missions |
NOTE: Supplementary tools may be developed and used as needed. In addition, since publication of the methodology report, this committee has determined that data on outcomes from Phase I awards are of limited relevance.
SOURCE: National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004, Table 1, p. 3. The contents of the table have been adjusted to focus on the specific program at the NIH.
Tools Utilized in the Current SBIR/STTR Study
Quantitative and qualitative tools being utilized in the current study of the SBIR/STTR programs include the following:
- Surveys. An extensive survey of NIH SBIR/STTR award recipients was commissioned as part of the analysis. These are described in depth below.9
- Case studies. In-depth case studies of 20 SBIR/STTR recipients of NIH awards were commissioned. These companies were geographically and demographically diverse and were at different stages of the company lifecycle.
- Workshops. Several workshops were commissioned to allow stakeholders, agency staff, and academic experts to provide insights into the programs’ operations, as well as to identify questions that should be addressed.
- Analysis of agency data. NIH provided the committee with a range of datasets covering various aspects of agency SBIR/STTR activities, which were analyzed and included as appropriate.
- Open-ended responses from SBIR/STTR recipients. For the first time, the survey—the 2014 Survey—solicited textual survey responses. More than 500 recipients provided narrative comments.
- Agency meetings. Discussions about program operations were held with NIH staff members. Most were helpful in providing information both about the program and the challenges that they faced.
- Literature review. Since the start of the committee’s research in this area, a number of papers have been published addressing various aspects of the SBIR/STTR programs. In addition, other organizations—such as the Government Accountability Office (GAO)—have reviewed particular parts of the SBIR/STTR programs. These works were relevant and are referenced in the course of this analysis.
Taken together with committee deliberations and the expertise brought to bear by the individual committee members, these tools provide the primary inputs into the analysis.
For the first-round study and for the current study, multiple research methodologies feed into every finding and recommendation. No findings or recommendations rested solely on data and analysis from the survey; conversely, survey data were used to support analysis throughout the report.
_______________
9The survey conducted as part of the current, second-round assessment of the SBIR/STTR programs is referred to as the “2014 Survey.”
COMMERCIALIZATION METRICS AND DATA COLLECTION
Recent congressional interest in the SBIR/STTR programs has to a considerable extent focused on bringing innovative technologies to market. This enhanced attention to the economic return from public investments made in small business innovation is understandable. However, in contrast to the Department of Defense (DoD), which may procure selected SBIR/STTR technologies, commercialization at NIH takes place almost entirely in private-sector markets.
In its 2009 report on the NIH SBIR/STTR programs,10 the committee charged with the first-round assessment held that a binary metric of commercialization was insufficient. It noted that the scale of commercialization is also important and that there are other important milestones both before and after the first dollar in sales that should be included in an appropriate approach to measuring commercialization.
Challenges in Tracking Commercialization
Despite substantial efforts at NIH, described below, significant challenges remain in tracking commercialization outcomes for the NIH SBIR/STTR programs. These include the following:
- Data limitations. NIH, like most other agencies, has not maintained a comprehensive electronic reporting system for post-award data. It also does not penalize companies for failing to report outcomes. Companies face few incentives to report their successes and failures in commercialization.
- Linear linkages. Tracking efforts usually seek to link a specific project to a specific outcome. Separating the contributions of one project is difficult for many companies, given that multiple projects typically contribute to both anticipated and unanticipated outcomes.
- Lags in commercialization. Data from the extensive DoD commercialization database suggest that most projects take at least 2 years to reach the market after the end of the Phase II award. They do not generate peak revenue for several years after this. Therefore, efforts to measure program productivity must account for these significant lags.
- Attribution problems. Commercialization is often the result of several awards, not just one, as well as other factors, so attributing company-level success to specific awards is challenging at best.
Why New Data Sources Are Needed
Congress often seeks evidence about the effectiveness of programs or indeed about whether they work at all. This interest has in the past helped to drive the
_______________
10National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Washington, DC: The National Academies Press, 2009.
development of tools such as the Company Commercialization Report (CCR) at DoD, which captures the quantitative commercialization results of companies’ prior Phase II projects. However, in the long-term the importance of tracking may rest more in its use to support program management. By carefully analyzing outcomes and CCR’s associated program variables, program managers will be able to manage their SBIR/STTR portfolios more successfully.
In this regard, the NIH SBIR/STTR programs can benefit from access to the survey data. The survey work provides quantitative data necessary to provide an evidence-driven assessment and, at the same time, allows management to focus on specific questions of interest. For example, at NIH the survey was tuned to include additional information on company efforts to meet U.S. Food and Drug Administration (FDA) requirements for clinical trials prior to commercialization.
SURVEY ANALYSIS
Traditional modes of assessing the NIH SBIR/STTR programs include case studies, interviews with program staff, review of documents, and other qualitative methods of assessment. These remain important components of the overall methodology, and a chapter in the current report is devoted to lessons drawn from case studies. However, qualitative assessment alone is insufficient.
2014 Survey
The 2014 Survey offers some significant advantages over other data sources. Specifically, it—
- provides a rich source of textual information in response to openended questions;
- probes more deeply into company demographics and agency processes;
- for the first time addresses principal investigators (PIs), not just company business officials;
- allows comparisons with previous datacollection exercises;
- generates the first SBIR/STTRrelated data on clinical trials; and
- addresses other Congressional objectives for the program beyond commercialization.
For these and other reasons, a survey was determined to be the most appropriate mechanism for developing quantitative approaches to the analysis of the SBIR/STTR programs. At the same time, however, there are limitations of survey research in this case. Box A-1 describes a number of areas where caution is required when reviewing results.
To take account of these limits, while retaining the utility and indeed explanatory power of survey-based methodology, the current report contextualizes
BOX A-1 Multiple Sources of Bias in Survey Responsea
Large innovation surveys involve multiple sources of potential bias that can skew the results in different directions. Some potential survey biases are noted below.
- Successful and more recently funded companies are more likely to respond. Research by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and increases the greater the award amount.b About 60 percent of respondents to the 2014 Survey received NIH awards during fiscal years (FY) 2006-2010. Winners from more distant years are difficult to reach: small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR/STTR awards. While there is evidence of bias for project-performance count variables such as the number of publications or patents associated with a publicly-subsidized project, there is also evidence that there may not be a response bias for commercialization measures.c
- Non-respondent bias. Very limited information is available about SBIR/STTR award recipients: company name, location, and contact information for the PI and the company point of contact, agency name, and date of award (data on woman and minority ownership are not considered reliable). No detailed data are available on applicants who did not win awards. It is therefore not feasible to undertake detailed analysis of non-respondents, but the possibility exists that they would present a different profile than would respondents.
- Success is self-reported. Self-reporting can be a source of bias, although the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospective analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third-party observation.d In short, company founders and/or PIs are in many cases simply the best source of information available.
- Survey sampled projects from PIs with multiple awards. Projects from PIs with large numbers of awards were underrepresented in the sample, because PIs could not be expected to complete a questionnaire for each of numerous awards over a 10-year time frame.
- Failed companies are difficult to contact. Survey experts point to an “asymmetry” in the survey’s ability to include failed companies for follow-up surveys in cases where the companies no longer exist.e It is worth noting that one cannot necessarily infer that the SBIR/STTR project failed; what is known is only that the company no longer exists.
- Not all successful projects are captured. For similar reasons, the survey could not include ongoing results from successful projects in companies that merged or were acquired before and/or after commercialization of the project’s technology. This is the outcome for many successful companies in this sector.
- Some companies may not accurately acknowledge SBIR/STTR contribution to project success. Some companies may be unwilling to acknowledge that
-
they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts. Other companies may overstate the importance of SBIR/STTR.
- Commercialization lags. Although the 2005 Survey broke new ground in data collection, the amount of sales made—and indeed the number of projects that generated sales—are inevitably undercounted in a snapshot survey taken at a single point in time. On the basis of successive data sets collected from NIH SBIR/STTR award recipients, it is clear that total sales from all responding projects will be considerably greater than can be captured in a single survey.f This underscores the importance of follow-on research based on the now-established survey methodology. Figure Box A-1 illustrates this impact in practice at DoD: projects from FY2006 onward have not yet completed commercialization as of August 2013.
FIGURE BOX A-1 The impact of commercialization lag.
SOURCE: DoD Company Commercialization Database.
_______________
a The limitations described here are drawn from the methodology outlined for the previous survey in National Research Council, An Assessment of the SBIR Program at the Department of Defense, Washington, DC: The National Academies Press, 2009.
b Albert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005.
c See, for example, Dora Gicheva and Albert N.Link, “Leveraging Entrepreneurship through
Private Investments: Does Gender Matter?,” Small Business Economics, 40:199-210, 2013 (finding that the probability of securing outside finance do not find any response bias); Alfred N. Link and John T. Scott, “Private Investor Participation and Commercialization Rates for Government-sponsored Research and Development: Would a Prediction Market Improve the Performance of the SBIR Programme?” Economica, 76:264-281, 2009 (finding that there is no response bias in the estimates for the probability of commercialization).
d Although economic theory is formulated on what is called “revealed preferences,” meaning that individuals and companies reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information, especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or company reveals its preferences and that all relevant markets for such preferences are operational. See Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments,” Administrative Science Quarterly, 29:52-73, 1984; Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions Norwell, MA: Kluwer Academic Publishers, 1998.
e Link and Scott, Evaluating Public Research Institutions.
f Data from the National Research Council assessment of the SBIR program at NIH indicate that a subsequent survey taken 2 years later would reveal substantial increases in both the percentage of companies reaching the market and the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Washington, DC: The National Academies Press, 2009.
the data by comparing results to those from the survey conducted as part of the first-round assessment of the SBIR/STTR programs (referred to below as the “2005 Survey”). This report also adds transparency by publishing the number of responses for each question and indeed each subgroup. As noted later in the discussion, the use of a control group was found infeasible for comparing Phase II and Phase I recipients, but feasible for comparing Phase IIB and Phase II recipients.
Grunwald Associates LLC was contracted to administer a survey to award recipients. The Academies’ 2014 Survey is built closely on the 2005 Survey, but it is also adapted to draw on lessons learned and includes some important changes discussed in detail below. A subgroup of this committee with particular expertise in survey methodology also reviewed the survey and incorporated current best practices. The 2014 survey covered NIH and the Department of Energy (DoE) simultaneously.11
_______________
11Delays at NIH and DoE in contracting with the Academies, combined with the need to complete work contracted with DoD, National Science Foundation (NSF), and the National Aeronautics and Space Administration (NASA) led us to proceed with the survey at the remaining three agencies first, in 2011, followed by the NIH-DoE survey in 2014.
The primary objectives of the 2014 Survey (in combination with the 2010 Phase IIB Survey) were as follows:
- Provide an update of the program “snapshot” taken in 2005, maximizing the opportunity to identify trends within the program;
- Probe more deeply into program processes, with the help of expanded feedback from participants and better understanding of program demographics; and
- Reduce costs and shrink the time required by combining three 2005 questionnaires—for the company, Phase I, and Phase II awards, respectively—into a single 2014 Survey questionnaire.
The survey was therefore designed to collect the maximum amount of data, consistent with the commitment to minimizing the burden on individual respondents.
In light of these competing considerations, the decision was made to administer the survey to PIs—the lead researcher on each project—rather than to the registered company point of contact (POC), who in many cases would be an administrator rather than a researcher. This decision was reinforced by difficulties in accessing current POC information. Key areas of overlap between the 2005 and 2014 surveys are captured in Table A-2.
Starting Date and Coverage
The 2014 Survey included awards made from FY2001 to FY2010 inclusive. This end date allowed for completion of Phase II-awarded projects (which nominally fund 2 years of research) and provided a further 2 years for commercialization. This time frame was consistent with the previous survey, administered in 2005, which surveyed awards from FY1992 to FY2001. It was also consistent with a previous GAO study, which in 1991 surveyed awards made through 1987.
The aim in setting the overall time frame at 10 years was to reduce the impact of difficulties in generating information about older awards, because some companies and PIs may no longer be in place and memories fade over time.
Determining the Survey Population
Following the precedent set by both the original GAO study and the first round of Academies analysis, we differentiate between the total population of SBIR/STTR recipients, the preliminary survey target population, and the effective population for this study, which is the population of respondents that were reachable.
The effective survey population is the denominator for the survey, used to determine response rates.
TABLE A-2 Similarities and Differences: 2005 and 2014 Surveys
Item | 2005 Survey | 2014 Survey |
Respondent selection | ||
Focus on Phase II winners | ||
All qualifying awards | ||
PIs | ||
POCs | ||
Max number of questionnaires per respondent | <20 | 2 |
Distribution | ||
No | ||
Telephone follow-up | ||
Questionnaire | ||
Company demographicsa | Identical | Identical |
Commercialization outcomes | Identical | Identical |
IP outcomes | Identical | Identical |
Women and minority participation | ||
Additional detail on minorities | ||
Additional detail on PIs | ||
New section on agency staff activities | ||
Information about technological categories | ||
New section on company recommendations for SBIR/STTR | ||
New section capturing open-ended responses | ||
a While company demographics in the two surveys appear to be identical, we note that the information collected about companies in the 2014 survey is not directly comparable to the surveys used in 2005. In addition, information about the company’s age was not included in the 2014 survey, but, as pointed out in reviewer comments, should be included in future evaluations of SBIR.
Initial Filters for Potential Recipients
Determining the effective study population required the following steps:
- acquisition of data from the sponsoring agencies—NIH and DoE—covering record-level lists of award recipients;
- elimination of records that did not fit the protocol agreed upon by the committee—namely, a maximum of two questionnaires per PI (in cases
-
where PIs received more than two awards), awards were selected first by program (STTR, then SBIR), then by agency (DoE and NIH, in that order), then by year (oldest first), and finally by random number; and
- elimination of records for which there were significant missing data—in particular, where emails and/or contact telephone numbers were absent.
This process of excluding awards either because they did not fit the selection profile approved by the committee or because the agencies did not provide sufficient or current contact information reduced the total award list provided by NIH from 3,851 awards to a preliminary survey population of 3,375 awards.
Secondary Filters to Identify Recipients with Active Contact Information
This nominal population still included many potential respondents whose contact information was complete but who were no longer associated with the contact information provided and hence effectively unreachable. This is unsurprising given that small businesses experience considerable turnover in personnel and that the survey reaches back to awards made in FY2001. Recipients may have switched companies, the company may have ceased to exist or been acquired, or telephone and email contacts may have changed, for example. Consequently, we utilized two further filters to help identify the effective survey population.
- First, contacts for which the email address bounced twice were eliminated. Because the survey was delivered via email, the absence of a working email address disqualified the recipient. This eliminated approximately 30 percent of the preliminary population.
- Second, email addresses that did not officially “bounce” (i.e., return to sender) may still in fact not be active. Some email systems are configured to delete unrecognized email without sending a reply; in other cases, email addresses are inactive but not deleted. So a non-bouncing email address did not equal a contactable PI.
Deployment
The 2014 Survey opened on December 3, 2014, and was deployed by email, with voice follow-up support. Up to four emails were sent to the effective population (emails discontinued once responses were received). In addition, two voice mails were delivered to non-respondents between the second and third and between the third and fourth rounds of email. In total, up to six efforts were made to reach each questionnaire recipient. After the members of the data subgroup of our committee concluded that sufficient data for the purposes had been collected, the survey closed on April 7, 2015. It was open for a total of 18 weeks.
Response Rates
Standard procedures were followed to conduct the survey. These data collection procedures were designed to increase response to the extent possible within the constraints of a voluntary survey and the survey budget. The population surveyed is a difficult one to contact and obtain responses from as evidence from the literature shows. Under these circumstances, the inability to contact and obtain responses always raises questions about potential bias of the estimates that cannot be quantified without substantial extra efforts that would require resources beyond those available for this work.
The lack of detailed applications data from the agency makes it impossible to estimate the possible impact of non-response bias. We, therefore, have no evidence either that non-response bias exists or that it does not.
Table A-3 shows the response rates at NIH by phase, based on both the preliminary study population prior to adjustment and the effective study population after all adjustments.
All subsequent references to the 2014 Survey in this report address only responses for awards made by NIH.
Effort at Comparison Group Analysis
Several readers of the reports in the first-round analysis of the SBIR/STTR programs suggested the inclusion of comparison groups in the analysis. We concurred that this should be attempted. There is no simple and easy way to acquire a comparison group for Phase II SBIR/STTR awardees. These are technology-based companies at an early stage of company development, which have the demonstrated capacity to undertake challenging technical research and to provide evidence that they are potentially successful commercializers. Given that the operations of the SBIR/STTR programs are defined in legislation and limited by
TABLE A-3 2014 Survey Response Rates at NIH
Preliminary population | 3,375 |
Not contactable | 1,723 |
Effective population | 1,652 |
Responses | 726 |
Surveys as Percentage of Awards Contacted | 43.9 |
Surveys as Percentage of Sample | 21.5 |
SOURCE: 2014 Survey.
TABLE A-4 SBIR/STTR Responses by Year of Award (Percent Distribution)
NIH Total | SBIR | STTR | PHASE IIB | |
Fiscal Year of Award | TOTAL | — | — | — |
2001 | 7.5 | 8.2 | 3.6 | |
2002 | 9.1 | 9.3 | 8.1 | |
2003 | 7.8 | 9.1 | 0.9 | |
2004 | 6.1 | 6.6 | 3.6 | |
2005 | 9.3 | 8.4 | 13.5 | 10.3 |
2006 | 10.8 | 11.5 | 7.2 | 24.1 |
2007 | 10.5 | 10.2 | 11.7 | 20.7 |
2008 | 11.7 | 10.9 | 15.3 | 20.7 |
2009 | 11.4 | 10.8 | 14.4 | 13.8 |
2010 | 16 | 14.9 | 21.6 | 10.3 |
BASE: ALL RESPONDENTS | 669 | 558 | 111 | 29 |
SOURCE: 2014 Survey.
the Small Business Administration (SBA) Policy Guidance, randomly assigned control groups were not a possible alternative. Efforts to identify a pool of SBIR/STTR-like companies were made by contacting the most likely sources—Dunn and Bradstreet and Hoovers—but these efforts were not successful, because sufficiently detailed and structured information about companies was not available.
In response, we sought to develop a comparison group from among Phase I awardees that had not received a Phase II award from the three surveyed agencies (DoD, NSF, and NASA) during the award period covered by the survey (FY20012010). After considerable review, however, the committee concluded that the Phase I-only group was not appropriate for use as a statistical comparison group.
NIH Responses and Respondents
Table A-4 shows NIH SBIR/STTR responses by year of award. The survey primarily reached companies that were still in business: overall, 94 percent of respondents indicated that the companies were still in business.12
_______________
122011 Survey, Question 4A.