|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. Appendix D
|
||||||||||||||||
 |
 |
Page 198 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 198
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
Appendix D
NRC Project Manager Survey
David H. Finifter
The College of William and Mary
February 7, 2007
INTRODUCTION
Federal agency R&D programs are increasingly assessed in terms of their quality, relevance and efficiency. One means of gauging the performance of the SBIR program in light of these criteria is to obtain assessments by agency project managers who manage contracted SBIR programs. The survey findings reported here represent the first systematic effort to obtain such data from SBIR project managers. The survey questions were constructed to correspond to the three criteria above. Also, in an effort to address the ever present assessment question of “compared to what,” the survey was designed to obtain respondent assessments of a comparison set of agency externally funded R&D projects.
Program performance of the SBIR program is very complex. Therefore, it is important to get as many perspectives on the SBIR program performance as possible. One group of participants in the process is agency project managers who manage contracted SBIR projects. This role only exists at agencies that contract SBIR awards (e.g., Department of Defense-DoD, Department of Energy-DoE, and the National Aeronautics and Space Administration-NASA) as opposed to administering awards as grants (e.g., National Institutes of Health-NIH and the National Science Foundation-NSF). Project managers take on different names and roles at the three agencies in question. At DoD, they are called Technical Points of Contact (TPOCs); at DoE, they are called Technical Project Managers (TPMs); and at NASA they are called Contract Officer’s Technical Representatives (COTRs).
This paper gives two perspectives of project managers based primarily on an electronic survey that was administered as part of the National Research Council (NRC) assessment of the SBIR program. First, the report examines the role of
OCR for page 199
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
the project manager in the SBIR project from topic selection to Phase III funding. Second, the report documents how project managers at these three agencies view the relative quality, usefulness and value to the agency of SBIR projects in comparison to other research at the agency.
Using the same time period of the NRC Phase II Survey of projects and firms completed as part of the NRC study as the reference point (1992-2001), we generated a list of Phase II SBIR projects for the three contracting agencies. We then requested from the SBIR program offices a list of project managers names and email addresses for as many of these individuals as possible. Naturally, there was significant attrition (absence of email addresses, absence of name for project manager, project manager having left the agency and/or was deceased, simply an error in the identified project manager). Given the constraints on collection of contact information and the time since many of the identified projects were implemented, the sample is a critical minimum size. A survey instrument was developed (based on Archibald-Finifter, 20001) to determine quality and usefulness of research and mission benefits of the SBIR Phase II projects as seen by the project managers and is provided as Annex A of this paper.
General Background
The sample of project managers was first based on the original database provided by Peter Cahill of BRTRC of 11,684 SBIR Phase II projects in the five SBIR agencies (the three listed above plus NSF and NIH, which award grants, not contracts, and hence do not have contract officers as such). For the sample of the three agencies in question (DoD, DoE, and NASA), there is an estimated n=7,945 based on the BRTRC sample since 68 percent of the total sample was for these three agencies). The 7,945 represents an estimate of the potential number of project managers that could receive surveys. The number is actually far less than that because many project managers have done more that one Phase II project and therefore received a modified questionnaire for multiple projects. In a similar study by Archibald-Finifter for the DoD Fast Track study, 51.5 percent of the full sample responded and 78.9 percent of the successful contacts responded.2 These were all recent relative to the survey data so the response rate was fairly high.
For the current survey, the actual sample of project managers/projects with known names was—5,650 for DoD, 1,488 for NASA, and 808 for DoE. These projects represent the requests for names and email addresses sent to the three agencies. In response, the agencies were able to locate names and email addresses
1
Robert B. Archibald and David H. Finifter, “Evaluation of the Department of Defense Small Business Innovation Research Program and Fast Track Initiative: A Balanced Approach,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Charles W. Wessner, ed., National Academy Press, Washington, DC, 2000.
2
Ibid.
OCR for page 200
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
for a limited subset of project managers—1,757 for DoD, 555 for DoE, and 272 for NASA. Finally, the actual responses we received for the survey amounted to a total of 513 (19.8 percent rate) including—347 (19.7 percent rate) for DoD, 84 (15.1 percent rate) for DoE, and 82 (30.1 percent rate) for NASA. The NASA sample was based on projects since 1997 only (the only project managers for which they had email information). The higher response rate is probably due in part to the fact that the sample is based on more recent projects. We were thus more likely to have respondents who were still available and who were more likely to have recalled the SBIR project about which they were being asked.
THE ROLE OF THE PROJECT MANAGER
Project Manager’s Role in Selection
Question 2 asked if the Project Manager was involved in generating the topic that led to this particular SBIR project.
In Question 2 of the questionnaire, the respondent was asked if he or she was involved in generating the topic that led to this particular Phase II award. Table App-D-1 provides the response to this question for the total sample of project managers and results broken down by agency. Nearly 70 percent of the total respondents were involved in generating the SBIR project. This was apparently the case for DoD and DoE. However, NASA COTR respondents were less likely to have been involved in generating the topic that led to the SBIR project (only 57 percent).
Question 3 asked the Project Manager when he or she became involved in the particular SBIR project.
Another dimension of the role of project manager in the SBIR process relates to when he or she became involved in the project. As seen in Table App-D-2, for the total sample of project managers, 23 percent became involved after Phase I but before Phase II. Another 10 percent became involved after Phase II started but before Phase II was completed. Nearly two percent became involved after Phase II was completed. Over 65 percent were involved before Phase I started
TABLE App-D-1 Involvement in Generating Topic
Involved in Generating the Topic that Led to this Particular SBIR Project?
Total Sample
DoD
DoE
NASA
Yes
357
(69.59%)
251
(72.33%)
59
(70.24%)
47
(57.32%)
No
156
(30.41%)
96
(27.67%)
25
(29.76%)
35
(42.68%)
TOTAL
513
347
84
82
OCR for page 201
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-2 When Project Manager Became Involved in Project?
When Involved in SBIR Project?
Total Sample
DoD
DoE
NASA
After Phase I, Before Phase II
118
(23%)
56
(16.14%)
25
(29.76%)
37
(45.12%)
After Phase II Started, Before Phase II Completed
53
(10.33%)
32
(9.22%)
10
(11.90%)
11
(13.41%)
After Phase II Completed
8
(1.56%)
8
(2.31%)
—
—
—
—
Before Phase I
334
(65.11%)
251
(72.33%)
49
(58.33%)
34
(41.46%)
TOTAL
513
347
84
82
(consistent with the previous question relating to involvement in project generation). This varied by agency as seen in Table App-D-2.
For example, around 45 percent of the NASA respondents became involved in the project after Phase I, but before Phase II. Also, for DoD respondents, over 72 percent became involved before Phase I compared to 58 percent for DoE and 41 percent for NASA respondents. The degree to which a project manager is attached to the SBIR project by helping to come up with the topic to timing of involvement will have a likely impact on the knowledge and “ownership” of the project and may affect the respondents’ view of the quality and usefulness of the project. This is analyzed further below.
Actual Role in Project
Question 10. What has your role been with respect to this SBIR project? List as many as apply.
The actual role of the project manager may vary from project-to-project, project manager-to-project manager (perhaps depending on skills and expertise of the project manager), and agency-to-agency. Question 10 of the survey asked about the project managers’ role(s) with respect to the particular SBIR project in question. The results are summarized in Table App-D-3. The respondents were to respond to as many of the roles as applied. For the total sample, over 96 percent
TABLE App-D-3 Role of Project Manager with Respect to This SBIR Project
Role of Project Manager
Total Sample
DoD
DoE
NASA
Technical
493
(96.10%)
331
(95.39%)
82
(97.62%)
80
(97.56%)
Financial
109
(21.25%)
86
(24.78%)
19
(22.62%)
4
(4.88%)
Commercialization Assistance
84
(16.37%)
68
(19.60%)
6
(7.14%)
10
(12.19%)
Other
55
(10.72%)
44
(12.68%)
6
(7.14%)
5
(6.10%)
TOTAL SAMPLE SIZE
513
347
84
82
NOTE: Multiple responses permitted.
OCR for page 202
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-4 Played a Role in Any Phase III Funding for This Project
Who Played a Role in Any Phase III Funding for this Project
Total Sample
DoD
DoE
NASA
Respondent Played that Role
60
(11.72%)
50
(14.45%)
4
(4.76%)
6
(7.32%)
Someone Else Has Played that Role
77
(15.04%)
59
(17.05%)
12
(14.29%)
6
(7.32%)
Unknown
375
(73.24%)
237
(68.50%)
68
(80.95%)
70
(85.37%)
TOTAL
512
346
84
82
of the respondents claimed a “technical” role, with around 21 percent claiming a “financial” role and around 16 percent claiming a “commercialization assistance” role. The “commercialization assistance” role was a bit higher for those in DoD (around 20 percent) and lower for DoE and NASA (only around 7 and 12 percent, respectively). Clearly, the “technical” role dominates in all agencies. It is notable that for NASA, only around 5 percent had a “financial” role.
Project Manager’s Role in Phase III Funding
Question 12. Have you or others played a role in any Phase III funding for this project?
The survey examined the role of the project manager in Phase III funding for the various projects listed (Question 12). Table App-D-4 summarizes the findings about the role. The overwhelming finding is that 73 percent of the respondents did not know who played such a role in Phase III funding efforts. The result was slightly lower for DoD respondents (68 percent) and higher for DoE and NASA (81 and 85 percent, respectively). Nearly 12 percent of the respondents in the overall sample said they played that role, ranging from 14 percent for DoD, to 7 percent for NASA, to 5 percent for DoE respondents. Overall, project managers seem to have a relatively minor role in Phase III funding for the agencies and projects studied. This is an important finding since project managers would be knowledgeable natural advocates for the project within the agency.
THE QUALITY, USEFULNESS, AND VALUE TO THE AGENCY OF THE SBIR PROJECT AS SEEN BY THE PROJECT MANAGER
As seen above, project managers tend to be fairly involved with the SBIR Phase II projects to which they are assigned, although the degree varies somewhat by agency. This involvement gives them a unique basis for evaluating the quality, usefulness, and value to the agency of each project. We used the survey of project managers to develop measures of these dimensions of program outcomes.
OCR for page 203
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-5 Ratings for SBIR Projects
Measure of Quality of SBIR Project
Total Sample
DoD
DoE
NASA
Mean Score
6.93
6.95
6.80
6.98
Standard Deviation
2.072
2.202
1.706
1.846
Median Score
7
8
7
8
Sample Size
513
347
84
82
Project Quality
Question 4. On a 1 to 10 scale, where 10 represents the best research ever produced in your research unit/office or for your research unit/office and 1 represents the worst research ever produced in your research unit/office or for your research unit/office, rate the quality of the research in this particular SBIR contract.
First, we use Questions 4 and 5 to arrive at a rating of the quality of SBIR Phase II projects from our sample as seen by the project managers. Question 4 asks the respondent to, on a scale of 1 to 10, where 10 represents the best research ever produced in their research unit/office or for their research unit/office and 1 represents the worst research ever produced in their research unit/office or for their research unit/office, rate the quality of the particular SBIR contract listed. The results for Question 4 are given in Table App-D-5 for the total sample of project managers and by each agency. The mean score for the total sample of 513 project managers/projects was 6.93 (with a standard deviation of 2.072) and the median score was 7. There were similar scores by agency as seen in Table App-D-5.
Question 5. On the same scale rate the average quality of the research projects conducted for your research unit/office from contracts other than SBIR contracts for the last two years.
The problem with this measure is that it does not adjust for “toughness of the grader.” To compensate for the fact that different project managers have different standards in mind when evaluating a research project, we asked Question 5—On the same scale (1 to 10) rate the average quality of the research projects conducted for their research unit/office from contracts other than SBIR contracts for the last two years. A summary of those scores is given in Table App-D-6. It shows for example that the mean score for non-SBIR projects was 7.29 (standard deviation of 1.594) and a median of 8.
A comparison of the scores for the SBIR projects and the average quality of relevant non-SBIR contracts allows us to gauge the relative quality of the SBIR projects. The differences in means between the average score for the sample of SBIR projects and the average non-SBIR project are given in Table App-D-7. For the total sample of SBIR projects, the SBIR projects were on average lower by
OCR for page 204
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-6 Ratings for Non-SBIR Research Projects
Measure of Average Quality of Research (non-SBIR)
Total Sample
DoD
DoE
NASA
Mean Score
7.29
7.27
7.24
7.45
Standard Deviation
1.594
1.635
1.712
1.268
Median Score
8
8
7
7
Sample Size
513
347
84
82
TABLE App-D-7 Mean Difference in Scores SBIR Quality Minus Average Non-SBIR Project Quality
Measure of Difference in Scores of Project Quality
Total Sample
DoD
DoE
NASA
Mean Difference in Score—SBIR Quality Minus Average Non-SBIR Project Quality
−.364*
−.320*
−.440**
−.476**
Standard Deviation
2.15
2.19
2.03
2.10
NOTES:
*Statistically significant at the .01 level;
**Statistically significant at the .05 level.
.364. Similar differences are shown for the DoD, DoE, and NASA subsamples. Using a t-test, these differences are statistically significantly different from zero. The average quality score is slightly lower for SBIR projects than for non-SBIR research projects (although the differences are statistically significant). This result differs from the result found in Archibald-Finifter3 which shows no statistical difference between the two scores for a sample of DoD projects.
Further examination of the results of the current survey shed some light on the slightly lower rating for SBIR projects. In looking at the distribution of scores for Question 3 (SBIR scores) and Question 4 (non-SBIR scores), we find that 7.4 percent of the respondents rank SBIR projects at a score of 3 or below compared to 1.9 percent for non-SBIR projects. The remainder of the distribution is much more similar. For example, around 49 percent of the SBIR sample ranked score above seven, compared to around 52 percent for the non-SBIR sample. This same skewed pattern of a few very low scores held for the three subsamples as well. Thus, the result seeming to favor non-SBIR research quality over SBIR quality is driven by a few outliers scoring the SBIR projects extremely low. The small difference in means and other evidence tends to show SBIR in a reasonably favorable light in terms of its quality and other dimensions of usefulness compared to non-SBIR research projects at the same agency.
3
Ibid.
OCR for page 205
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-8 Distribution of Scores: SBIR Quality and Average Non-SBIR Project Quality (Total Sample)
Score of Project Quality
Quality of Research (SBIR)
Average Quality of Research (Non-SBIR)
Number of Responses
Percent
Number of Responses
Percent
1
8
1.56
7
1.36
2
12
2.34
2
0.39
3
18
3.51
1
0.19
4
32
6.24
11
2.14
5
56
10.92
50
9.75
6
48
9.36
50
9.75
7
90
17.54
126
24.56
8
129
25.15
162
31.58
9
89
17.35
84
16.37
10
31
6.04
20
3.90
TOTAL
513
100.00
513
100.00
Linkage to Research Mission
Question 6. Has the research conducted for this SBIR contract affected the way that your research unit/office conducts research or the type of research your research unit/office obtains in other contracts? (List as many as apply.)
The next question we raised in the survey dealt more with usefulness of the research for the research unit/office. In particular, we asked Question 6—Has the
TABLE App-D-9 Distribution of Scores: SBIR Quality and Average Non-SBIR Project Quality (DoD)
Score of Project Quality
Quality of Research (SBIR)
Average Quality of Research (Non-SBIR)
Number of Responses
Percent
Number of Responses
Percent
1
8
2.31
7
2.02
2
9
2.59
0
0.00
3
13
3.75
1
0.29
4
22
6.34
7
2.02
5
38
10.95
35
10.09
6
26
7.49
30
8.65
7
53
15.27
88
25.36
8
85
24.50
109
31.41
9
68
19.60
58
16.71
10
25
7.20
12
3.46
TOTAL
347
100.00
347
100.00
OCR for page 206
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-10 Distribution of Scores: SBIR Quality and Average Non-SBIR Project Quality (DoE)
Score of Project Quality
Quality of Research (SBIR)
Average Quality of Research (Non-SBIR)
Number of Responses
Percent
Number of Responses
Percent
1
0
0.00
0
0.00
2
1
1.19
2
2.38
3
2
2.38
0
0.00
4
6
7.14
3
3.57
5
11
13.10
8
9.52
6
12
14.29
11
13.10
7
17
20.24
19
22.62
8
24
28.57
22
26.19
9
9
10.71
13
15.48
10
2
2.38
6
7.14
TOTAL
84
100.00
84
100.00
research conducted for this SBIR contract affected the way that your research unit/office conducts research or the type of research your research unit/office obtains in other contracts? (List as many as apply.) These results are summarized in Table App-D-12.
The responses to this question varied across agencies somewhat. Of those project managers who responded “No, this project was a separate project, and the knowledge generated by this SBIR contract has had no impact on the other
TABLE App-D-11 Distribution of Scores: SBIR Quality and Average Non-SBIR Project Quality (NASA)
Score of Project Quality
Quality of Research (SBIR)
Average Quality of Research (Non-SBIR)
Number of Responses
Percent
Number of Responses
Percent
1
0
0.00
0
0.00
2
2
2.44
0
0.00
3
3
3.66
0
0.00
4
4
4.88
1
1.22
5
7
8.54
7
8.54
6
10
12.20
9
10.98
7
20
24.39
19
23.17
8
20
24.39
31
37.80
9
12
14.63
13
15.85
10
4
4.88
2
2.44
TOTAL
82
100.00
82
100.00
OCR for page 207
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-12 Effect of SBIR Project’s Research on Your Research Unit
Research Effect
Total Sample
DoD
DoE
NASA
No, this project was a separate project, and the knowledge generated by this SBIR contract has had no impact on the other research we conduct or sponsor.
166 (32.4%)
88 (25.4%)
42 (50.0%)
36 (43.9%)
Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract by encouraging the firm to apply for additional SBIR awards.
148 (28.8%)
108 (31.1%)
19 (22.6%)
21 (25.6%)
Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract in other research we conduct or sponsor.
234 (45.6%)
184 (53.0%)
24 (28.6%)
26 (31.7%)
Yes, but this project found a blind alley, so we have not followed up on this line of inquiry.
49 (9.5%)
35 (10.1%)
6 (7.1%)
8 (9.8%)
TOTAL SAMPLE
513
347
84
82
NOTE: Multiple responses permitted.
research we conduct or sponsor,” 32 percent of the total sample gave this response of no linkage. However, the response was only 25 percent from DoD project managers, 50 percent from DoE project managers, and 44 percent from NASA project managers. The low percentage response for DoD is indicates relatively high linkage and usefulness and this is probably not a surprise for an agency that is so mission driven.
“Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract by encouraging the firm to apply for additional SBIR awards” yielded some positive response. For the total sample, nearly 29 percent cited this response. The breakdown by agency was not dramatic—percentages around 31, 23, and 26, respectively for DoD, DoE, and NASA.
The response category “Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract in other research we conduct or sponsor” led to some interesting results. The overall sample showed a response of around 46 percent. That ranged from the high of 53 percent for DoD to a low of around 29 and 32 percent for DoE and
OCR for page 208
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
NASA, respectively. This may reflect the greater size and scope within DoD, or it might suggest a greater effort by DoD in finding alternative uses for the research within the agency.
In general, the results indicate that there is a relatively high agency usage rate or attempted usage rate for SBIR projects in our sample. In some cases this lead to additional SBIR awards and in some cases it influences other non-SBIR research. In any case, potential impact is relatively high for all three agencies.
One other response on this question is notable. “Yes, but this project found a blind alley, so we have not followed up on this line of inquiry” had a response rate of 9.5 percent for the total sample (around 10, 7, and 10 percent for DoD, DoE, and NASA samples, respectively). Having a 10 percent “blind alley” rate is relatively small and can be viewed as “good forecasting” by the agencies. Alternatively, we would expect agencies to take some risks with the SBIR selections and perhaps the 10 percent rate of “blind alleys” is too low for the goals of the program. This result merits further study.
Comparative Value of SBIR Projects
Question 8. In comparison to a dollar spent in your research unit/office on other R&D projects, how did a dollar spent on this SBIR project rank?
Information on another dimension of quality and usefulness of the SBIR program was sought through Question 8—“In comparison to a dollar spent in your research unit/office on other R&D projects, how did a dollar spent on this SBIR project rank” is shown in Table App-D-13. For the total sample around 28 percent responded that the particular SBIR project had fewer benefits for their agency’s mission than the average dollar spend on other contracts they sponsor. Another 28 percent responded that the SBIR project had more benefits SBIR
TABLE App-D-13 Comparative Dollar Value of Projects
Dollar Value
Total Sample
DoD
DoE
NASA
SBIR project had fewer benefits for your agency’s mission than the average dollar spent on other contracts you sponsor
145
(28.27%)
93
(26.80%)
26
(30.95%)
26
(31.71%)
SBIR project had more benefits for your agency’s mission than the average dollar spent on other research contracts you sponsor
143
(27.88%)
114
(32.85%)
14
(16.67%)
15
(18.29%)
Same Benefits
225
(43.86%)
140
(40.35%)
44
(52.38%)
41
(50.00%)
TOTAL SAMPLE SIZE
513
347
84
82
OCR for page 209
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
project had more benefits for their agency’s mission than the average dollar spent on other research contracts they sponsor. This response varied by agency quite a bit—The response from around 33 percent for DoD respondents, but only around 17 to 18 percent for DoE and NASA, respectively. The project managers who responded the same benefits amounted to around 44 percent for the total sample, ranging from around 40 percent for DoD respondents to 52 and 50 percent for DoE and NASA respondents respectively. Overall the conclusion is that over 71 percent of the SBIR projects had at least as high benefits for the mission of the agency as the average dollar spent on non-SBIR projects.
Abundance of Fundable SBIR Proposals
Another dimension of SBIR project and program success was approached by survey Question 9. This question asked: “In general do you find that your research unit/office has had more good SBIR proposals than you can fund?” This question gets at the value of the marginal SBIR project and the consequences of expanding the funded SBIR projects. The results are impressive. As seen in Table App-D-14, while around 24 percent of the total sample of respondents claimed that there were about the right number of SBIR proposals currently, over 63 percent responded that there were more fundable SBIR proposals than they can fund, with only 13 percent claiming that there are fewer fundable proposal than they can fund. This result is fairly consistent for DoD and DoE respondents. NASA results are somewhat less favorable with around 21 percent responding that there are fewer fundable proposals than they can fund. However, over 56 percent still responded that there are more fundable projects than they can fund.
Analysis of “Ownership Bias” of Findings
As a check on bias of responses by the project managers, we broke the sample up into project managers with a potentially strong degree of “ownership” in the project and those with less potential for “ownership.” We defined
TABLE App-D-14 Relative Number of Fundable SBIR Projects
Relative Number of Fundable SBIR Projects
Total Sample
DoD
DoE
NASA
More fundable proposals than can fund
220
(63.4%)
156
(65.8%)
24
(61.5%)
40
(56.3%)
About the right number of proposals
82
(23.6%)
54
(22.8%)
12
(30.8%)
16
(22.5%)
Fewer fundable proposals than can fund
45
(13.0%)
27
(11.4%)
3
(7.7%)
15
(21.1%)
Respondents
347
237
39
71
OCR for page 210
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-15 Analysis of Ownership Effects—Entire Sample
Area of Interest
Ownership Group (n=391)
Remaining Project Managers (n=122)
Research Quality (Difference between Q3-Q4)
−.261
−.697
Usefulness of Research (Q5)
a. No, not useful
113 (28.90%)
53 (43.44%)
b. Yes, more SBIR
116 (29.67%)
32 (26.23%)
c. Yes, general follow-up
189 (48.34%)
45 (36.88%)
d. Yes, but blind alley
46 (11.76%)
3 (2.46%)
Mission Benefits (Q6)
a. More than average
29.67%
22.13%
b. Same as average
44.50%
41.80%
c. Less than average
25.83%
36.07%
“ownership” as those project managers who had a potential stake in the project as demonstrated either by being involved in defining the topic or if they were involved with the firm before the Phase I proposal. These project managers might be inclined to rate the project more highly and bias the results toward more favorable outcomes. At the same time, these same project managers could also be more knowledgeable about the project and its outcomes. There were 391 of the 513 who were in this “ownership group.” Table App-D-15 through Table App-D-18 summarize the key measures of quality and usefulness of the projects by ownership group membership for the total sample and by agency. In general, the ownership group does rate the SBIR projects more highly in terms of quality and usefulness measures. Again, whether this is bias or greater knowledge or some combination is not known, but there do seem to be differences.
Project Outcomes
Question 13. Has this SBIR project been commercialized?
It is sometimes argued that there is a trade-off between the research potential (intrinsic use) of an SBIR project and its potential for commercialization.4 Table App-D-19 shows the rate of commercialization. This commercialization rate is around 34 percent for the total sample and varies slightly across agency (around 35 percent for DoD, 30 percent for DoE, and 35 percent for NASA). These rates reflect the knowledge base of the project managers.
4
Robert Archibald and David Finifter, “Evaluating the NASA Small Business Innovation Research Program: Preliminary Evidence of a Tradeoff Between Commercialization and Basic Research,” Research Policy 32:605-619, 2003.
OCR for page 211
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-16 Analysis of Ownership Effects—DoD
Area of Interest
Ownership Group (n=271)
Remaining Project Managers (n=76)
Research Quality (Difference between Q3-Q4)
−.229
–.645
Usefulness of Research (Q5)
a. No, not useful
63 (23.25%)
25 (32.89%)
b. Yes, more SBIR
86 (31.73%)
22 (28.95%)
c. Yes, general follow-up
148 (54.61%)
36 (47.37%)
d. Yes, but blind alley
33 (12.18%)
2 (2.63%)
Mission Benefits (Q6)
a. More than average
35.06%
25.00%
b. Same as average
40.96%
38.16%
c. Less than average
23.99%
36.84%
TABLE App-D-17 Analysis of Ownership Effects—DoE
Area of Interest
Ownership Group (n=65)
Remaining Project Managers (n=19)
Research Quality (Difference between Q3-Q4)
−.415
−.526
Usefulness of Research (Q5)
a. No, not useful
34 (52.31%)
8 (42.10%)
b. Yes, more SBIR14
(21.54%)
5 (26.32%)
c. Yes, general follow-up
18 (27.69%)
6 (31.58%)
d. Yes, but blind alley
5 (7.69%)
1 (5.26%)
Mission Benefits (Q6)
a. More than average
15.38%
21.05%
b. Same as average
52.31%
52.63%
c. Less than average
32.31%
26.32%
TABLE App-D-18 Analysis of Ownership Effects—NASA
Area of Interest
Ownership Group (n=55)
Remaining Project Managers (n=27)
Research Quality (Difference between Q3-Q4)
−.236
−.963
Usefulness of Research (Q5)
a. No, not useful
16 (29.09%)
20 (74.07%)
b. Yes, more SBIR
16 (29.09%)
5 (18.52%)
c. Yes, general follow-up
23 (41.82%)
3 (11.11%)
d. Yes, but blind alley
8 (14.54%)
0 (0.00%)
Mission Benefits (Q6)
a. More than average
20.00%
14.81%
b. Same as average
52.73%
44.44%
c. Less than average
27.27%
40.74%
OCR for page 212
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-19 Project Commercialized?
Has this SBIR Project Been Commercialized?
Total Sample
DoD
DoE
NASA
Yes
175 (34.45%)
123 (35.55%)
24 (29.63%)
28 (34.57%)
No
333 (65.55%)
223 (64.45%)
57 (70.37%)
53 (65.43%)
TOTAL
508
346
81
81
Question 14. Does this SBIR project have noncommercial/intrinsic use (perhaps in research)? (Elaborate—see annex).
Table App-D-20 shows how respondents evaluated the noncommercial/intrinsic use (perhaps research) for these same SBIR projects. The intrinsic use rate for the total sample is around 66 percent. This rate varies somewhat across agencies (69 percent for DoD, 54 percent for DoE, and 63 percent for NASA).
Table App-D-21 shows the relationship between whether the projects in our survey were commercialized and/or their intrinsic research potential. Table App-D-22 through Table App-D-24 show the same results by agency. Overall, the results show that SBIR projects are highly successful in producing either commercial and/or noncommercial/intrinsic use outcomes.
The results from Table App-D-21 are compelling. Based on the responses of the project managers, nearly 74 percent of the SBIR projects showed a commercial or intrinsic use or both. In fact, over 26 percent (134 projects) were reported to have both a commercial and an intrinsic use. Thus, while there might be some trade-off between research and commercial potential, there may also be some degree of complementarity. The results differ somewhat by agency and these are shown in Table App-D-22 through Table App-D-24.
How Project Managers Relate to Phase III
One final dimension of outcome of an SBIR project that was covered by the project manager survey relates to Phase III funding. Question 11 probed this issue as follows: “This SBIR project received Phase III funding from your agency in
TABLE App-D-20 Does SBIR Project Have Non-commercial/Intrinsic Use (Perhaps in Research)?
Does SBIR Project Have Non-commercial/Intrinsic Use (Perhaps in Research)?
Total Sample
DoD
DoE
NASA
Yes
336 (65.75%)
239 (69.08%)
45 (54.22%)
52 (63.41%)
No
175 (34.25%)
107 (30.92%)
38 (45.78%)
30 (36.59%)
TOTAL
511
346
83
82
OCR for page 213
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-21 Relationship Between Commercial and Non-commercial Outcomes (Total Sample)
Commercialized
Intrinsic Use (Research)
No
Yes
Total
No
132
200
332 (65.48%)
Yes
41
134
175 (34.52%)
Total
173 (34.12%)
334 (65.88%)
507
TABLE App-D-22 Relationship Between Commercial and Non-commercial Outcomes (DoD Sample)
Commercialized
Intrinsic Use (Research)
No
Yes
Total
No
80
142
222
(64.35%)
Yes
26
97
123
(35.65%)
Total
106
(30.72%)
239
(69.28%)
345
TABLE App-D-23 Relationship Between Commercial and Non-commercial Outcomes (DoE Sample)
Commercialized
Intrinsic Use (Research)
No
Yes
Total
No
33
24
57
(70.37%)
Yes
5
19
24
(29.63%)
Total
38
(46.91%)
43
(53.09%)
81
TABLE App-D-24 Relationship Between Commercial and Non-commercial Outcomes (NASA Sample)
Commercialized
Intrinsic Use (Research)
No
Yes
Total
No
19
34
53
(65.43%)
Yes
10
18
28
(34.57%)
Total
29
(35.80%)
52
(64.20%)
81
the form of: Further non-SBIR R&D funding; Direct procurement of the product of this SBIR Procurement through incorporation of the result of this project into a system; No Phase III from agency; Unknown.” Results of this question are shown in Table App-D-25. The result which stands out the most is that for the total sample, around 58 percent of the projects were known to have not received
OCR for page 214
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
TABLE App-D-25 SBIR Project Received Phase III Funding from Your Agency
Phase III Funding from Your Agency—Form
Total Sample
DoD
DoE
NASA
Direct Procurement of the Product of this SBIR
15
(2.9%)
10
(2.9%)
0
(0.0%)
5
(6.1%)
Procurement through Incorporation of the Result of this Project into System
18
(3.5%)
11
(3.2%)
4
(4.8%)
3
(3.7%)
No Phase III from Agency
299
(58.3%)
214
(61.7%)
39
(46.4%)
46
(56.1%)
Further non-SBIR R&D funding
53
(10.3%)
41
(11.8%)
7
(8.3%)
5
(6.1%)
Unknown
128
(24.9%)
71
(20.5%)
34
(40.5%)
23
(28.0%)
TOTAL
513
347
84
82
Phase III funding from the agency and another 25 percent were unknown. As seen in the table, the results vary somewhat by agency. Phase III direct procurement of the product of the SBIR project occurred in only 2.9 percent of the time for the total sample. That same percentage was the case for DoD respondents, but the results for the other two agencies varied greatly from zero cases for DoE to just above six percent for NASA. Procurement of the SBIR project through incorporation of the result of the project into the system was also relatively low, with a rate of 3.5 percent for the total sample with a range from 3.2 to 4.8 percent across the agencies. For the total sample, there were over ten percent of the respondents who said the projects were supported further by non-SBIR R&D funding.
CONCLUSIONS
The SBIR project managers in our sample appear to be a rather engaged group with respect to the SBIR Program. They were engaged in the projects early and often. In general, the project managers ranked the quality of the SBIR research as close to the quality of research undertaken at their respective research units. The projects seemed to be useful to the mission at the various agencies and affected the subsequent research program (both SBIR and non-SBIR projects.) The project managers valued the projects highly compared to non-SBIR projects. Most agreed that there is a relative abundance of fundable SBIR proposals that do not get funded. Surprisingly, SBIR project managers seem to have little involvement in what is already a relatively low rate of Phase III activity. Finally, as viewed by the project managers, the SBIR projects under examination had a very high combined commercialization and intrinsic (research) use rate.
OCR for page 215
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
REMAINING SECTIONS OF THE REPORT
The remaining parts of the report are in annexes. Annex A contains the base survey. Annex B contains three parts that are responses to the open-ended questions in the survey.
Annex A
The Base Questionnaire
Please name your research unit/office.
Were you involved in defining or generating the topic, which led to this particular SBIR project?
Yes
No
When did you first become involved with this SBIR Phase II project?
Before the Phase I proposal
After the Phase I proposal but before the Phase II proposal
After the Phase II proposal, but before the Phase II project was completed
After the Phase II project was completed
On a 1 to 10 scale, where 10 represents the best research ever produced in your research unit/office or for your research unit/office and 1 represents the worst research ever produced in your research unit/office or for your research unit/office, rate the quality of the research in this particular SBIR contract.
1 2 3 4 5 6 7 8 9 10
On the same scale rate the average quality of the research projects conducted for your research unit/office from contracts other than SBIR contracts for the last two years.
1 2 3 4 5 6 7 8 9 10
Has the research conducted for this SBIR contract affected the way that your research unit/office conducts research or the type of research your research unit/office obtains in other contracts? List as many as apply.
No, this project was a separate project, and the knowledge generated by this SBIR contract has had no impact on the other research we conduct or sponsor.
Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract by encouraging the firm to apply for additional SBIR awards.
Yes, this project produced results that have been useful to us, and we
OCR for page 216
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
have tried to follow up on the ideas initiated in this SBIR contract in other research we conduct or sponsor.
Yes, but this project found a blind alley, so we have not followed up on this line of inquiry.
Please describe how this SBIR project may have had an impact on your agency’s mission. (Optional)
In comparison to a dollar spent in your research unit/office on other R&D projects, did a dollar spent on this SBIR project:
Yield more benefits for your agency’s mission that the average dollar spent on other contracts sponsored by your research unit/office.
Yield the same level of benefits from your agency’s mission as the average dollar spent on other contracts sponsored by your research unit/office.
Yield fewer benefits for your agency’s mission than that average dollar spent on other contracts sponsored by your research unit/office.
In general do you find that your research unit/office has had more good SBIR proposals than you can fund?
Yes
No
About the right number of good proposals
What has your role been with respect to this SBIR project? List as many as apply.
Technical
Financial
Commercialization assistance
Other (please specify)
This SBIR project received Phase III funding from your agency in the form of:
Further non-SBIR R&D funding
Direct procurement of the product of this SBIR
Procurement through incorporation of the result of this project into a system
No Phase III from agency
Unknown
Have you or others played a role in any Phase III funding for this project?
I have played that role
Someone else has played that role
Unknown
OCR for page 217
An Assessment of the SBIR Program at the National Aeronautics and Space Administration
Has this SBIR project been commercialized?
Yes
No
Does this SBIR project have non-commercial/intrinsic use (perhaps in research)? Elaborate.
Yes
No
If you have any comments that you think would help us evaluate the effectiveness of the SBIR Program in producing useful research for your agency, please feel free to comment. (Optional)
Annex B1
QUESTION 15. If you have any comments that you think would help us evaluate the effectiveness of the SBIR Program in producing useful research for your agency, please feel free to comment. (Optional)
Annex B2
QUESTION 7. Please describe how this SBIR project may have had an impact on your agency’s mission (Optional)
Annex B3
QUESTION 14. Does this SBIR project have noncommercial/intrinsic use (perhaps in research)? Elaborate.