An Assessment of the SBIR Program at the National Aeronautics and Space Administration (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix D NRC Project Manager Survey David H. Finifter The College of William and Mary February 7, 2007 INTRODUCTION Federal agency R&D programs are increasingly assessed in terms of their quality, relevance and efficiency. One means of gauging the performance of the SBIR program in light of these criteria is to obtain assessments by agency project managers who manage contracted SBIR programs. The survey findings reported here represent the first systematic effort to obtain such data from SBIR project managers. The survey questions were constructed to correspond to the three cri- teria above. Also, in an effort to address the ever present assessment question of “compared to what,” the survey was designed to obtain respondent assessments of a comparison set of agency externally funded R&D projects. Program performance of the SBIR program is very complex. Therefore, it is important to get as many perspectives on the SBIR program performance as pos- sible. One group of participants in the process is agency project managers who manage contracted SBIR projects. This role only exists at agencies that contract SBIR awards (e.g., Department of Defense-DoD, Department of Energy-DoE, and the National Aeronautics and Space Administration-NASA) as opposed to administering awards as grants (e.g., National Institutes of Health-NIH and the National Science Foundation-NSF). Project managers take on different names and roles at the three agencies in question. At DoD, they are called Technical Points of Contact (TPOCs); at DoE, they are called Technical Project Managers (TPMs); and at NASA they are called Contract Officer’s Technical Representa- tives (COTRs). This paper gives two perspectives of project managers based primarily on an electronic survey that was administered as part of the National Research Council (NRC) assessment of the SBIR program. First, the report examines the role of 198 

APPENDIX D 199 the project manager in the SBIR project from topic selection to Phase III fund- ing. Second, the report documents how project managers at these three agencies view the relative quality, usefulness and value to the agency of SBIR projects in comparison to other research at the agency. Using the same time period of the NRC Phase II Survey of projects and firms completed as part of the NRC study as the reference point (1992-2001), we generated a list of Phase II SBIR projects for the three contracting agencies. We then requested from the SBIR program offices a list of project managers names and email addresses for as many of these individuals as possible. Naturally, there was significant attrition (absence of email addresses, absence of name for project manager, project manager having left the agency and/or was deceased, simply an error in the identified project manager). Given the constraints on collection of contact information and the time since many of the identified projects were implemented, the sample is a critical minimum size. A survey instrument was developed (based on Archibald-Finifter, 2000) to determine quality and useful- ness of research and mission benefits of the SBIR Phase II projects as seen by the project managers and is provided as Annex A of this paper. General Background The sample of project managers was first based on the original database provided by Peter Cahill of BRTRC of 11,684 SBIR Phase II projects in the five SBIR agencies (the three listed above plus NSF and NIH, which award grants, not contracts, and hence do not have contract officers as such). For the sample of the three agencies in question (DoD, DoE, and NASA), there is an estimated n=7,945 based on the BRTRC sample since 68 percent of the total sample was for these three agencies). The 7,945 represents an estimate of the potential number of project managers that could receive surveys. The number is actually far less than that because many project managers have done more that one Phase II project and therefore received a modified questionnaire for multiple projects. In a similar study by Archibald-Finifter for the DoD Fast Track study, 51.5 percent of the full sample responded and 78.9 percent of the successful contacts responded.  These were all recent relative to the survey data so the response rate was fairly high. For the current survey, the actual sample of project managers/projects with known names was—5,650 for DoD, 1,488 for NASA, and 808 for DoE. These projects represent the requests for names and email addresses sent to the three agencies. In response, the agencies were able to locate names and email addresses   Robert B. Archibald and David H. Finifter, “Evaluation of the Department of Defense Small Business Innovation Research Program and Fast Track Initiative: A Balanced Approach,” in N ­ ational ­ Research Council, The Small Business Innovation Research Program: An Assessment of the ­Department of Defense Fast Track Initiative, Charles W. Wessner, ed., National Academy Press, Washington, DC, 2000.   Ibid. 

200 APPENDIX D for a limited subset of project managers—1,757 for DoD, 555 for DoE, and 272 for NASA. Finally, the actual responses we received for the survey amounted to a total of 513 (19.8 percent rate) including—347 (19.7 percent rate) for DoD, 84 (15.1 percent rate) for DoE, and 82 (30.1 percent rate) for NASA. The NASA sample was based on projects since 1997 only (the only project managers for which they had email information). The higher response rate is probably due in part to the fact that the sample is based on more recent projects. We were thus more likely to have respondents who were still available and who were more likely to have recalled the SBIR project about which they were being asked. THE ROLE OF THE PROJECT MANAGER Project Manager’s Role in Selection Question 2 asked if the Project Manager was involved in generating the topic that led to this particular SBIR project. In Question 2 of the questionnaire, the respondent was asked if he or she was involved in generating the topic that led to this particular Phase II award. Table App-D-1 provides the response to this question for the total sample of project managers and results broken down by agency. Nearly 70 percent of the total respondents were involved in generating the SBIR project. This was appar- ently the case for DoD and DoE. However, NASA COTR respondents were less likely to have been involved in generating the topic that led to the SBIR project (only 57 percent). Question 3 asked the Project Manager when he or she became involved in the particular SBIR project. Another dimension of the role of project manager in the SBIR process re- lates to when he or she became involved in the project. As seen in Table App-D-2, for the total sample of project managers, 23 percent became involved after Phase I but before Phase II. Another 10 percent became involved after Phase II started but before Phase II was completed. Nearly two percent became involved after Phase II was completed. Over 65 percent were involved before Phase I started TABLE App-D-1  Involvement in Generating Topic Involved in Generating the Topic that Led to this Particular SBIR Project? Total Sample DoD DoE NASA Yes 357 (69.59%) 251 (72.33%) 59 (70.24%) 47 (57.32%) No 156 (30.41%)   96 (27.67%) 25 (29.76%) 35 (42.68%) TOTAL 513 347 84 82 

APPENDIX D 201 TABLE App-D-2  When Project Manager Became Involved in Project? When Involved in SBIR Total Project? Sample DoD DoE NASA After Phase I, Before Phase II 118 (23%)   56 (16.14%) 25 (29.76%) 37 (45.12%) After Phase II Started, Before   53 (10.33%)   32 (9.22%) 10 (11.90%) 11 (13.41%) Phase II Completed After Phase II Completed    8 (1.56%)    8 (2.31%) —— —— Before Phase I 334 (65.11%) 251 (72.33%) 49 (58.33%) 34 (41.46%) TOTAL 513 347 84 82 (consistent with the previous question relating to involvement in project genera- tion). This varied by agency as seen in Table App-D-2. For example, around 45 percent of the NASA respondents became involved in the project after Phase I, but before Phase II. Also, for DoD respondents, over 72 percent became involved before Phase I compared to 58 percent for DoE and 41 percent for NASA respondents. The degree to which a project manager is attached to the SBIR project by helping to come up with the topic to timing of involvement will have a likely impact on the knowledge and “ownership” of the project and may affect the respondents’ view of the quality and usefulness of the project. This is analyzed further below. Actual Role in Project Question 10. What has your role been with respect to this SBIR project? List as many as apply. The actual role of the project manager may vary from project-to-project, project manager-to-project manager (perhaps depending on skills and expertise of the project manager), and agency-to-agency. Question 10 of the survey asked about the project managers’ role(s) with respect to the particular SBIR project in question. The results are summarized in Table App-D-3. The respondents were to respond to as many of the roles as applied. For the total sample, over 96 percent TABLE App-D-3  Role of Project Manager with Respect to This SBIR Project Role of Project Manager Total Sample DoD DoE NASA Technical 493 (96.10%) 331 (95.39%) 82 (97.62%) 80 (97.56%) Financial 109 (21.25%)   86 (24.78%) 19 (22.62%)   4 (4.88%) Commercialization Assistance   84 (16.37%)   68 (19.60%)   6 (7.14%) 10 (12.19%) Other   55 (10.72%)   44 (12.68%)   6 (7.14%)   5 (6.10%) TOTAL SAMPLE SIZE 513 347 84 82 NOTE: Multiple responses permitted. 

202 APPENDIX D TABLE App-D-4  Played a Role in Any Phase III Funding for This Project Who Played a Role in Any Total Phase III Funding for this Project Sample DoD DoE NASA Respondent Played that Role   60 (11.72%)   50 (14.45%)   4 (4.76%)   6 (7.32%) Someone Else Has Played that Role   77 (15.04%)   59 (17.05%) 12 (14.29%)   6 (7.32%) Unknown 375 (73.24%) 237 (68.50%) 68 (80.95%) 70 (85.37%) TOTAL 512 346 84 82 of the respondents claimed a “technical” role, with around 21 percent claiming a “financial” role and around 16 percent claiming a “commercialization assistance” role. The “commercialization assistance” role was a bit higher for those in DoD (around 20 percent) and lower for DoE and NASA (only around 7 and 12 percent, respectively). Clearly, the “technical” role dominates in all agencies. It is notable that for NASA, only around 5 percent had a “financial” role. Project Manager’s Role in Phase III Funding Question 12. Have you or others played a role in any Phase III funding for this project? The survey examined the role of the project manager in Phase III funding for the various projects listed (Question 12). Table App-D-4 summarizes the findings about the role. The overwhelming finding is that 73 percent of the respondents did not know who played such a role in Phase III funding efforts. The result was slightly lower for DoD respondents (68 percent) and higher for DoE and NASA (81 and 85 percent, respectively). Nearly 12 percent of the respondents in the overall sample said they played that role, ranging from 14 percent for DoD, to 7 percent for NASA, to 5 percent for DoE respondents. Overall, project managers seem to have a relatively minor role in Phase III funding for the agencies and projects studied. This is an important finding since project managers would be knowledgeable natural advocates for the project within the agency. THE QUALITY, USEFULNESS, AND VALUE TO THE AGENCY OF THE SBIR PROJECT AS SEEN BY THE PROJECT MANAGER As seen above, project managers tend to be fairly involved with the SBIR Phase II projects to which they are assigned, although the degree varies somewhat by agency. This involvement gives them a unique basis for evaluating the quality, usefulness, and value to the agency of each project. We used the survey of project managers to develop measures of these dimensions of program outcomes. 

APPENDIX D 203 TABLE App-D-5  Ratings for SBIR Projects Measure of Quality of SBIR Project Total Sample DoD DoE NASA Mean Score 6.93 6.95 6.80 6.98 Standard Deviation 2.072 2.202 1.706 1.846 Median Score 7 8 7 8 Sample Size 513 347 84 82 Project Quality Question 4. On a 1 to 10 scale, where 10 represents the best research ever pro- duced in your research unit/office or for your research unit/office and 1 represents the worst research ever produced in your research unit/office or for your research unit/office, rate the quality of the research in this particular SBIR contract. First, we use Questions 4 and 5 to arrive at a rating of the quality of SBIR Phase II projects from our sample as seen by the project managers. Question 4 asks the respondent to, on a scale of 1 to 10, where 10 represents the best research ever produced in their research unit/office or for their research unit/office and 1 represents the worst research ever produced in their research unit/office or for their research unit/office, rate the quality of the particular SBIR contract listed. The results for Question 4 are given in Table App-D-5 for the total sample of project managers and by each agency. The mean score for the total sample of 513 project managers/projects was 6.93 (with a standard deviation of 2.072) and the median score was 7. There were similar scores by agency as seen in Table App-D-5. Question 5. On the same scale rate the average quality of the research projects conducted for your research unit/office from contracts other than SBIR contracts for the last two years. The problem with this measure is that it does not adjust for “toughness of the grader.” To compensate for the fact that different project managers have dif- ferent standards in mind when evaluating a research project, we asked Question 5—On the same scale (1 to 10) rate the average quality of the research projects conducted for their research unit/office from contracts other than SBIR contracts for the last two years. A summary of those scores is given in Table App-D-6. It shows for example that the mean score for non-SBIR projects was 7.29 (standard deviation of 1.594) and a median of 8. A comparison of the scores for the SBIR projects and the average quality of relevant non-SBIR contracts allows us to gauge the relative quality of the SBIR projects. The differences in means between the average score for the sample of SBIR projects and the average non-SBIR project are given in Table App-D-7. For the total sample of SBIR projects, the SBIR projects were on average lower by 

204 APPENDIX D TABLE App-D-6  Ratings for Non-SBIR Research Projects Measure of Average Quality of Research (non-SBIR) Total Sample DoD DoE NASA Mean Score 7.29 7.27 7.24 7.45 Standard Deviation 1.594 1.635 1.712 1.268 Median Score 8 8 7 7 Sample Size 513 347 84 82 TABLE App-D-7  Mean Difference in Scores SBIR Quality Minus Average Non-SBIR Project Quality Measure of Difference in Scores of Project Quality Total Sample DoD DoE NASA Mean Difference in Score—SBIR –.364* –.320* –.440** –.476** Quality Minus Average Non-SBIR Project Quality Standard Deviation 2.15 2.19 2.03 2.10 NOTES: *Statistically significant at the .01 level; **Statistically significant at the .05 level. .364. Similar differences are shown for the DoD, DoE, and NASA subsamples. Using a t-test, these differences are statistically significantly different from zero. The average quality score is slightly lower for SBIR projects than for non-SBIR research projects (although the differences are statistically significant). This result differs from the result found in Archibald-Finifter which shows no statistical difference between the two scores for a sample of DoD projects. Further examination of the results of the current survey shed some light on the slightly lower rating for SBIR projects. In looking at the distribution of scores for Question 3 (SBIR scores) and Question 4 (non-SBIR scores), we find that 7.4 percent of the respondents rank SBIR projects at a score of 3 or below compared to 1.9 percent for non-SBIR projects. The remainder of the distribution is much more similar. For example, around 49 percent of the SBIR sample ranked score above seven, compared to around 52 percent for the non-SBIR sample. This same skewed pattern of a few very low scores held for the three subsamples as well. Thus, the result seeming to favor non-SBIR research quality over SBIR quality is driven by a few outliers scoring the SBIR projects extremely low. The small difference in means and other evidence tends to show SBIR in a reasonably favor- able light in terms of its quality and other dimensions of usefulness compared to non-SBIR research projects at the same agency.   Ibid. 

APPENDIX D 205 TABLE App-D-8  Distribution of Scores: SBIR Quality and Average Non- SBIR Project Quality (Total Sample) Quality of Research Average Quality of Research (SBIR) (Non-SBIR) Score of Number of Number of Project Quality Responses Percent Responses Percent 1 8 1.56 7 1.36 2 12 2.34 2 0.39 3 18 3.51 1 0.19 4 32 6.24 11 2.14 5 56 10.92 50 9.75 6 48 9.36 50 9.75 7 90 17.54 126 24.56 8 129 25.15 162 31.58 9 89 17.35 84 16.37 10 31 6.04 20 3.90 TOTAL 513 100.00 513 100.00 Linkage to Research Mission Question 6. Has the research conducted for this SBIR contract affected the way that your research unit/office conducts research or the type of research your re- search unit/office obtains in other contracts? (List as many as apply.) The next question we raised in the survey dealt more with usefulness of the research for the research unit/office. In particular, we asked Question 6—Has the TABLE App-D-9  Distribution of Scores: SBIR Quality and Average Non- SBIR Project Quality (DoD) Quality of Research Average Quality of Research (SBIR) (Non-SBIR) Score of Number of Number of Project Quality Responses Percent Responses Percent 1 8 2.31 7 2.02 2 9 2.59 0 0.00 3 13 3.75 1 0.29 4 22 6.34 7 2.02 5 38 10.95 35 10.09 6 26 7.49 30 8.65 7 53 15.27 88 25.36 8 85 24.50 109 31.41 9 68 19.60 58 16.71 10 25 7.20 12 3.46 TOTAL 347 100.00 347 100.00 

206 APPENDIX D TABLE App-D-10  Distribution of Scores: SBIR Quality and Average Non- SBIR Project Quality (DoE) Average Quality of Research Quality of Research (SBIR) (Non-SBIR) Score of Number of Number of Project Quality Responses Percent Responses Percent 1 0 0.00 0 0.00 2 1 1.19 2 2.38 3 2 2.38 0 0.00 4 6 7.14 3 3.57 5 11 13.10 8 9.52 6 12 14.29 11 13.10 7 17 20.24 19 22.62 8 24 28.57 22 26.19 9 9 10.71 13 15.48 10 2 2.38 6 7.14 TOTAL 84 100.00 84 100.00 research conducted for this SBIR contract affected the way that your research unit/office conducts research or the type of research your research unit/office obtains in other contracts? (List as many as apply.) These results are summarized in Table App-D-12. The responses to this question varied across agencies somewhat. Of those project managers who responded “No, this project was a separate project, and the knowledge generated by this SBIR contract has had no impact on the other TABLE App-D-11  Distribution of Scores: SBIR Quality and Average Non- SBIR Project Quality (NASA) Average Quality of Research Quality of Research (SBIR) (Non-SBIR) Score of Number of Number of Project Quality Responses Percent Responses Percent 1 0 0.00 0 0.00 2 2 2.44 0 0.00 3 3 3.66 0 0.00 4 4 4.88 1 1.22 5 7 8.54 7 8.54 6 10 12.20 9 10.98 7 20 24.39 19 23.17 8 20 24.39 31 37.80 9 12 14.63 13 15.85 10 4 4.88 2 2.44 TOTAL 82 100.00 82 100.00 

APPENDIX D 207 TABLE App-D-12  Effect of SBIR Project’s Research on Your Research Unit Total Research Effect Sample DoD DoE NASA No, this project was a separate 166 (32.4%)   88 (25.4%) 42 (50.0%) 36 (43.9%) project, and the knowledge generated by this SBIR contract has had no impact on the other research we conduct or sponsor. Yes, this project produced results 148 (28.8%) 108 (31.1%) 19 (22.6%) 21 (25.6%) that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract by encouraging the firm to apply for additional SBIR awards. Yes, this project produced results 234 (45.6%) 184 (53.0%) 24 (28.6%) 26 (31.7%) that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract in other research we conduct or sponsor. Yes, but this project found a blind 49 (9.5%) 35 (10.1%) 6 (7.1%) 8 (9.8%) alley, so we have not followed up on this line of inquiry. TOTAL SAMPLE 513 347 84 82 NOTE: Multiple responses permitted. research we conduct or sponsor,” 32 percent of the total sample gave this response of no linkage. However, the response was only 25 percent from DoD project managers, 50 percent from DoE project managers, and 44 percent from NASA project managers. The low percentage response for DoD is indicates relatively high linkage and usefulness and this is probably not a surprise for an agency that is so mission driven. “Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract by encouraging the firm to apply for additional SBIR awards” yielded some positive response. For the total sample, nearly 29 percent cited this response. The breakdown by agency was not dramatic—percentages around 31, 23, and 26, respectively for DoD, DoE, and NASA. The response category “Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract in other research we conduct or sponsor” led to some interesting results. The overall sample showed a response of around 46 percent. That ranged from the high of 53 percent for DoD to a low of around 29 and 32 percent for DoE and 

208 APPENDIX D NASA, respectively. This may reflect the greater size and scope within DoD, or it might suggest a greater effort by DoD in finding alternative uses for the research within the agency. In general, the results indicate that there is a relatively high agency usage rate or attempted usage rate for SBIR projects in our sample. In some cases this lead to additional SBIR awards and in some cases it influences other non-SBIR research. In any case, potential impact is relatively high for all three agencies. One other response on this question is notable. “Yes, but this project found a blind alley, so we have not followed up on this line of inquiry” had a response rate of 9.5 percent for the total sample (around 10, 7, and 10 percent for DoD, DoE, and NASA samples, respectively). Having a 10 percent “blind alley” rate is relatively small and can be viewed as “good forecasting” by the agencies. Alter- natively, we would expect agencies to take some risks with the SBIR selections and perhaps the 10 percent rate of “blind alleys” is too low for the goals of the program. This result merits further study. Comparative Value of SBIR Projects Question 8. In comparison to a dollar spent in your research unit/office on other R&D projects, how did a dollar spent on this SBIR project rank? Information on another dimension of quality and usefulness of the SBIR program was sought through Question 8—“In comparison to a dollar spent in your research unit/office on other R&D projects, how did a dollar spent on this SBIR project rank” is shown in Table App-D-13. For the total sample around 28 percent responded that the particular SBIR project had fewer benefits for their agency’s mission than the average dollar spend on other contracts they sponsor. Another 28 percent responded that the SBIR project had more benefits SBIR TABLE App-D-13  Comparative Dollar Value of Projects Dollar Value Total Sample DoD DoE NASA SBIR project had fewer benefits 145 (28.27%)   93 (26.80%) 26 (30.95%) 26 (31.71%) for your agency’s mission than the average dollar spent on other contracts you sponsor SBIR project had more benefits 143 (27.88%) 114 (32.85%) 14 (16.67%) 15 (18.29%) for your agency’s mission than the average dollar spent on other research contracts you sponsor Same Benefits 225 (43.86%) 140 (40.35%) 44 (52.38%) 41 (50.00%) TOTAL SAMPLE SIZE 513 347 84 82 

APPENDIX D 209 project had more benefits for their agency’s mission than the average dollar spent on other research contracts they sponsor. This response varied by agency quite a bit—The response from around 33 percent for DoD respondents, but only around 17 to 18 percent for DoE and NASA, respectively. The project managers who responded the same benefits amounted to around 44 percent for the total sample, ranging from around 40 percent for DoD respondents to 52 and 50 percent for DoE and NASA respondents respectively. Overall the conclusion is that over 71 percent of the SBIR projects had at least as high benefits for the mission of the agency as the average dollar spent on non-SBIR projects. Abundance of Fundable SBIR Proposals Another dimension of SBIR project and program success was approached by survey Question 9. This question asked: “In general do you find that your research unit/office has had more good SBIR proposals than you can fund?” This question gets at the value of the marginal SBIR project and the consequences of expanding the funded SBIR projects. The results are impressive. As seen in Table App-D-14, while around 24 percent of the total sample of respondents claimed that there were about the right number of SBIR proposals currently, over 63 per- cent responded that there were more fundable SBIR proposals than they can fund, with only 13 percent claiming that there are fewer fundable proposal than they can fund. This result is fairly consistent for DoD and DoE respondents. NASA results are somewhat less favorable with around 21 percent responding that there are fewer fundable proposals than they can fund. However, over 56 percent still responded that there are more fundable projects than they can fund. Analysis of “Ownership Bias” of Findings As a check on bias of responses by the project managers, we broke the sample up into project managers with a potentially strong degree of “owner- ship” in the project and those with less potential for “ownership.” We defined TABLE App-D-14  Relative Number of Fundable SBIR Projects Relative Number of Fundable Total SBIR Projects Sample DoD DoE NASA More fundable proposals than can 220 (63.4%) 156 (65.8%) 24 (61.5%) 40 (56.3%) fund About the right number of proposals   82 (23.6%)   54 (22.8%) 12 (30.8%) 16 (22.5%) Fewer fundable proposals than can   45 (13.0%)   27 (11.4%)   3 (7.7%) 15 (21.1%) fund Respondents 347 237 39 71 

210 APPENDIX D TABLE App-D-15  Analysis of Ownership Effects—Entire Sample Area of Interest Ownership Group Remaining Project (n=391) Managers (n=122) Research Quality (Difference between Q3-Q4) –.261 –.697 Usefulness of Research (Q5) a. No, not useful 113 (28.90%) 53 (43.44%) b. Yes, more SBIR 116 (29.67%) 32 (26.23%) c. Yes, general follow-up 189 (48.34%) 45 (36.88%) d. Yes, but blind alley   46 (11.76%)   3 (2.46%) Mission Benefits (Q6) a. More than average 29.67% 22.13% b. Same as average 44.50% 41.80% c. Less than average 25.83% 36.07% “ownership” as those project managers who had a potential stake in the project as demonstrated either by being involved in defining the topic or if they were involved with the firm before the Phase I proposal. These project managers might be inclined to rate the project more highly and bias the results toward more fa- vorable outcomes. At the same time, these same project managers could also be more knowledgeable about the project and its outcomes. There were 391 of the 513 who were in this “ownership group.” Table App-D-15 through Table App- D-18 summarize the key measures of quality and usefulness of the projects by ownership group membership for the total sample and by agency. In general, the ownership group does rate the SBIR projects more highly in terms of quality and usefulness measures. Again, whether this is bias or greater knowledge or some combination is not known, but there do seem to be differences. Project Outcomes Question 13. Has this SBIR project been commercialized? It is sometimes argued that there is a trade-off between the research potential (intrinsic use) of an SBIR project and its potential for commercialization.  Table App-D-19 shows the rate of commercialization. This commercialization rate is around 34 percent for the total sample and varies slightly across agency (around 35 percent for DoD, 30 percent for DoE, and 35 percent for NASA). These rates reflect the knowledge base of the project managers.   Robert Archibald and David Finifter, “Evaluating the NASA Small Business Innovation Research Program: Preliminary Evidence of a Tradeoff Between Commercialization and Basic Research,” Research Policy 32:605-619, 2003. 

APPENDIX D 211 TABLE App-D-16  Analysis of Ownership Effects—DoD Ownership Group Remaining Project Area of Interest (n=271) Managers (n=76) Research Quality (Difference between Q3-Q4) –.229 –.645 Usefulness of Research (Q5) a. No, not useful   63 (23.25%) 25 (32.89%) b. Yes, more SBIR   86 (31.73%) 22 (28.95%) c. Yes, general follow-up 148 (54.61%) 36 (47.37%) d. Yes, but blind alley   33 (12.18%)   2 (2.63%) Mission Benefits (Q6) a. More than average 35.06% 25.00% b. Same as average 40.96% 38.16% c. Less than average 23.99% 36.84% TABLE App-D-17  Analysis of Ownership Effects—DoE Area of Interest Ownership Group Remaining Project (n=65) Managers (n=19) Research Quality (Difference between Q3-Q4) –.415 –.526 Usefulness of Research (Q5) a. No, not useful 34 (52.31%) 8 (42.10%) b. Yes, more SBIR 14 (21.54%) 5 (26.32%) c. Yes, general follow-up 18 (27.69%) 6 (31.58%) d. Yes, but blind alley   5 (7.69%) 1 (5.26%) Mission Benefits (Q6) a. More than average 15.38% 21.05% b. Same as average 52.31% 52.63% c. Less than average 32.31% 26.32% TABLE App-D-18  Analysis of Ownership Effects—NASA Ownership Group Remaining Project Area of Interest (n=55) Managers (n=27) Research Quality (Difference between Q3-Q4) –.236 –.963 Usefulness of Research (Q5) a. No, not useful 16 (29.09%) 20 (74.07%) b. Yes, more SBIR 16 (29.09%)   5 (18.52%) c. Yes, general follow-up 23 (41.82%)   3 (11.11%) d. Yes, but blind alley   8 (14.54%)   0 (0.00%) Mission Benefits (Q6) a. More than average 20.00% 14.81% b. Same as average 52.73% 44.44% c. Less than average 27.27% 40.74% 

212 APPENDIX D TABLE App-D-19  Project Commercialized? Has this SBIR Project Been Total Commercialized? Sample DoD DoE NASA Yes 175 (34.45%) 123 (35.55%) 24 (29.63%) 28 (34.57%) No 333 (65.55%) 223 (64.45%) 57 (70.37%) 53 (65.43%) TOTAL 508 346 81 81 Question 14. Does this SBIR project have noncommercial/intrinsic use (perhaps in research)? (Elaborate—see annex). Table App-D-20 shows how respondents evaluated the noncommercial/ intrinsic use (perhaps research) for these same SBIR projects. The intrinsic use rate for the total sample is around 66 percent. This rate varies somewhat across agencies (69 percent for DoD, 54 percent for DoE, and 63 percent for NASA). Table App-D-21 shows the relationship between whether the projects in our survey were commercialized and/or their intrinsic research potential. Table App-D-22 through Table App-D-24 show the same results by agency. Overall, the results show that SBIR projects are highly successful in producing either commercial and/or noncommercial/intrinsic use outcomes. The results from Table App-D-21 are compelling. Based on the responses of the project managers, nearly 74 percent of the SBIR projects showed a commer- cial or intrinsic use or both. In fact, over 26 percent (134 projects) were reported to have both a commercial and an intrinsic use. Thus, while there might be some trade-off between research and commercial potential, there may also be some degree of complementarity. The results differ somewhat by agency and these are shown in Table App-D-22 through Table App-D-24. How Project Managers Relate to Phase III One final dimension of outcome of an SBIR project that was covered by the project manager survey relates to Phase III funding. Question 11 probed this issue as follows: “This SBIR project received Phase III funding from your agency in TABLE App-D-20  Does SBIR Project Have Non-commercial/Intrinsic Use (Perhaps in Research)? Does SBIR Project Have Non-commercial/Intrinsic Use Total (Perhaps in Research)? Sample DoD DoE NASA Yes 336 (65.75%) 239 (69.08%) 45 (54.22%) 52 (63.41%) No 175 (34.25%) 107 (30.92%) 38 (45.78%) 30 (36.59%) TOTAL 511 346 83 82 

APPENDIX D 213 TABLE App-D-21  Relationship Between Commercial and Non-commercial Outcomes (Total Sample) Intrinsic Use (Research) Commercialized No Yes Total No 132 200 332 (65.48%) Yes 41 134 175 (34.52%) Total 173 (34.12%) 334 (65.88%) 507 TABLE App-D-22  Relationship Between Commercial and Non-commercial Outcomes (DoD Sample) Intrinsic Use (Research) Commercialized No Yes Total No   80 142 222 (64.35%) Yes   26   97 123 (35.65%) Total 106 (30.72%) 239 (69.28%) 345 TABLE App-D-23  Relationship Between Commercial and Non-commercial Outcomes (DoE Sample) Intrinsic Use (Research) Commercialized No Yes Total No 33 24 57 (70.37%) Yes  5 19 24 (29.63%) Total 38 (46.91%) 43 (53.09%) 81 TABLE App-D-24  Relationship Between Commercial and Non-commercial Outcomes (NASA Sample) Intrinsic Use (Research) Commercialized No Yes Total No 19 34 53 (65.43%) Yes 10 18 28 (34.57%) Total 29 (35.80%) 52 (64.20%) 81 the form of: Further non-SBIR R&D funding; Direct procurement of the product of this SBIR Procurement through incorporation of the result of this project into a system; No Phase III from agency; Unknown.” Results of this question are shown in Table App-D-25. The result which stands out the most is that for the total sample, around 58 percent of the projects were known to have not received 

214 APPENDIX D TABLE App-D-25  SBIR Project Received Phase III Funding from Your Agency Phase III Funding from Your Total Agency—Form Sample DoD DoE NASA Direct Procurement of the   15 (2.9%)   10 (2.9%)   0 (0.0%)   5 (6.1%) Product of this SBIR Procurement through   18 (3.5%)   11 (3.2%)   4 (4.8%)   3 (3.7%) Incorporation of the Result of this Project into System No Phase III from Agency 299 (58.3%) 214 (61.7%) 39 (46.4%) 46 (56.1%) Further non-SBIR R&D   53 (10.3%)   41 (11.8%)   7 (8.3%)   5 (6.1%) funding Unknown 128 (24.9%)   71 (20.5%) 34 (40.5%) 23 (28.0%) TOTAL 513 347 84 82 Phase III funding from the agency and another 25 percent were unknown. As seen in the table, the results vary somewhat by agency. Phase III direct procurement of the product of the SBIR project occurred in only 2.9 percent of the time for the total sample. That same percentage was the case for DoD respondents, but the results for the other two agencies varied greatly from zero cases for DoE to just above six percent for NASA. Procurement of the SBIR project through incorpora- tion of the result of the project into the system was also relatively low, with a rate of 3.5 percent for the total sample with a range from 3.2 to 4.8 percent across the agencies. For the total sample, there were over ten percent of the respondents who said the projects were supported further by non-SBIR R&D funding. CONCLUSIONS The SBIR project managers in our sample appear to be a rather engaged group with respect to the SBIR Program. They were engaged in the projects early and often. In general, the project managers ranked the quality of the SBIR research as close to the quality of research undertaken at their respective research units. The projects seemed to be useful to the mission at the various agencies and affected the subsequent research program (both SBIR and non-SBIR projects.) The project managers valued the projects highly compared to non-SBIR projects. Most agreed that there is a relative abundance of fundable SBIR proposals that do not get funded. Surprisingly, SBIR project managers seem to have little in- volvement in what is already a relatively low rate of Phase III activity. Finally, as viewed by the project managers, the SBIR projects under examination had a very high combined commercialization and intrinsic (research) use rate. 

APPENDIX D 215 REMAINING SECTIONS OF THE REPORT The remaining parts of the report are in annexes. Annex A contains the base survey. Annex B contains three parts that are responses to the open-ended ques- tions in the survey. Annex A The Base Questionnaire 1. Please name your research unit/office. 2. Were you involved in defining or generating the topic, which led to this particular SBIR project? • Yes • No 3. When did you first become involved with this SBIR Phase II project? • Before the Phase I proposal • After the Phase I proposal but before the Phase II proposal • After the Phase II proposal, but before the Phase II project was completed • After the Phase II project was completed 4.  a 1 to 10 scale, where 10 represents the best research ever produced in On your research unit/office or for your research unit/office and 1 represents the worst research ever produced in your research unit/office or for your research unit/office, rate the quality of the research in this particular SBIR contract. 1 2 3 4 5 6 7 8 9 10 5.  the same scale rate the average quality of the research projects conducted On for your research unit/office from contracts other than SBIR contracts for the last two years. 1 2 3 4 5 6 7 8 9 10 6. Has the research conducted for this SBIR contract affected the way that your research unit/office conducts research or the type of research your research unit/office obtains in other contracts? List as many as apply. • No, this project was a separate project, and the knowledge generated by this SBIR contract has had no impact on the other research we conduct or sponsor. • Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract by encourag- ing the firm to apply for additional SBIR awards. • Yes, this project produced results that have been useful to us, and we 

216 APPENDIX D have tried to follow up on the ideas initiated in this SBIR contract in other research we conduct or sponsor. • Yes, but this project found a blind alley, so we have not followed up on this line of inquiry. 7. Please describe how this SBIR project may have had an impact on your agency’s mission. (Optional) 8.  comparison to a dollar spent in your research unit/office on other R&D In projects, did a dollar spent on this SBIR project: • Yield more benefits for your agency’s mission that the average dollar spent on other contracts sponsored by your research unit/office. • Yield the same level of benefits from your agency’s mission as the average dollar spent on other contracts sponsored by your research unit/office. • Yield fewer benefits for your agency’s mission than that average dollar spent on other contracts sponsored by your research unit/office. 9.  general do you find that your research unit/office has had more good SBIR In proposals than you can fund? • Yes • No • About the right number of good proposals 10.  What has your role been with respect to this SBIR project? List as many as apply. • Technical • Financial • Commercialization assistance • Other (please specify) 11.  This SBIR project received Phase III funding from your agency in the form of: • Further non-SBIR R&D funding • Direct procurement of the product of this SBIR •  Procurement through incorporation of the result of this project into a system • No Phase III from agency • Unknown 12. Have you or others played a role in any Phase III funding for this project? • I have played that role • Someone else has played that role • Unknown 

APPENDIX D 217 13. Has this SBIR project been commercialized? • Yes • No 14.  Does this SBIR project have non-commercial/intrinsic use (perhaps in re- search)? Elaborate. • Yes • No 15.  you have any comments that you think would help us evaluate the effec- If tiveness of the SBIR Program in producing useful research for your agency, please feel free to comment. (Optional) Annex B1 QUESTION 15.  If you have any comments that you think would help us evalu- ate the effectiveness of the SBIR Program in producing useful research for your agency, please feel free to comment. (Optional) Annex B2 QUESTION 7.  Please describe how this SBIR project may have had an impact on your agency’s mission (Optional) Annex B3 QUESTION 14.  Does this SBIR project have noncommercial/intrinsic use (per- haps in research)? Elaborate.