The National Center for Science and Engineering Statistics (NCSES) of the National Science Foundation (NSF) is responsible for national reporting of the research and development (R&D) activities that occur in all sectors of the U.S. economy (National Science Foundation, n.d.-b). For most sectors, including the business and higher education sectors, NCSES collects data on these activities on a regular basis. NCSES has been proactive in seeking systematic independent reviews of its program and improving their surveys based on these reviews. Two of these reviews have been conducted by the National Research Council (NRC): Measuring Research and Development Expenditures in the U.S. Economy (National Research Council, 2005), and National Patterns of R&D Resources: Future Directions for Content and Methods, Summary of a Workshop (National Research Council, 2013). The surveys on the industrial sector and academic sector were revised in 2008 and 2010, respectively, with an intention to capture information on innovation taking place in business firms and R&D performed by universities and colleges in nonscience and nonengineering fields.
However, data on R&D within the entire nonprofit sector have not been collected in 18 years, a time period that has seen the dynamic and rapid growth of the sector (Salamon, 2012). The 2013 NRC workshop summary cited above pointed out issues with NCSES’ current modeling approach to estimation of “other nonprofit” R&D. At that workshop, Michael Cohen, NRC, stated that “it is reasonable to conclude that the current method is unlikely to provide high-quality estimates” (National Research Coun-
Statement of Task
An ad hoc steering committee will organize a public workshop on issues involved in measuring research and development (R&D) expenditures and related topics, such as employment of science and engineering personnel in R&D, in the nonprofit sector of the U.S. economy. The workshop will have the specific objective of identifying issues for the collection of intramural research and development expenditures by nonprofit organizations, considering the goals, content, statistical methodology, data quality, and data products associated with this data collection. The workshop will also consider data uses and the needs of data users and the relevance and adequacy of the resulting products for meeting current and emerging data needs for information about research and development expenditures for this sector. Following the workshop, a designated rapporteur will prepare an individually authored summary of the presentations and discussion.
cil, 2013, p. 51). Based partly on this 2013 workshop, NCSES decided to design and implement a new survey of nonprofits, hired a contractor (ICF International) to begin planning for it, and commissioned the workshop summarized in this report to provide a forum to discuss conceptual and design issues and methods. Specifically, NCSES sought to benefit from the combined expertise of national and international experts in survey methodology and nonprofit R&D, as well as to listen to and learn from representatives from a number of different nonprofit organizations. The statement of task for this workshop is provided in Box 1-1.
NCSES, formerly the Division of Science Resources Statistics of the National Science Foundation, was established by Section 505 of the America COMPETES Reauthorization Act of 2010. This act mandated, among other requirements, that NCSES provide statistical data on U.S. R&D performance and funding. The legislation also required that NCSES focus more attention on U.S. competitiveness relative to other countries. In response, NCSES produces a variety of reports, briefs, and tabular data made available to the research community. The National Patterns of R&D Resources, the primary product produced with relevance to R&D in the nonprofit sector, is published annually.
R&D expenditures by nonprofit organizations are an important component of total U.S. R&D spending, with disproportionate impacts in certain fields, such as biomedical research. Nonprofit organizations both
provide and receive R&D funds from other organizations. In its National Patterns of R&D Resources, NCSES separates R&D in the academic sector (most of which are not-for-profit institutions) from that in “other nonprofits.” While some of the R&D expenditures for “other nonprofits” are measured in collections such as the Survey of Federal Funds for Research and Development, which provides an estimate of funding provided by the federal government to nonprofits, NCSES has only occasionally collected information on the level of intramural and extramural R&D expenditures in the entire U.S. nonprofit sector. The most recent survey of “other nonprofits,” the 1996–1997 NSF Nonprofit R&D Survey, released results in 2001 (National Science Foundation, 2001; The Gallup Organization, 2000). The only previous collection was in 1973.
Since the previous survey of R&D in the “other nonprofit” sector, the gap in information has been filled by model-based estimates of nonprofit R&D in the NCSES National Patterns of R&D Resources releases (National Science Foundation, n.d.-a), using elasticity ratios derived from the 1996–1997 survey data. However, this model relies on assumptions that various relationships have not changed over time, which are questionable. In addition to the 2013 NRC report cited above, a 2005 study (National Research Council, 2005, p. 8) concluded that
in reviewing the attempts by NSF to collect data on the nonprofit sector, the panel noted that there were evident problems that were well documented in the methodology report on this survey. Nonetheless, the panel recommends that another attempt should be made to make a survey-based, independent estimate of the amount of R&D performed in the nonprofit sector (Recommendation 3.10). The panel also recommends that NSF evaluate the possibility of collecting for nonprofit institutions the same science and engineering variables that pertain to academia (Recommendation 5.3).
FOCUS OF THE WORKSHOP
At the workshop, John Gawalt, director of NCSES, described the context and importance of the planned new survey, with additional detail provided by Mark Boroush, senior analyst in the R&D Statistics Program at NCSES. National Patterns of R&D Resources includes data from separate surveys of various R&D-performing sectors of the U.S. economy (see Box 1-2) and consists of data on R&D expenditures by source of funds, sector of performance, character of work (basic research, applied research, or development), and international comparisons. Data from National Patterns feed into various international publications and databases, such as the OECD main science and technology indicators; the science and technology data of the Institute of Statistics of the United Nations Educational,
Data Sources Used in National Patterns of R&D Resources
- Higher Education Research and Development Survey (HERD)
- Survey of Federal Funds for Research and Development (Federal Funds)
- Survey of Federal Science and Engineering Support to Universities, Colleges and Nonprofit Institutions (Federal Support)
- Business R&D and Innovation Survey (BRDIS)
- Survey of State Government R&D (State R&D)
- Nonprofit Research and Development Survey
SOURCES: Boroush (2014), National Research Council (2013).
Scientific and Cultural Organization (UNESCO); and Eurostat’s statistics database.
Although NCSES has been gathering data from the other R&D-performing sectors at regular intervals, extending as far back as the 1950s in some cases, they have recognized that without the broader nonprofit sector they “have a gap, in what’s becoming an increasingly important part of the economy and part of the research environment,” stated Gawalt. R&D that is performed in academic institutions, a major portion of the R&D in the broader nonprofit sector, is covered annually by the Higher Education Research and Development Survey.1 However, the lack of recent data on R&D in other parts of the nonprofit sector provided the impetus for the current workshop. Particular emphasis in the survey will be placed on measuring R&D performance more so than funding activities, according to Gawalt. Overall, the National Patterns of R&D Resources is important to U.S. policy, Boroush noted, because “there is a lot of concern about how much money is being spent on R&D in all of the sectors … and it’s not just the U.S., but about how the U.S. compares to the other major nations.” The new R&D survey on the nonprofit sector is expected to generate estimates for the sector, plus contribute to data requirements of the international science and technology community.
A primary objective of this new survey, from the viewpoint of NCSES, is to fill data gaps in the National Patterns of R&D Resources, and to do so
1The Higher Education Research and Development Survey, also referred to as HERD, is an annual census of institutions that expended at least $150,000 in separately budgeted R&D in the fiscal year. Available: http://www.nsf.gov/statistics/srvyherd/ [October 2014].
in a way that is compatible with data collected on the other sectors of the U.S. economy and appropriate for international comparisons. Lester Salamon, chair of the workshop steering committee, stated that this survey has implications beyond filling these specific gaps. He emphasized the need to understand the nonprofit sector, given its enormous size and scope as well as its contribution to identifying new forms of R&D beyond physical production processes and new technology. These new forms of R&D involve new social processes that, in his view, can potentially relieve or resolve a variety of social and economic maladies. As such, they have important implications for the broader service economy, which now dominates the U.S. and other advanced economies. He noted, “The nonprofit sector, of course, is at the center of the service world. It therefore not only fills a gap in a bunch of tables, but I think it fills a conceptual gap in the understanding of contemporary R&D.” These new understandings have the potential to inform how NSF thinks about R&D across sectors, he added.
There is an inherent tension between the narrower goal of providing improved-quality data for the existing National Patterns of R&D Resources release and providing a broader understanding of the nonprofit sector ’s diverse R&D activities as articulated above by Salamon. Aware of this potential tension, the steering committee made an explicit decision that the workshop should begin with a broader discussion of R&D activities in the nonprofit sector before focusing on the specific methodology for the survey design.
THE WORKSHOP AND THIS SUMMARY REPORT
The purposes of the workshop and this summary were (1) to identify concepts and issues for the design of a survey of R&D expenditures made by (other) nonprofit organizations, considering the goals, content, statistical methodology, data quality, and data products associated with this data collection and (2) to consider the broader usefulness and relevance of the data for meeting current and emerging data needs, including fostering a better understanding of the nature of the nonprofit sector and its R&D activities. Given these dual purposes, the steering committee identified nine topics important to consider through presentations and discussions at the workshop:
- The nonprofit sector is not well understood, and a discussion of the sector needed to begin with establishing a common understanding of this diverse sector.
- R&D is occurring in the nonprofit sector, but much of it may differ from traditional forms of R&D. Workshop participants needed to
learn about these activities directly from nonprofit organizations themselves.
- Because of these potential differences, new language to describe R&D activities to respondents may need to be identified in order to elicit accurate reporting on the survey.
- Flowing from this notion, workshop discussions should address the design of screening prompts so that they are meaningful to respondents.
- Improved data sources now exist that may aid in sampling for the survey, and these should be explored in greater detail.
- Obtaining a good response rate has been historically problematic, and specific strategies to maximize response should be discussed.
- Specifically included in these strategies should be engaging and collaborating with nonprofit associations to help explain the importance of the survey to their member organizations.
- Linking this survey of R&D activities to broader themes that resonate within the nonprofit sector, such as social innovation, should be discussed as a strategy to enhance the usefulness of the survey to the nonprofit sector.
- It may not be possible to field a nonprofit survey annually, so discussions should be included to address ways of moving forward beyond a single implementation of a new survey.
These nine topics informed the workshop agenda and this report. The agenda items are in accord with the issues mentioned in the above statement of task.
The workshop presentations and discussions provided a variety of views and suggested a range of ideas and strategies to the sponsoring agency. The workshop agenda is presented in Appendix A. The workshop presentations are available on the webpage of the Committee on National Statistics at http://sites.nationalacademies.org/dbasse/cnstat/index.htm. This summary report generally follows the workshop structure, summarizing the presentations and the discussions on each topic. It is important to note that a workshop is not a consensus activity—no consensus recommendations or other findings are offered in this report. Individual presenters, steering committee members, and workshop participants were encouraged to offer their own opinions and suggested strategies, and these are reported, with attribution, throughout this report.
Chapter 2 focuses on examining the size, scope, and nature of the U.S. nonprofit sector based on current research. Chapter 3 presents the definition that NCSES currently uses for R&D. It then offers a view of the nonprofit sector and its R&D activities based on presentations from representatives of six different nonprofit organizations. This chapter also
presents seven challenges to designing the survey that participants identified through their discussion. Chapter 4 is devoted to the issues of sample design, including the benefits and drawbacks of various approaches, potential sources of data to facilitate the process, and lessons learned from the previous survey and from international examples. Chapter 5 focuses on the design of the survey instrument, strategies for increasing response rates, and potential outputs and uses of the survey data.
Finally, Chapter 6 offers a summary of the key themes that were identified through the workshop discussions. These themes are consistent with the steering committee’s initial outline of issues, and include
- NCSES’ need to fill the data gaps in the National Patterns of R&D Resources release in a way that is compatible for between-sector and international comparisons;
- the additional need to facilitate a more accurate statistical portrait of the scope and nature of R&D in the nonprofit sector, and the inherent tension between this need and NCSES’ primary need for the survey;
- an understanding of the scope of the nonprofit sector, and types of organizations that should be included;
- the unique nature of R&D within the nonprofit sector, and whether these potential R&D activities should be measured in the survey;
- the importance of statistical efficiency2 for estimating R&D expenditures at the national level, versus designing a sample that is inclusive of the diversity of R&D in the entire sector;
- the complex relationships existing among organizations in the nonprofit sector, and the impact these relationships may have on building a sampling frame and selecting a statistically efficient sample;
- the need to establish and maintain communication between NCSES and the nonprofit sector;
- strategies for achieving the correct respondent within an organization for completing the survey;
- strategies for obtaining a good response rate; and
- a need to move away from the questionnaire format used in the 1996–1997 survey and to develop strategies for “screening in” respondents through a redesign of screening questions.
2The statistical efficiency of an estimate from a sample survey refers to the quality of the estimate (such as the size of the variance or mean squared error) for a given cost (or sample size).
This summary is limited to the views expressed either during the activities undertaken in planning the workshop or at the workshop itself. Therefore, all views expressed are those of the workshop presenters and other workshop participants.