Measuring Research and Development Expenditures in the U.S. Nonprofit Sector Conceptual and Design Issues: Summary of a Workshop (2015) / Chapter Skim
Currently Skimming:
5 Question Design and Survey Implementation
5 Question Design and Survey Implementation
Pages 85-116
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 85... ...
As discussed in the next session of the workshop, they can affect response rate and ultimately the usability of the data that NCSES collects. As Donald Dillman, Washington State University and workshop steering committee member shared, the science of survey design has been advancing rapidly, yielding evidence-based approaches that can inform the survey planning stage.
|
From page 86... ...
Understanding How Instructions and Clarifications Affect Survey Responses The discussions held during this workshop have made clear that there are complex concepts involved in what the NSF Nonprofit R&D Survey should measure. Much discussion took place about which organizations should be included in the population of nonprofits and eligible for sampling on the survey.
|
From page 87... ...
. SOURCE: National Science Foundation/National Center for Science and Engineering Statistics Survey of Research and Development Funding and Performance by Nonprofit Organizations, 1996–1997.
|
From page 88... ...
88 MEASURING R&D EXPENDITURES IN THE U.S. NONPROFIT SECTOR • Second, eye movement research indicates that respondents spend more time looking at the beginning of a question than at the end, but there has been little to no research on whether the phenom enon of skipping over clarification provided at the end of a ques tion occurs when the information is visual rather than auditory (Graesser et al., 2006)
|
From page 89... ...
These experiments were included as part of an existing national survey. The cumulative response rate was 38.8 percent.
|
From page 90... ...
3. Asking multiple questions is more effective than providing clari fying instructions.
|
From page 91... ...
With instructions, the errors occur depending on whether respondents actually read the instructions, but in the multiplequestion format, errors vary because the potential increased ability of respondents to recall the information needed. The multiple-question format also took more time, which is an important tradeoff to consider, Redline suggested.
|
From page 92... ...
However, Redline indicated that individuals often do not read through the questionnaire even when instructed to do so. Results from Exploratory Interviews with Nonprofits Ronda Britt, NCSES, shared the results of the first wave of exploratory interviews that NSF and their contractor, ICF International, are conducting with research-performing nonprofit organizations.
|
From page 93... ...
She strongly suggested that NSF also capture nonprofits "that do service delivery and are mission driven." The interviews in San Antonio were completed just prior to the workshop. There were four interviews conducted with two large research institutes and two smaller organizations with limited research activities.
|
From page 94... ...
This work has been "published in peer-reviewed journals, so there was definitely research going on there," explained Britt. Box 5-4 brings together the various terms for R&D that nonprofit organizations reported using, both in this workshop and during the exploratory interviews that NSF has conducted.
|
From page 95... ...
Britt suggested that an open-ended category could allow nonprofit organizations to identify the
|
From page 96... ...
Crosby also indicated that his nonprofit organization publishes a list of peer organizations, adding it is a source that NSF may consider using to identify other research-performing nonprofits. Another purpose of the exploratory interviews was to explore finding the correct respondents for the NSF nonprofit survey.
|
From page 97... ...
Discussion of Potential Approaches to Screening Carol House, NRC, led a discussion about a proposed approach to screening potential respondents to the NSF Nonprofit R&D Survey. She began by presenting the instructions included with the 1997 nonprofit survey, adding that "they're very complex instructions, and then they ask the respondent to use that definition when answering the remainder of the questionnaire." As shown above in Box 5-1, these instructions define research, development, and science and engineering.
|
From page 98... ...
(Consider total costs, including both direct and indirect costs in both internally and externally funded research or development.) ___ Less than $50,000 ___ $50,000-$249,999 ___ $250,000- $999,999 ___ $1,000,000-$4,999,999 ___ $5,000,000-$24,999,999 ___ $25 million or more SOURCE: NSF/NCSES Survey of Research and Development Funding and Performance by Nonprofit Organizations, 1996–1997.
|
From page 99... ...
• roduced findings that are published in academic journals or presented at P conferences? • eveloped new approaches to social service delivery and formally evalu D ated the results?
|
From page 100... ...
During the planning phase of the survey, Berry and his colleagues identified previous survey research focused on the nonprofit sector and determined that low overall response rate was a significant limitation. For example, one study of 5,000 nonprofits had only a 17 percent response rate, raising concern about selection bias.
|
From page 101... ...
Berry stated the survey was mailed to respondents, noting that a greater number of feasible options, including web surveys, exist today. Box 5-7 shows how the initial mailing and three follow-up mailings were scheduled to increase response rate.
|
From page 102... ...
Strategies for Improving Survey Response Rates Berry described his 1999 survey of nonprofits, which he implemented following Dillman's Total Design Method (Dillman, 1978)
|
From page 103... ...
Do not offer a choice of response mode unless you make it easy to respond by either mode. Doing so lowers response rates.
|
From page 104... ...
Using only an email contact would have been possible because they had an email contact for each person in the sample. However, using email only to get a web response was likely to yield a 20 to 25 percent response rate based on experience with other surveys of students conducted by the Social and Economic Sciences Research Center at Washington State University.
|
From page 105... ...
Research supports the use of multiple contacts and multiple response modes, and the need to demonstrate that responding is both easy and important to do. Dillman reiterated this point by providing a draft implementation sequence (see Box 5-9)
|
From page 106... ...
As Berry and Dillman suggested, Fecso stressed the importance of collecting useful data to enhance the sample frame at the outset, and he also recommended that NSF focus on obtaining a good response rate. Fecso said that obtaining good response from the big R&D-producing nonprofit organizations was particularly important, and that NSF should use screening tools to improve the efficiency of implementation.
|
From page 107... ...
Fecso indicated that effective screener items, tailored depending on how likely the organization is to be an R&D performer, can also help boost response rate. Tailoring can also be used to make the questionnaires more personal, suggested Fecso.
|
From page 108... ...
Third, reaching these selected organizations requires planning for multiple contacts. Stone stated her view that if it were not possible to develop a sampling approach that captures the full nonprofit sector, then the survey ought to be characterized differently and focused on large research institutes.
|
From page 109... ...
POTENTIAL OUTPUTS FROM THE SURVEY The workshop looked at the required outputs from the NSF Nonprofit R&D Survey and considered which additional outputs may be valuable to NSF or to the broader nonprofit community. A summary of that discussion follows.
|
From page 110... ...
. Data from National Science Foundation, National Center for Science and Engineering Statistics, National Patterns of R&D Resources (annual series)
|
From page 111... ...
A second source of funding is from the business/industry sector. As noted in the 2013 National Research Council report, "NCSES assumes that the annual growth in funding from industry to nonprofit organizations changes in constant proportion to 1 Organizations reporting on these surveys are unlikely to convert and report the value of volunteer labor as an R&D expenditure.
|
From page 112... ...
. Data from National Science Foundation, National Center for Science and Engineering Statistics, National Patterns of R&D Resources (annual series)
|
From page 113... ...
Thus, NCSES would like to improve the way that it estimates R&D in the nonprofit sector for the years in which it will not be conducting the survey. Other Potential Outputs Feller initiated a discussion of the ways in which the data from the NSF Nonprofit R&D Survey could be used beyond the National Patterns of R&D Resources.
|
From page 114... ...
David also noted that the techniques that NSF is developing to learn about R&D in the nonprofit sector will have implications relevant to measuring the activities of the service sector beyond the nonprofit arena, increasing their overall impact. Salamon expressed his interest in expanding the scope of the NSF nonprofit survey by exploring ways to marshal external funding for NSF to enable expanding the pool of organizations surveyed, or by conducting a broader survey with a research team outside of NSF following parallel methods so that the data could be used together for NSF's purposes, as well the purposes of other stakeholders.
|
From page 115... ...
. Those respondents said that they would be interested in • seeing the data presented by tax exempt status and field of research; • learning whether other nonprofit organizations faced similar restrictions on annual funding of research; • identifying other nonprofits doing work in the same field, and those of the same size, to determine whether they were duplicat ing any efforts; • learning about recovered indirect cost allocation; and • knowing the geographic location of peer organizations.
|
From page 116... ...
He proposed that while recognizing the present planned survey has resource constraints and limitations, NSF could build in exploratory modules that serve as springboards to the nonprofit sector itself, other government agencies, or outside research groups to initiate a larger study of the sector, which he said seems necessary. In addition, Feller indicated that the National Academies could play a role in bringing together "a national forum on this type of issue, which serves the primary purpose of bringing visibility to an area and an issue that really is not on anybody's main agenda." CHAPTER SUMMARY Overall, participants discussed ways to design the questionnaire, plan for successful implementation, and achieve good response rates.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.