Measuring Research and Development Expenditures in the U.S. Nonprofit Sector Conceptual and Design Issues: Summary of a Workshop (2015) / Chapter Skim
Currently Skimming:
4 Approaches to Survey and Sample Design
4 Approaches to Survey and Sample Design
Pages 55-84
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 55... ...
Survey: understanding the diverse and unique nature of R&D in the nonprofit sector; using the correct language for communication about R&D; accounting for the interconnections among nonprofits; identifying the correct respondents; and understanding the financial and labor resources within nonprofits. With those challenges in mind, this chapter summarizes presenters' and participants' guidance on design and measurement for the proposed survey of nonprofits from five sources: the Frascati Manual, international experiences, the 1996–1997 NSF survey of nonprofits, a discussion with sampling experts, and an overview of data sources available for creating a sample frame.
|
From page 56... ...
are available to assist with identifying the nonprofit population and developing a sampling frame. • Narrative data about program accomplishments from nonprofits' IRS Form 990s may be used to identify key words that nonprofit organizations use to describe R&D activities.
|
From page 57... ...
was changed. According to the Frascati Manual, the PNP sector includes "non-market, private non-profit institutions serving households (i.e., the general public)
|
From page 58... ...
GUIDANCE: INTERNATIONAL EXPERIENCES The approaches of other countries to measuring R&D in the nonprofit sector can inform the design phase of the NSF Nonprofit R&D Survey. Thus, the workshop included presentations describing the experiences of Italy and the UK within a European context, as well as a presentation about Canada's approach.
|
From page 59... ...
Geuna explained that the UK PNP sector consists primarily of charities, trusts, and medical research. Wellcome Trust is the single largest foundation in the UK and is particularly important as a large cancer research charity.
|
From page 60... ...
TABLE 4-1 R&D Funded by Nonprofits, in Units of Purchasing Power Standard per Inhabitant at Constant 2005 Prices Country 2003 2004 2005 2006 2007 2008 2009 2010 2011 European Union (28 countries) 5.4 5.6 5.8 6.5 6.8 7.1 7.0 7.1 7.2 Euro Area (17 countries)
|
From page 61... ...
The screening resulted in a target population of 200 organizations that performed intramural, extramural, and/or nonmarket research for the main survey. That survey was a census of the entire target population.
|
From page 62... ...
Geuna noted, "In Italy, building this kind of list of organizations is difficult, but we have a specific income tax law that allows us to identify research-performing nonprofit institutions." Additionally, a census of all institutions was conducted in 2012. An initial population of approximately 600 organizations was identified for a screening survey to help surveyors ascertain whether the institutions were conducting research according to the Frascati Manual.
|
From page 63... ...
Now we have computer scientists and engineers collaborating in the production of intangible new digital services." Geuna said that he believes the definitions of R&D will need to be updated beyond the ways in which they are articulated in the 2002 version of the Frascati Manual. The traditional way of identifying what is included in "service" in R&D described in the manual is outdated and 2 According to the European Public Sector Innovation Scoreboard website, "Following the Europe 2020 Innovation Union flagship initiative, the European Commission launched a pilot European Public Sector Innovation Scoreboard (EPSIS)
|
From page 64... ...
Carol House, Committee on National Statistics and study director, presented information about the Canadian process on behalf of Louise Earl of Statistics Canada. She said that the results from the survey are published annually and used as a key component in the Canadian series Gross Domestic Expenditures on Research and Development.3 Although the total nonprofit sector in Canada is sizable and accounts for up to 12 percent of the country's total workforce, Statistics Canada follows the Frascati guidance to define a population of approximately 140 nonprofits with R&D activities4 -- consisting of private philanthropic foundations, voluntary health organizations, associations, societies, and research institutes -- and conducts a census of these organizations.
|
From page 65... ...
Among the challenges that Statistics Canada faces in collecting data on R&D from nonprofit organizations are difficulties in frame construction -- both identifying private nonprofit organizations with R&D
|
From page 66... ...
Perhaps, she posited, Statistics Canada includes only those likely to be conducting R&D in the annual census. Salamon emphasized that this number of nonprofit organizations did not constitute the entire number of nonprofits in Canada.
|
From page 67... ...
Potential units are identified based on media, responses to related science and technology surveys that identify recipients of R&D funds, other Statistics Canada surveys, associations of nonprofit organizations, taxfiling information, registration of private nonprofits and charities with the Canada Revenue Agency, Internet sources, and other aspects. Subject matter officers in Statistics Canada then review potential candidate organizations based on R&D activity -- funding R&D only, performing
|
From page 68... ...
He suggested reflecting on previous iterations is the way that most repetitive surveys improve over time, but typically the iterations are far more frequent than 18 years apart. During the last survey, NSF screened 9,112 nonprofit organizations to
|
From page 69... ...
Those decisions then inform the ideal sample design for those purposes, he said. Fecso also offered suggestions for NSF to consider in the planning phase of the new survey, while acknowledging that the design process is already well under way.
|
From page 70... ...
Fecso closed by noting that the information quality guidelines of NSF, the statistical standards from the Office of Management and Budget, and expertise from survey design experts all will provide NSF with a great deal of instructions and guidance about the basic elements of the design. GUIDANCE: OVERVIEW FROM SAMPLING EXPERTS Michael Larsen, associate professor in the department of statistics at George Washington University, and Phillip Kott, senior research statistician at RTI International, discussed five major activities involved in developing an effective and efficient sample for the NSF Nonprofit R&D Survey, as well as issues tied to these activities: identifying the overall population and the target population; establishing a sampling frame and stratifying the sample; considering various sampling ideas to increase the efficiency of the sample; identifying helpful auxiliary information to gather; and making use of pilot study data to develop an adaptive sam 5 In statistical testing, type I error is the incorrect rejection of a true null hypothesis.
|
From page 71... ...
; the section on "Data Sources Available for Creating a Sampling Frame" later in this chapter provides a detailed discussion of sources of data on nonprofit organizations that could be used to form a sample frame for the survey. Larsen said he agrees with Fecso that making use of existing datasets
|
From page 72... ...
"Not doing a one size fits all approach for every strata or every size category is something that could maximize the benefit of your sample," stated Larsen. 6 Strataare distinct groups or subpopulations identified on the sampling frame that can be sampled separately for the purpose of efficiency or to ensure that each subpopulation is covered by a minimum sample size.
|
From page 73... ...
The auxiliary data, he noted, will be important in defining the probabilities of selection for each organization on the sampling frame. This will be the case for defining strata and for sampling from those strata by systematic or PPS-based processes.
|
From page 74... ...
With the large database that NSF will have, he suggested experimenting with some fairly sophisticated techniques as part of the pilot test to try to find a good replacement for missing data and to rely less on statistical models to do imputation. Larsen said useful auxiliary information could also be gathered through a two-phase sampling approach.
|
From page 75... ...
Even in the absence of this approach, Larsen stated, the pilot data will be useful for studying the stratification plan, quality of segment classifications, response rates, and missing rates. In addition, the data from the pilot study will be useful for identifying variables correlated with the outcome that can be used to adjust sampling plans, develop models for missing data, and conduct estimations.
|
From page 76... ...
Nathan Dietz, National Center for Charitable Statistics, recommended excluding child organizations in cases where it is possible to establish that the parent organization is responsible for research, particularly if there is interest in obtaining data from smaller nonprofits that are likely not to be huge R&D producers. GUIDANCE: DATA SOURCES AVAILABLE FOR CREATING A SAMPLING FRAME Paul Arnsberger, a senior statistician with the Statistics of Income (SOI)
|
From page 77... ...
Arnsberger focused his remarks on three public micro-datasets that SOI makes available -- the Exempt Organizations Business Master File (EO BMF) , the Exempt Organizations Financial Extract, and the Exempt Organizations Sample Studies (see Box 4-3)
|
From page 78... ...
Exempt Organizations Financial Extracts The exempt organizations financial extracts are created annually and include data from those currently active organizations from all subsection codes that file IRS Forms 990, 990-EZ, or 990-PF, Arnsberger explained. Another division in IRS completes the data transcription for the extract for their administrative purposes and provides the file to SOI.
|
From page 79... ...
The EO financial extracts currently contain just fewer than 610,000 records. Exempt Organizations Sample Studies Arnsberger described three files that SOI creates by selecting samples of exempt organizations called the EO sample studies.
|
From page 80... ...
TABLE 4-3 Distribution of Records in Exempt Organizations Sample Studies by Source Form, Tax Year 2010 Form Number of Records Number of Fields Financial Fields (%) 990 19,476 856 70 990-EZ 1,260 202 65 990-PF 10,898 308 69 Total 31,634 1,366 69 SOURCE Arnsberger and Dietz (2014)
|
From page 81... ...
NCCS Data Products The NCCS core file is produced annually by NCCS itself. This file contains more variables than does the EO BMF, but fewer variables than do the EO sample studies.
|
From page 82... ...
We extract that information, and have developed a nice little database of programs that nonprofit organizations run that we classify using a Nonprofit Program Classification system we developed." Dietz suggested that the most relevant information for NSF to use in putting together the sampling frame for the NSF Nonprofit R&D Survey is from Part III of IRS Form 990, Program Service Accomplishments. Dietz drew attention to question 4 in this section of the form (see Appendix C)
|
From page 83... ...
The numbers of various types of organizations that are included will depend on the allocations assigned to them, he explained. Jankowski explained that the data mining technique that NCCS plans to conduct is just one potential technique that may be used when the actual sample is drawn.
|
From page 84... ...
The presentations and discussions summarized in this chapter yielded a number of suggestions for designing the sample for the NSF Nonprofit R&D Survey, while acknowledging that significant challenges remain. Many participants emphasized using the pilot and design phase wisely to maximize the efficiency of the sample and to make preparations for successful implementation.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.