Measuring Research and Development Expenditures in the U.S. Economy Interim Report (2004) / Chapter Skim
Currently Skimming:
Appendix A: A Quality Profile of the Federal Research and Development Surveys
Appendix A: A Quality Profile of the Federal Research and Development Surveys
Pages 24-67
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 24... ...
They all use ORC Macro as a contractor and a similar data collection and editing methoclology. The industrial survey is much different, is much more complex, uses the Census Bureau's Business list, formerly known as the Standard Statistical Establishment List (SSEL)
|
From page 25... ...
It Louis benefit this survey to have a well-researched form designed for the web, with the appropriate embeddect edits. It would facilitate the calculation of response rates and give indications of which items were causing cliff~culty to respondents.
|
From page 26... ...
It is the institutional coordinator who oversees the entire effort and clecicles what other people need to provide, what sources of data to use, who coordinates the entire effort, then reviews the data. At the other extreme, the Survey of Industrial Research and Development is mailed to a company, with no contact person designated and no overt recognition of the possible ctifficulty of responding.
|
From page 27... ...
Coverage would be deficient if the frame did not include all the relevant federal agencies and subagencies that fund R&D activities. However, agencies are i(lentifiecl by using such sources as the president's annual report, OMB budget documents, and respondent agencies.
|
From page 28... ...
The system consists of a data collection component, which allows survey respondents to enter their data online, ant! a monitoring component, which allows ORC Macro to monitor support requests, data entry, and data issues.
|
From page 29... ...
There is no known item nonresponse, since agency respondents must answer the questions before the data can be submitted. Appendix A-6
|
From page 30... ...
. agency respondent for the purvey ot federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, because many agencies cannot report until those data are available.
|
From page 31... ...
and is the only source of comprehensive data on federal science and engineering support to individual academic and nonprofit institutions. Federal policy makers, state and local government officials, university policy analysts, R&D managers, and nonprofit institution administrators use it.
|
From page 32... ...
to report such data on the Survey of Fecleral Science and Engineering Support to Universities, Colleges, ant! Nonprofit Institutions.
|
From page 33... ...
The key variables are: Academic institution Geographic location (within the United States) · Highest degree granted Historically black colleges and universities Appendix A- ~ O
|
From page 34... ...
No data are available on the extent to which federal agency respondents check their codes with universities. Other problems discussed in the methodology report (National Science Foundation, 2001)
|
From page 35... ...
SURVEY OF RESEARCH AND DEVELOPMENT EXPENDITURES AT UNIVERSITIES AND COLLEGES Introduction Objectives Appendix A-12
|
From page 36... ...
and the NSF Survey of Fecleral Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions populations, institutions with a highest degree-granting status of ~ (cloctorate-granting) are compared to make sure all science and engineering doctoral degree-granting institutions are inclucle`1 in the academic R&D expenditures survey population.
|
From page 37... ...
An explanation of the survey and survey population and information on downloading a paper copy of the survey form from the survey web site was includecl. An acknowledgment of receipt and expected completion ciate were requested.
|
From page 38... ...
Respondents who still refused were thanked} for their time and informed that NSF personnel might contact them in the future. The Survey Respondent Nothing was inclucled in the methodology report (National Science Foundation, 2002b)
|
From page 39... ...
of item nonresponse rates, although data for missing items was imputed. Potential Sources of Error in the Data Collection Procedure Little is known about the mechanics of response, including the reaction to the reminders.
|
From page 40... ...
_~~ -ear ~~ ^ Hi ~~ Aim mu ~~~ .~_v. , ~^ ~ J _ , An institutional response code changes the status of institutions from "awaiting response" to "waiting processing." After data entry, the institution response code becomes one of the following: Awaiting correction: arithmetical error requiring correction by sending an "eclit letter." Awaiting verification: significant trencts in data that require confirmation by sending an edit letter.
|
From page 41... ...
for any item nonresponse. Imputation rates for all data cells were calculated for all universities and colleges and for various institution classes, determined by highest degree grantee!
|
From page 42... ...
to be outliers. Fecleral basic research amounts and total basic research amounts were imputed for all non-FFRDC institutions that met the following criteria: Appendix A- ~ 9 .
|
From page 43... ...
Data for the years in which no response was received were imputed. For each institution, key variables that were formerly imputed were compared with subsequent submissions to determine whether the imputed data accurately represented the growth patterns shown by the reported data.
|
From page 44... ...
Not all academic institutions are included. Only those granting master's or doctoral degrees in science and engineering or other institutions that reported R&D expenditures of $150,000 or more, as well as all historically black colleges anc!
|
From page 45... ...
A paper and a web-basecl survey were both prepared for the 2001 survey asking for the amount of science and engineering research space and the adequacy of the space. Don Diliman, an expert on questionnaire design who suggested rewording of instructions ant!
|
From page 46... ...
Questions asked are: · The physical infrastructure used for network communications · Plans for future upgrading and uses of information technology Capacity for high-speec! computations Infrastructure for wireless communication Survey Nonresponse Response rates were 90 percent for the academic institutions and 88 percent for the biomedical institutions.
|
From page 47... ...
-- I -- - r ~7 A hypothesis for the lower response rates for biomedical institutions is that these organizations are less likely than universities to have already collected these data in some form for a purpose other that responding to the survey. Also, these institutions are generally smaller than academic institutions and may have receivect only one small grant from NTH.
|
From page 48... ...
The other items imputed for academic institutions were: Total amount of S&E instruction space Total amount of non-S&E instruction space Amount of S&E research space in individual fields of science Amount of S&E instructional space in individual fields of science These items were imputed by forming totals for the respondents and then ratios. such as the ratio of total S&E instruction space to total S&E research space, and then applying the ratio to the aIreacly-imputecl total amount of research space.
|
From page 49... ...
The survey is sponsored and funded by NSF and is carried out by the U.S. Census Bureau.
|
From page 50... ...
The Census Bureau's Standard Statistical Establishment List was used. This list, now known as the Business List, is upciatecl annually and contains all nonfarm entities that the Census Bureau knows about.
|
From page 51... ...
In 1993 Kusch and Ricciardi undertook a study at the Census Bureau to measure the extent of changes in company codes. That report shows that there is a high clegree of consistency in coding (U.S.
|
From page 52... ...
In 2002, there were two broad groups, based on what was known about the company's R&D expenditures from the previous data of the last four years. The first group was the "knowns"; it includecl companies that had reported R&D expenditures at least once in the years 1998-2001: the second group was the "unknowns." The reasons for this change were to improve the state estimates, to take advantage of historic data available in the Census Bureau's processing program, and to improve the industry-level estimates.
|
From page 53... ...
A Census Bureau document by W Ricciardi describes the clevelopment of the measures of size ant!
|
From page 54... ...
Data Collection Basic Data Collection Procedure The NSF staff along with the Census Bureau staff has met with respondents to determine the sources of data used by the respondents to complete the questionnaire, to examine how the respondents made estimates, and to identify problems encountered by the respondents. As a result of these "response analysis" studies, questionnaire items have been removed, items have been simplified, and additional instructions written for Appendix A-3 ~
|
From page 55... ...
~ 1 Telephone follow-up was used to encourage response from companies among the 300 largest R&D performers, based on total R&D expenditures in the previous survey. The R&D survey analysts at the Census Bureau made the telephone follow-up calls.
|
From page 56... ...
The Census Bureau sloes not mail to a specific person, although a person's name may be available to them from a previous cycle. Company spokespersons say that it is often left to mailroom personnel at the company to decide to whom to deliver the form.
|
From page 57... ...
The recommendations of the authors to alleviate these problems are as follows (U.S. Census Bureau, ~ 995b)
|
From page 58... ...
support staff positions are not equivalent to a college clegree (U.S. Census Bureau, ~ 995b)
|
From page 59... ...
. , ~ ~ With the amputator rates as a poor proxy for item nonresponse rates, one can see that even for items that are frequently reported in many federal surveys, net sales and total employment, the imputed ratios can be very high.
|
From page 60... ...
Another Census Bureau analyst, D Boncl, also documents some of the nonsampling errors arising in this survey (U.S.
|
From page 61... ...
Most of the discussion in this paper relies on those memoranda. Keying of Data According to BoncI, the Data Processing Division at the Census Bureau keys the data according to well-documentec]
|
From page 62... ...
Boncl clescribed the imputation procedure (U.S. Census Bureau, ~994c)
|
From page 63... ...
on companies as publicly owned can be used to match domestic sales, domestic employment, total or company-funded R&D, anct, in some cases, feclerally funded R&D and then impute data. The Census Bureau's SSEL can also be used for verifying and imputing domestic employment and domestic sales data.
|
From page 64... ...
First, the top 50 companies in a state, as measured by payroll, are included in a certainty stratum. Thus, these larger firms remain in the sample from one year to the next.
|
From page 65... ...
industry over all states. For example, State A may have 2 percent of the payroll in a given industry.
|
From page 66... ...
Census Bureau, Washington, DC. 1994cl A Survey of Processing Errors in the Survey of Industrial Research and Development.
|
From page 67... ...
Washington, DC: U.S. Census Bureau.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.