Each of the four surveys uses this system, either exclusively or for the majority of its cases. In recent years the survey has been delivered to the respondents electronically, rather than in paper format through the mail. The respondent uses the electronic survey form, and ORC Macro follows up by e-mail or telephone. It keeps logs, generated by the system, of the progress of the agencies or institutions in reporting. The web-based system contains embedded edits, questioning the entering of erroneous data. Respondents are asked to check the data, repair arithmetical errors, and explain trend differences. In some of the surveys, data cannot be submitted to the National Science Foundation (NSF) until all error messages have been dealt with.

Not all respondents submit data on the web-generated form, so that some of the surveys have to maintain a manual editing system. The web-based system is also used to tabulate data.

There are many advantages to a web-based system. It should reduce the amount of error caused by carelessness or not having data from prior years to check against. It allows NSF to know which questions are giving respondents difficulty, since one can keep track of edit failures. It speeds up the processing. Also, there is complete knowledge of the status of every survey participant throughout the delivery, edit, and imputation stages.

There are a few disadvantages to the system. In some cases, it forces respondents to provide data that they are unsure about. In this sense, it forces imputation by the respondent, rather than by NSF. When a statement is made in NSF publications that there is no item nonresponse, it conveys an impression of accuracy that may very well be misleading. Another disadvantage seems to be an inability to deal with weighted data, although that could probably be overcome. However, these four surveys have fewer than 1,000 respondents and no sampling, no nonresponse adjustment by weighting, and no other weighting is done.

The industry survey is primarily a mail survey, although the forms have been put on the web. It would benefit this survey to have a well-researched form designed for the web, with the appropriate embedded edits. It would facilitate the calculation of response rates and give indications of which items were causing difficulty to respondents. It would also speed up the processing. Since this survey is many times larger than the other four, a more streamlined procedure would save considerable amounts of time and money.

Some Concerns About the Surveys

Before each of the surveys is reviewed separately in this appendix, I raise some concerns that are common when these five surveys are looked at as a package.

One concern is that of providing prior-year data to respondents. On the industry survey, prior-year data are printed on the form. For the other surveys, prior-year data are available to respondents. Although many survey researchers believe that having prior-year data improves the quality of response by prompting respondents, others believe that it gives an easy way out. Research is needed on this issue.

Cutoff amounts are often used to define the survey universe. For example, academic institutions with less than $150,000 in R&D expenditures in the previous year

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement