Appendix A
Interview and Focus Group Guides



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Appendix A Interview and Focus Group Guides

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Surveying Resources for Science and Engineering: Assessing the Sciences Resources Studies (SRS) Portfolio

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies INTRODUCTION. (Read to the interviewee:) We're interested in asking you about the SRS surveys you work with and how well you think they address important policy issues and research questions involving science resources. The topics we want to cover are summarized below. Please be assured that your replies will be confidential—we will not make available outside our study any information that would link your name with your responses. Question Area 1. SURVEY CONTENT. (Read to the Interviewee:) We're interested in your assessment of the relevance of the survey content to important policy and research issues. (Ask for each survey:) Which policy and research issues do the questions on your survey(s) address and which issues are most important in your view? Are different/additional survey questions needed to address these issues more fully? What other current or emerging policy and research issues might your survey address that it does not address now? Could the survey add useful questions on these topics? What would be the survey questions/topics you would suggest adding, in priority order? Would these questions/topics need to be added permanently or just one-time? Conversely, are there questions that the survey could drop because they're obsolete? Question Area 2. SURVEY QUALITY. (Read to the interviewee:) We're interested in your assessment of the quality of the survey along several dimensions and the possible effects of quality concerns on the policy and research uses of the data. Areas of quality concern include the definition of the target population, coverage of the population, sample design, nonresponse, response errors, frequency of data collection, and timeliness of data products that may affect the use of the data for policy or research. (Ask for each survey:) What quality concerns or "problems" affect the surveys you work with and the data they obtain? How severe are these problems in your view? What methodological or operational improvements would you suggest implementing for the survey that would improve its usefulness, in priority order? Question Area 3. FIT WITH OTHER SURVEYS. (Read to the interviewee:) We're interested in your assessment of how well the survey fits with other surveys carried out by SRS or other agencies.

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies (Ask for each survey:) In what ways could coordination of the survey with other surveys on science and engineering resources be improved? Question Area 4. DATA USERS AND DATA USES. (Read to the interviewee:) We're interested in your assessment of the audience for the survey and the publications and other products based upon them. (Ask for each survey:) Who uses the survey? Are there different types of users for the survey? How well does the survey serve the needs of these different types of users in terms of content, accuracy of data, level of detail, timeliness, and forms of dissemination? What improvements would you suggest implementing, in priority order, that would make the data more useful to current users? . . .to users not now being reached? Question Area 5. USE OF DATA IN PUBLICATIONS AND ANALYSES. (Read to the interviewee:) We're interested in your assessment of how well the data from the survey are used in SRS publications, including Science and Engineering Indicators. (Ask for each survey:) Are the data used descriptively? . . .analytically? Are the data analyzed appropriately? What additional analyses would you suggest, in priority order, that SRS should conduct and publish of the data? Question Area 6. OVERALL ASSESSMENT OF SRS'S PROGRAM. Finally, we're interested in other comments you may have on SRS's program and how well it meets users' needs for data for policy analysis and research.

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Surveying Resources for Science and Engineering: Assessing the Sciences Resources Studies (SRS) Portfolio

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies INTRODUCTION. (Read to the interviewee:) We're interested in asking you about the kinds of policy analysis or research you conduct on science and technology resources issues, the data you use, and their adequacy for your analysis purposes. We're particularly interested in your views about the usefulness of data from the National Science Foundation's Science Resources Studies Division (which we will refer to as SRS), but if you use data from other sources, we would like to hear about those data, too. Please be assured that your replies will be confidential—we will not make available outside our study any information that would link your name with your responses. Question Area 1. POLICY/RESEARCH ISSUES AND DATA SOURCES. (Read to the Interviewee:) We're interested in knowing more about the policy or research issues you work with and the data sources you use. (Ask:) Please tell us the key policy and/or research issues that you are interested in and work on. (If obtained prior to the interview, ask the interviewee to elaborate as necessary and possible). What kinds of data do you use to work on these issues and what are their sources? (If a list of data sources was obtained prior to the interview, ask the interviewee to elaborate as necessary and possible). From among the data sources you tap, which SRS surveys—and which summaries and tables within publications using SRS data—address policy or research issues that are important to your work and which issues? If you don't use SRS data, why not? (Probe to see if s/he uses SRS data but doesn't know it.) Question Area 2. SURVEY CONTENT. (Read to the Interviewee:) We're interested in your assessment of how relevant existing data on science and technology resources—from SRS and other sources—is to your analysis and research. (Ask:) What is your assessment of the overall content of the surveys that you use and its relevance to your policy or research issues? Do these surveys present data at the level of detail you require? What current or emerging policy and research issues you are concerned about are not addressed by the SRS and other surveys that you use? Could useful survey questions on these topics be added to these surveys or a new survey? If so, what would be the survey questions/topics you would suggest adding to which surveys, in priority order? Would these questions/topics need to be added permanently or just one-time?

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Are there questions that any of these surveys could drop because they're obsolete? Question Area 3. SURVEY QUALITY. (Read to the interviewee:) We're interested in your assessment of the quality of the survey data you use along several dimensions and the possible effects of quality problems on the policy and research uses of the data. Areas of quality concern that may affect the use of the data for policy or research include the definition of the target population, coverage of the population, sample design, nonresponse, response errors, frequency of data collection, or the timeliness of data products. (Ask:) What quality problems affect the survey data you work? How severe are these problems in your view? What methodological improvements would you suggest implementing for the surveys that you use that would improve their usefulness—in priority order? Question Area 4. FIT WITH OTHER SURVEYS. (Read to the interviewee:) We're interested in your assessment of how well data from the SRS surveys that you use fit with other surveys conducted by SRS or other agencies and organizations. (Ask:) In what ways could coordination of SRS surveys with other surveys be improved to provide data that can be integrated in a way that would be useful in your work? Do you see any unnecessary overlaps or redundancies in SRS surveys and other surveys you use? Are there any specific inconsistencies between SRS and other surveys in the ways that data are collected that make the data from these surveys difficult to integrate and use fully? Question Area 5. ACCESS TO DATA AND PUBLICATIONS. (Read to the interviewee:) We're interested in how well you are able to access the data and publications you need for your work and your assessment of the publications you use. (Ask for each survey used by the interviewee:) How do you typically access data from SRS and other sources? (via the web, data files, customized tables, off-the-shelf publications) Are you able to access data from these sources in an effective way? Do you find descriptive and analytical presentation of data in SRS and other publications useful and appropriate? Would different/additional summaries, tables, or analyses be useful in addressing the policy or research issues you work on? Please specify, in priority order, what SRS and other sources might add.

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Question Area 6. FUTURE DATA NEEDS. (Read to the interviewee:) We're interested in the science and technology issues you see on the horizon that may require different or additional data. (Ask for each survey used by the interviewee:) What new science resources issues do you see on the horizon that may require different or additional data? What types of data do you anticipate will be required? Do you anticipate that existing surveys can address these data needs or will these new issues require additional surveys to address them? What immediate or long-term changes would you propose, in priority order? Question Area 7. OVERALL ASSESSMENT OF SRS'S SURVEY AND PUBLICATIONS PROGRAM. We're interested in other comments you may have on SRS's program and how well it meets your needs for data for policy analysis and research.

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Surveying Resources for Science and Engineering: Assessing the Sciences Resources Studies (SRS) Portfolio Part C: Questions for SRS Information Services Group INTRODUCTION (Read to the interviewee:) We're interested in asking you about SRS data and their collection, analysis, and dissemination. In particular, we want to know how well you think the data, analyses, and publications serve the public, policymakers, and researchers who are interested in science resources. Please be assured that your replies will be confidential—we will not make available outside our study any information that would link your name with your responses. Question Area 1. DATA USERS AND DATA USES. (Read to the interviewee:) We're interested in your assessment of the audience for SRS data and other data on science and engineering resources: Who uses SRS data? What different types of users can you describe? Who doesn't use SRS data that, in your opinion, should be reached? Why aren't they reached—why don't they use SRS data? How well does SRS data serve the needs of these different types of users in terms of content, accuracy of data, level of detail, timeliness, and forms of dissemination? How do they obtain SRS data—publications, tables, special requests, web usage, etc.? How is access to data an issue? What improvements would you suggest implementing that would make the data more useful to current users? Different data? Different analyses? Different means of accessing data? Question Area 2. USE OF DATA IN PUBLICATIONS AND ANALYSES. (Read to the interviewee:) We're interested in your assessment of how well the data from the survey are used in SRS publications, including Science and Engineering Indicators. For those users who do access SRS data in one form or another, are they finding what they want? (i.e., raw data, data tables, brief analyses, charts and graphs, analytical reports, etc.? What different or additional uses of the data—or certain data sets—would you suggest, in priority order, that SRS consider?

OCR for page 117
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Question Area 3. SURVEY CONTENT. (Read to the Interviewee:) We're interested in your assessment of the relevance of the survey content to important policy and research issues. To what extent can you identify areas in which the relevance or currency of certain SRS data are issues in their use? Which policy and research issues do users find information on? Which issues do they not find information on? Are different/additional survey questions needed to address these issues more fully? What other current or emerging policy and research issues on science and engineering resources might SRS other data sources that it does not address now? Could useful survey questions be added on these topics? What would be the survey questions/topics you would suggest adding, in priority order? Would these questions/topics need to be added permanently or just one Conversely, are there questions that the survey could drop because they're obsolete? Question Area 4. SURVEY QUALITY. (Read to the interviewee:) We're interested in your assessment of the quality of the survey along several dimensions and the possible effects of quality concerns on the policy and research uses of the data. Areas of quality concern include the definition of the target population, coverage of the population, sample design, nonresponse, response errors, frequency of data collection, and timeliness of data products that may affect the use of the data for policy or research. What quality concerns or ''problems" affect SRS data you work with? How severe are these problems in your view? What methodological or operational improvements would you suggest implementing for the survey that would improve its usefulness, in priority order? Question Area 5. FIT WITH OTHER SURVEYS. (Read to the interviewee:) We're interested in your assessment of how well SRS surveys fit with each other and with surveys carried out by other agencies. In what ways could coordination of SRS surveys with each other and other surveys on science and engineering resources be improved? Question Area 6. OVERALL ASSESSMENT OF SRS'S PROGRAM. Finally, we're interested in other comments you may have on SRS's program and how well it meets users' needs for data for policy analysis and research.