EXECUTIVE SUMMARY

Background

Interest in predicting demand and supply for doctoral scientists and engineers began in the 1950s, and since that time there have been repeated efforts to forecast impending shortages or surpluses. As the importance of science and engineering has increased in relation to the American economy, so has the need for indicators of the adequacy of future demand and supply for scientific and engineering personnel. This need, however, has not been met by databased forecasting models, and accurate forecasts have not been produced.

Forecast error may proceed from many sources. Models may be based on incorrect assumptions about overall structure, included variables, lag structure, and error structure. Data used for estimation may be flawed or aggregated at an inappropriate level. Further, unanticipated events beyond those considered in the model may occur that could ruin the accuracy of even the best forecasts. As Leslie and Oaxaca (1993) have described in their thorough review article, virtually all models of demand and supply have been flawed by at least one (and, in many cases, all) of these problems.

In order to assess the methodology of forecasting the demand and supply of doctoral scientists and engineers, the National Science Foundation (NSF) and Sloan Foundation funded the National Research Council to assemble a committee of experts for a workshop on the topic. The task of the committee was not to find fault with past efforts, but to provide guidance to the NSF and to scholars in this area about how models (and the forecasts derived



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
FORECASTING DEMAND AND SUPPLY OF DOCTORAL SCIENTISTS AND ENGINEERS: Report of a Workshop on Methodology EXECUTIVE SUMMARY Background Interest in predicting demand and supply for doctoral scientists and engineers began in the 1950s, and since that time there have been repeated efforts to forecast impending shortages or surpluses. As the importance of science and engineering has increased in relation to the American economy, so has the need for indicators of the adequacy of future demand and supply for scientific and engineering personnel. This need, however, has not been met by databased forecasting models, and accurate forecasts have not been produced. Forecast error may proceed from many sources. Models may be based on incorrect assumptions about overall structure, included variables, lag structure, and error structure. Data used for estimation may be flawed or aggregated at an inappropriate level. Further, unanticipated events beyond those considered in the model may occur that could ruin the accuracy of even the best forecasts. As Leslie and Oaxaca (1993) have described in their thorough review article, virtually all models of demand and supply have been flawed by at least one (and, in many cases, all) of these problems. In order to assess the methodology of forecasting the demand and supply of doctoral scientists and engineers, the National Science Foundation (NSF) and Sloan Foundation funded the National Research Council to assemble a committee of experts for a workshop on the topic. The task of the committee was not to find fault with past efforts, but to provide guidance to the NSF and to scholars in this area about how models (and the forecasts derived

OCR for page 1
FORECASTING DEMAND AND SUPPLY OF DOCTORAL SCIENTISTS AND ENGINEERS: Report of a Workshop on Methodology from them) might be improved and what role NSF should play in their improvement. Another issue for the committee was the responsible reporting of forecasts to policymakers. Virtually no forecast is error-free, and some uncertainty is always associated with them. Policymakers who use the results of forecasting are usually not technically proficient in the arcana of the forecaster's art. If forecasts are to be used responsibly, policymakers need to be informed about the assumptions upon which the forecasts rest, and forecasters need to track the validity of their assumptions and the accuracy of their forecasts over time. The committee then assessed the information on these issues provided at the workshop as well as in the forecasting literature and arrived at the following recommendations. Recommendations The forecasting process is ongoing. Forecasters must learn from their mistakes. The whole forecasting exercise needs to be placed within an administrative framework that facilitates an evaluation process and a process to correct errors. The Science Resources Study Division (SRS) of NSF is the locus of action in the federal government to bring about the improvement of data and forecasts of the market for doctoral scientists and engineers. The committee has therefore directed most of its recommendations to SRS, since this division should be able to encourage improvements in the construction and use of forecasting models for highly trained scientists and engineers, even when SRS should not carry out the work itself. Recommendation 1. The producers of forecasts should take into account the variety of consumers of forecasts of demand and supply for scientists and engineers.

OCR for page 1
FORECASTING DEMAND AND SUPPLY OF DOCTORAL SCIENTISTS AND ENGINEERS: Report of a Workshop on Methodology NSF should recognize that there are five distinct communities of clients for data and projections on the supply and demand of scientists and engineers, each with different needs and interests. NSF's data collection and forecasting activities should keep the needs of these different communities in mind. They are: students making career decisions, federal, state, and private funding agencies, industrial and academic employers of scientists, Congress in its role as funder and policymaker, and the scientific community that conducts research and produces studies of the market and its participants. Representatives of all these user communities spoke at the workshop. All expressed dissatisfaction with the current state of data and forecasting. Students need qualitative projections of likely career outcomes and probabilities of success, with particular attention to the state of the job market in a few years when they will seek employment. These qualitative projections require timely data, but they need not be based on broad surveys or censuses. Research funding agencies and Congress need relatively long-term projections of supply and demand factors by specific discipline that can be used to guide policy on training support and institutional development. These projections should take careful account of the ease with which one kind of labor can be substituted for another and the incentives and behavioral responses that operate. The importance of contingencies and of forecast uncertainty should be emphasized. The needs of employers are varied, but they also benefit from an early warning about shortages. As producers of Ph.D.s, academic institutions can take steps to expand or contract Ph.D. enrollment given convincing evidence of emerging labor market trends.

OCR for page 1
FORECASTING DEMAND AND SUPPLY OF DOCTORAL SCIENTISTS AND ENGINEERS: Report of a Workshop on Methodology Finally, the forecasting community should be more forthcoming about the appropriate use of forecasts and the nature of their underlying assumptions. Recommendation 2. The NSF should not produce or sponsor “official” forecasts of supply and demand of scientists and engineers, but should support scholarship to improve the quality of underlying data and methodology. NSF should not produce or sponsor “official” forecasts of supply and demand in the markets for scientists and engineers, but should continue to take the lead in collecting and making available data on these markets. A clear organizational separation should be made between data collection and modeling/forecasting activities undertaken for NSF's own policy use or for use by federal agencies. For example, convert the SRS into a National Center for Science Statistics on the model of the National Center for Health Statistics (NCHS), the National Center for Educational Statistics (NCES), or the Energy Information Agency (EIA) and remove modeling and forecasting activities to a separate policy unit. Or, for example, forecasts could be produced by an outside agency with statistical expertise. Agencies such as the Bureau of the Census or Bureau of Economic Analysis may be well suited to undertake such forecasts. If asked to produce forecasts of scientific and engineering personnel for its own use or the use of other agencies, the NSF policy unit should avoid endorsing or emphasizing “gap” models that do not incorporate behavioral adjustment to demand and supply and consequently may give unwary users a misleading impression of likely market outcomes. NSF should avoid suggesting that there is a single best level of detail and model complexity for the forecasts needed by various users and should instead maintain that model structure will depend on user needs and objectives. The committee reviewed the history of NSF projections, most notably those in The State of Academic Science and Engineering.

OCR for page 1
FORECASTING DEMAND AND SUPPLY OF DOCTORAL SCIENTISTS AND ENGINEERS: Report of a Workshop on Methodology In response to the perception that there could be a connection between NSF funding and its projections, especially regarding projections of shortages, the committee believed that NSF should limit itself to data collection and dissemination, and use external “arms-length ” forecasts for its policy needs to avoid the possible conflict of interest that might occur if it produced its own forecasts. Recommendation 3: Undertake a comprehensive review of data collection in the light of forecasting needs. NSF's SRS should undertake a comprehensive review of its data management program, preferably in coordination with Bureau of Labor Statistics (BLS), NCHS, and NCES. It should seek the production of more timely and useful data on the market for scientists and engineers. In addition, it should coordinate definitions and categories across agencies to facilitate a consistent picture of the different stages in the market, from student training and degree choice to mid-career transitions across and our of science and engineering fields. Moreover, sample sizes have been reduced since the late 1970s, which makes modeling difficult for small fields, specific employment sectors, and for rare events (such as mid-career changes). Recommendation 4. Data that enhance forecasts should be widely available and be disseminated on a timely basis. NSF's SRS should establish three high-priority data objectives. These are: (1) production of timely, descriptive statistics on employment and salaries by field of training, occupation, and sector, in a consistent time-series format that permits tracking and projections of trends; (2) production of an individual-level Public Use Sample, containing a consistent time series of cross-sections of doctoral recipients, and when available, nondoctoral recipients; and (3) production of a Public Use dataset panel of scientists and

OCR for page 1
FORECASTING DEMAND AND SUPPLY OF DOCTORAL SCIENTISTS AND ENGINEERS: Report of a Workshop on Methodology engineers to analyze transitions from the educational system to employment and transitions across fields, occupations, and activities. NSF should process these data, establish rules for access so that they are widely available, and institute less burdensome mechanisms to protect confidentiality. For example, in producing a Public Use panel, NSF might recruit respondents who are willing to provide vitae in a standard format without confidentiality restrictions. This format could be made available on the web; to make the process even easier for respondents, NSF could code respondent vitae. This would enhance modeling of individual and institutional behavior in response to changes in funding, demographically driven demand, compensation, etc. Standard coding of vitae would permit collection of more detailed data than the Survey of Doctorate Recipients form. Both modelers and policy advisors at the workshop complained that data were not timely. At present, there is no way of knowing in a timely way whether market mechanisms are working to alleviate shortages or gluts. Policies calibrated to outdated evidence may provide too little too late, or too much too late. Furthermore, if indicators show that market-clearing mechanisms are working rapidly, the need for policy change might be obviated altogether. NFS's SRS should collect and disseminate data that enable a variety of forecasting exercises that differ along some or all of the following dimensions: Unconditional vs. conditional (“What if”) forecasts. Multiple levels of disaggregation by field, e.g., physical sciences, physics, solid state physics, materials science. Optional variables to forecast, e.g., jobs, salaries, quality, productivity, occupation. Various sectors to forecast, e.g., academic tenure track/ other, postdoctoral positions, or industrial sectors. What to forecast, e.g., stocks, flows, transitions, or careers. Permit forecasts to go beyond means and standard errors

OCR for page 1
FORECASTING DEMAND AND SUPPLY OF DOCTORAL SCIENTISTS AND ENGINEERS: Report of a Workshop on Methodology to more complete descriptions of the distribution of possible outcomes. Various time horizons for forecasts. Finally, the NSF's SRS website currently focuses almost exclusively on providing relatively simple tabulations that are useful for casual policy analysis but not very useful for either career planning by students or for research on the science and engineering market. The data management program and website should be redesigned to service these neglected user communities (or in the case of students, the public and private organizations and associations that provide career guidance), and to provide links to other data sources from BLS and NCES that are important for analysis of the markets for scientists and engineers. Recommendation 5. NSF should develop a research program to improve forecasting. The Directorate of Social, Behavioral and Economic Sciences of the NSF should commission behavioral studies of scientists and engineers early in their careers, as well as studies of forecasting methods and evaluations of past forecasting exercises. Particular emphasis should be placed on critical parameters of market response, such as wage-sensitivity of field and occupation switching, and the determinants of the ability of employers to restructure research jobs in response to supply conditions in various fields. Emphasis should be placed on the difficult issues of measuring the quality and productivity of scientists and engineers, the quality of worker-to-job matches, and quality of life for scientists and engineers. This work should be conducted through the ordinary peer-reviewed research support process already in place at NSF. SRS should facilitate the dissemination of results of these studies but should stop short of sponsoring or endorsing specific forecasts or methods.

OCR for page 1
FORECASTING DEMAND AND SUPPLY OF DOCTORAL SCIENTISTS AND ENGINEERS: Report of a Workshop on Methodology This page in the original is blank.