4

Statistical Models and Administrative Records as Supplements to Surveys

As part of its charge, the steering committee was asked to consider the following two issues: (1) the appropriateness of the use of specific statistical models to estimate R&D funds going to and spent by nonprofit organizations, and (2) the effects of the timeliness and quality of the census and survey inputs into National Patterns in comparison with alternative sources of such information. Relevant to the second issue is the question of whether administrative sources of information, which would generally be more timely, might have comparable quality to that currently collected on the five census and surveys used as inputs into National Patterns. This issue is important as it is now generally recognized that response rates for surveys, including government surveys, are in a period of decline.

Although the response rates for the censuses and surveys that supply the inputs to National Patterns continue to support very reliable estimates, it remains important for the National Center for Science and Engineering Statistics (NCSES), which produces National Patterns, to keep abreast of alternative sources of information in case the response rates to its censuses and surveys experience a decline that threatens the reliability of the estimates. This concern was reflected in the recommendation of a recent National Research Council (2010, p. 60) report:

Recommendation 4.3: The Division of Science Resources Statistics [the predecessor to NCSES] should initiate work with other federal agencies to develop several demonstration projects to test for the best methods to move to a system based at least partly on administrative records.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 49
4 Statistical Models and Administrative Records as Supplements to Surveys As part of its charge, the steering committee was asked to consider the following two issues: (1) the appropriateness of the use of specific statistical models to estimate R&D funds going to and spent by nonprofit organizations, and (2) the effects of the timeliness and quality of the census and survey inputs into National Patterns in comparison with alternative sources of such information. Relevant to the second issue is the question of whether administrative sources of information, which would generally be more timely, might have comparable quality to that currently collected on the five census and surveys used as inputs into National Patterns. This issue is important as it is now generally recognized that response rates for surveys, including government surveys, are in a period of decline. Although the response rates for the censuses and surveys that supply the inputs to National Patterns continue to support very reliable esti- mates, it remains important for the National Center for Science and Engi- neering Statistics (NCSES), which produces National Patterns, to keep abreast of alternative sources of information in case the response rates to its censuses and surveys experience a decline that threatens the reliability of ­ the estimates. This concern was reflected in the recommendation of a recent National Research Council (2010, p. 60) report: Recommendation 4.3: The Division of Science Resources Statistics [the predecessor to NCSES] should initiate work with other federal agencies to develop several demonstration projects to test for the best methods to move to a system based at least partly on administrative records. 49

OCR for page 49
50 NATIONAL PATTERNS OF R&D RESOURCES The steering committee decided to address this topic by including a work- shop session on the quality of the information from STAR METRICS, an ongoing effort to collect administrative records information on R&D fund- ing from a variety of academic institutions. The first issue was included in the charge because no census or survey on R&D funds from or to nonprofit organizations has been carried out by NCSES since 1996-1997 (see Boroush, 2007). Instead, NCSES has used a statistical model to estimate major components of R&D funds to and from nonprofit organizations. The question for consideration at the workshop was whether this approach results in useful estimates, or that NCSES should consider either re-fielding such a survey or identifying rel- evant sources of information. An earlier National Research Council (2005, p. 8) report made the following recommendation on this issue: “The panel recommends that another attempt should be made to make a survey-based, independent estimate of the amount of R&D performed in the nonprofit sector.” The workshop participants were asked to consider this issue anew. USING STATISTICAL MODELS TO ESTIMATE R&D FUNDS TO AND FROM NONPROFIT INSTITUTIONS Nonprofit institutions both provide and receive R&D funds to and from other organizations, including other nonprofit institutions. These funds are often captured by NCSES on its R&D censuses and surveys when the provider or the recipient is not a nonprofit institution. For example, NCSES can estimate the amount of R&D funding that is provided by the federal government to nonprofit institutions from the responses to the Sur- vey of Federal Funds for Research and Development (known as the Federal Funds Survey). Also, from the Higher Education Research and Develop- ment Survey (and its predecessor, the Survey of Research and Development Expenditures at Universities and Colleges), NCSES is able to estimate the amount of funds academic institutions received from nonprofit institutions. However, as noted above, NCSES has not fielded a survey that comprehen- sively covers R&D funds from industry to nonprofit institutions or from nonprofit institutions to other nonprofit institutions for more than 15 years. Given this lack of survey information and given deficiencies in other poten- tial data sources, NCSES has used statistical models to provide estimates of these components of R&D funding. However, these models rely on assumptions that various relationships have not changed appreciably over time, and so they are questionable. The questions raised by this situation were addressed at the workshop by Michael Cohen of the Committee on National Statistics, who described and critiqued the statistical models currently used by NCSES, and Jeff A ­ lexander of SRI International, who discussed the recent dynamics of non-

OCR for page 49
STATISTICAL MODELS AND ADMINISTRATIVE RECORDS 51 profit R&D funding, and alternative administrative sources of information on R&D funds. Current NCSES Estimation of Nonprofit R&D Funds Cohen provided a description of the statistical models currently used by NCSES. To produce current estimates of industry funding of R&D per- formed by nonprofit organizations, NCSES assumes that the annual growth in funding from industry to nonprofit organizations changes in constant proportion to the annual growth in industry-to-­ industry funding. Since industry-to-industry funding is estimated on an annual basis, this assump- tion allows inferences about annual growth of funding from industry to nonprofit organizations, which has not been directly estimated since 1997. Estimating the amount of R&D funds provided by a nonprofit institution to another makes use of an analogous ratio relating the relative change in those funds to the percentage change in R&D funds sent from academic institutions to nonprofit institutions. Although the cost of the current approach is minimal, since it involves no new data collection and only a modest amount of analysis time to recompute the relationship as described above, Cohen said it is reasonable to conclude that the current method is unlikely to provide high-quality estimates, for two reasons. First, as reported by Boroush: “Of the NPOs [nonprofit organizations] surveyed for 1996 and 1997, 59 percent did not respond. . . .” (see p. 49, http://www.nsf.gov/statistics/nsf02303/pdf/sectc.pdf [June 2013]). However, Boroush noted (personal communication) that the respondents accounted for between 80 and 90 percent of total R&D funds. Second, and much more important, the reliance on what must be viewed as a heroic assumption, namely, that the percentage yearly change in one sec- tor of R&D funding divided by the percent yearly change for another sector would remain stable for even a few years, let alone 17 years, is worrisome. It raises the serious concern that the quality of these estimates has seriously deteriorated over time. Cohen discussed some alternative estimation approaches that might have been considered by NCSES. First, one could posit the constancy of functions other than the ratio of yearly percentage change for one sector to another. Second, within the same basic estimation strategy, there is an alternative technique that derives from the possibility that the stability of the above ratio might be enhanced by partitioning industries into sub- groups, each of which might have more homogeneity of the ratio. That is, instead of using a single ratio, multiple ratios would be fit for industries with certain characteristics. Similarly, this approach could be applied to groupings of nonprofit institutions. Third, instead of estimating a single ratio to be used across time, one could view the ratios over time as a time

OCR for page 49
52 NATIONAL PATTERNS OF R&D RESOURCES series, which could be smoothed or forecast through the application of a time-series model. Cohen acknowledged that the possible advantages of any of these alternatives cannot be assessed given the absence of a survey for 15 years. Available and Missing Data on Nonprofit R&D Funding In addressing the question as to whether the stability assumption that NCSES uses is likely to obtain, Alexander asserted that the nonprofit sector has changed both qualitatively and quantitatively since 1996, with those dynamics likely affecting nonprofit R&D activities. Although the scale of nonprofit R&D activity remains relatively small in comparison with other sectors, it can have a disproportionate impact in specific fields, for example, in biomedical research. Alexander pointed out that there are nonsurvey sources of information on nonprofit R&D funds, although they provide only partial informa- tion toward estimates of total R&D activity: • For the amounts moving to and from the nonprofit sector, the NCSES Survey of Federal Science and Engineering Support to ­ niversities, U Colleges and Nonprofit Institutions provides information on funds to non- profit institutions from the federal government, the Business R&D and Innovation Survey (BRDIS) provides some information on R&D funds going from industry to nonprofit institutions, and the Higher Education Research and Development (HERD) survey may provide limited informa- tion on the R&D funds that universities provide to nonprofit institutions (in the form of pass-through funding). • For R&D funds provided by nonprofit institutions, BRDIS collects data on funding to industry (though it commingles those funds with govern- ment research funding), and HERD provides information on R&D funding from nonprofit institutions to colleges and universities. The most significant gap, currently not covered by any surveys or censuses, is how much R&D funding goes from nonprofit institutions to other non- profit institutions. A secondary issue is the quality of BRDIS in accounting for R&D funding between industry and nonprofit institutions. The Nonprofit Sector in Detail To better understand the issue, Alexander discussed the nature of n ­ onprofit institutions. The Internal Revenue Service (IRS) code defines a nonprofit institution as follows: “The exempt purposes set forth in section 501(c)(3) are charitable, religious, educational, scientific, liter-

OCR for page 49
STATISTICAL MODELS AND ADMINISTRATIVE RECORDS 53 ary, testing for public safety, fostering national or international ama- teur sports competition, and preventing cruelty to children or animals.” This definition includes charitable organizations and private foundations. Well-known sponsors of R&D are family foundations (e.g., Packard and ­ Ford), issue-specific foundations (e.g., Susan G. Komen and the American ­ Association for Cancer Research), and corporate foundations (e.g., Intel and Amgen). Well-known recipients of R&D funds are research insti- tutes (e.g., SRI and Battelle), hospitals, and universities and colleges. Some nonprofit institutions are known both as funders and as recipients, including the Howard Hughes Medical Institute and the Monterey Bay ­ Aquarium Research Institute (which is an operating unit of the David and Lucile Packard Foundation). One key statistic relevant to data collection on nonprofit organizations is the high degree of concentration in grant- making: the largest organizations, which make up less than 1 percent of all nonprofit organizations, account for 59 percent of all nonprofit grant funding.1 Alexander said that there is a useful taxonomy, the National Taxonomy for Exempt Entities, which is a classification by organizational mission. There are 26 major groups, with a leading letter that indicates area of primary interest, and each group has two-digit subcategories. Codes are assigned by IRS examiners for tax purposes. Three groups—U20, H30, and V05—are particularly relevant to this discussion. U20 covers organiza- tions that focus broadly on scientific research and inquiry or that engage in interdisciplinary scientific activities (e.g., the Research Triangle Institute); H30 covers organizations that conduct research to improve the preven- tion, diagnosis, and treatment of cancer (e.g., the Fred Hutchinson Cancer Research Center); and V05 covers organizations whose primary purpose is to carry out research or policy analysis in the social sciences (e.g., the American Enterprise Institute for Public Policy Research). Alexander then discussed changes in the scale and nature of the non- profit sector since 1996. First, the number of nonprofit organizations increased from less than 60,000 in 1999 to more than 90,000 in 2008. During the same period, grants by nonprofit organizations increased from about $20 billion to more than $50 billion (in nominal dollars). Between 1998 and 2009, the organizations’ assets increased by about 50 percent, including the market correction in 2008-2009. Another major change is that many of what are now the largest private foundations were started quite recently: only two foundations listed among the ten largest funders of science and technology grants in 1998 remained in the top ten in 2010. 1  Data from National Center for Charitable Statistics (NCCS) data on 501(c)3 private foundations. Available: http://nccsdataweb.urban.org/PubApps/nonprofit-overview-sumRpt. php?v=fin&t=pf&f=0 [March 2013].

OCR for page 49
54 NATIONAL PATTERNS OF R&D RESOURCES Another change is the rise of “venture philanthropy.” In the late 1990s, technology entrepreneurs established philanthropic foundations with a dif- ferent approach to funding, featuring: (1) greater emphasis on outcomes and performance, (2) a closer relationship between the foundation and the grantees, and (3) a strategy of funding a portfolio of “social investments” in the way that venture capital firms fund start-up companies. Alexander added that this rise had also changed philanthropy through the use of mul- tiple initiatives to improve outcome evaluation, efforts to measure grantee satisfaction with the funding process, and a greater focus on innovation in programs and service delivery. At the same time, however, he said, nonprofit organizations are fac- ing the same pressures as all institutions because of the “great recession.” As a result, nonprofit organizations may face structural changes, with the growing use of program-related investments and dedicated venture phil- anthropic funds. Alexander added that as a result of these recent changes, the funds both provided by and going to nonprofit organizations are more concentrated in a smaller number of organizations, and are awarded using a somewhat different set of objectives. Also, the shift to outcomes-oriented investments may complicate the formulation of questions that attempt to assess the R&D component of grants. Lastly, he said, he expects that non- profit R&D performers are likely to diversify into new areas of activity. Alexander said that there are three nonsurvey data sources of R&D funding to and from nonprofit institutions: (1) various federal data reposi- tories, including USASpending.gov, the Federal Audit Clearinghouse, and the Internal Revenue Service Master Business File; (2) third-party data pro- viders, including organizational profile repositories (in particular Guidestar ­ and the NCCS); and (3) nonprofit self-reporting, that is, annual reports and financial statements and IRS Form 990 (for details, see SRI International, 2008). To assess the data in each of these repositories, Alexander used the quality criteria for nonsurvey data sources from Statistics Canada’s ­ uality Q Guidelines (Statistics Canada, 2009). For federal repositories, the major concern is that, except for the IRS, the coverage is poor, since the other sources focus on nonprofit organizations that receive federal grants. For third-party data repositories, the major concern is that detailed information is lacking. For nonprofit self-reporting, the major concerns are coverage and the level of detail, especially since the field for “Nonprofit Program Classification” is not required and is rarely used. Alexander said that in his opinion, given the various recent dynamics in the sector he noted earlier, there is a strong likelihood that the current National Patterns’ estimates for nonprofit R&D funding, both funders and performers, are not accurate. Evidence in support of this conclusion is that the most recent (2008) National Patterns’ estimate of total nonprofit R&D funding was $12.6 billion, but the report on private foundation grant-

OCR for page 49
STATISTICAL MODELS AND ADMINISTRATIVE RECORDS 55 making by the NCSS was $53.8 billion, and the overall nonprofit R&D funding includes more than private foundation activities. Other evidence comes from the Foundation Center, which estimates that total founda- tion grant-making has grown much faster over the past 10 years than the nonprofit R&D funding estimate from National Patterns. Furthermore, Alexander asserted, current administrative data collections are insufficient to collect consistent and comprehensive information on nonprofit R&D activities because research activities are often embedded in nonresearch programs, and classification systems do not treat R&D funds consistently. Also, nonprofit organizations have little incentive to provide details on their activities, and the data they do provide are delivered in nonmachine- readable formats; in particular, most IRS Form 990s are published online as image pdfs. However, Alexander noted, some of these data deficiencies are becom- ing less troublesome. First, more reports are becoming machine readable with e-filing of IRS 990s and greater use of extensible markup language (XML) standards for reporting of financial data. As more nonprofit institu- tions make more information available online, the data quality is improv- ing. Foundations in particular have launched sectorwide efforts to increase the transparency of their grant-making activities and to establish technical and reporting standards. One example is the Glasspockets initiative, which is building an online, real-time database of grants awarded by major foun- dations using data feeds from those foundations.2 Finally, there are some options for applying text analytics to grant project descriptions and IRS Form 990 narratives to assist in classifying R&D activities. In conclusion, Alexander provided some information that might be useful in designing a new sample survey of R&D funding from and to nonprofit organizations. First, while most nonprofit organizations are either only sponsors or performers of R&D work, there are some that play both roles. The amounts allotted are skewed, in the sense that a rela- tively small number of nonprofit organizations provide most of the R&D funds, which could make data collection easier. Also, nonprofit funders tend to focus on specific areas of application, which could also facilitate data collection. E-filing and Survey Possibilities: Discussion In response to Alexander’s presentation, Karen Kafadar asked how the quality of the nonsurvey data sources is assessed. Alexander replied that there is no third party that assesses the data reported. IRS may ask for a clarification or for modifications through an audit. He said that he 2  For information, see http://www.glasspockets.org [January 2013].

OCR for page 49
56 NATIONAL PATTERNS OF R&D RESOURCES used “quality” to refer to the level of detail that is available in addition to the R&D amounts. Kafadar then suggested that more administrative data might be readily available if more nonprofit organizations change to e-filing, and Alexander agreed. He added that businesses are moving toward use of extensible business reporting language (XBRL), which is a common software standard for financial reporting. However, although the use of XBRL does enhance machine readability, the content may still be less than satisfactory. Christopher Hill asked how independently organized industrial col- laborative R&D efforts—such as the National Center for Manufacturing Sciences, Sematech, and SRC—are categorized. Alexander answered that some are included as nonprofit organizations: for instance, Sematech is a 501(c)(6) organization, and the Electric Power Research Institute is a 501(c)(3). Joel Horowitz asked why one focuses on this component of R&D spending given that it is dwarfed by that sponsored by the federal gov- ernment, business and industry, and universities and colleges? Alexander responded that current trends in overall patterns of R&D expenditures, as well as sudden growths or losses, are worth knowing. Yet he acknowledged that if one is making international comparisons, knowing this smallish sec- tor better may not be that important. Hill asked why there has not been a survey on nonprofit R&D for 15 years. Alexander said that a first reason is the sampling frame might be difficult to generate. There are 1.6 million nonprofit organizations, most of which are church congregations. However, he noted, using a sample that included a certainty stratum of the largest nonprofit organizations would easily collect a very large fraction of total R&D funding for either per- formers or funders and that the remainder could be sampled using a fairly small sampling rate. Alexander said a second reason is that development of the questionnaire could be complicated given the heterogeneity of the organizations and institutions to be surveyed. He suggested that the most cost-effective approach would be to piggy-back on a survey already fielded by an entity collecting nonprofit data. Fernando Galindo-Rueda suggested the possibility of collecting such information as a byproduct of the data collections carried out by the Bureau of Economic Analysis in producing the national income and product accounts. STAR METRICS AND VIVO Interest in STAR METRICS for this workshop reflects current trends of declining response rates for federal censuses and surveys and increasing

OCR for page 49
STATISTICAL MODELS AND ADMINISTRATIVE RECORDS 57 costs per interview.3 The cost issue raises the question of whether admin- istrative sources of information on R&D funds might be used in concert with the NCSES surveys to improve the quality of the information on R&D funding. Clearly, whenever a federal R&D grant is provided or whenever an academic institution awards or is awarded a grant, some formal docu- mentation exists about the grant. A compilation of this documentation might serve as a source of information for R&D grants. STAR METRICS is an attempt at such a compilation. One particular possibility the steering committee wanted the workshop participants to consider is to use STAR METRICS information to improve the quality of the census and survey information through editing and imputation techniques. John King of the U.S. Department of Agriculture (USDA) provided a presentation on the cur- rent status of STAR METRICS and a related program called VIVO.4 Overview King started by observing that there is a need for common data stan- dards and open platforms to improve our understanding of how science is done. STAR METRICS establishes such data standards to support empirical studies of science impacts. VIVO provides an open platform that helps dif- ferent scientific institutions meet that standard. His presentation explored these two constructs and how they interact. STAR METRICS is a platform for the collection and analysis of data on R&D investments that relies on automated harvesting from systems of records. It is intended to provide new applications and tools to meet research needs and policy requirements in the future, and to minimize any adminis- trative burden from structuring this information to support various analytic purposes. Accordingly, STAR METRICS uses a common format consisting of the principal investigator, program information, abstract/­ roposal, and p obligated funds.5 It includes administrative data about individuals involved in the research, payments to vendors, and subawards, as well as informa- tion about any reports and data for analysis from the research. Quarterly updates to STAR METRICS are made by matching through use of grant numbers. Although the initial structure of STAR METRICS was developed 3  STAR METRICS—Science and Technology for America’s Reinvestment: Measuring the Effects of Research on Innovation, Competitiveness, and Science—is a federal collaboration with research institutions; for more information, see https://www.starmetrics.nih.gov [Janu- ary 2013]. 4  VIVO is an international network for researchers in agricultural fields; for more informa- tion, see http://vivoweb.org [January 2013]. 5  recent paper (Porter, Newman, and Newman, 2012) examined methods to mine the A program descriptions in the abstracts and proposals to identify the prominent topical themes addressed in the research. The authors are continuing their work on the subject.

OCR for page 49
58 NATIONAL PATTERNS OF R&D RESOURCES to collect data for federal R&D funds, the same accounting framework is possible for nonfederal research support. King said that the number of institutions currently making up the STAR METRICS community is around 80, but new institutions are joining every month. A challenge to STAR METRICS is to make the data compatible for many different types of research questions, King said. For instance, there are needs for both extramural and intramural research reports on dif- ferent aspects of R&D funding. With this data, one can address such research questions as: (1) Does intramural research engage different topics of inquiry? (2) How can R&D portfolios across programs, agencies, or departments be compared? (3) How does scientific discovery differ across settings? and (4) What incentives and rewards do scientists encounter? King continued with a description of VIVO. VIVO permits USDA researchers to identify colleagues who are carrying out related research, which can accelerate collaboration, and it is also a public-facing expertise locator that portrays the full scope of USDA research. Finally, it provides a connection to other VIVO institutions via its ontologic structure. VIVO thus provides a uniform data structure across USDA’s science agencies, it is a source of clean data to document outcomes of intramural science, and it enables sharing of similar information with other federal R&D agencies. A possible application of VIVO is to provide topic modeling using natural language processing, with topic tags provided for each document in the database. It is planned that VIVO will provide expertise locators for review panels and funding announcements. King expressed the hope that VIVO would support the analysis of research gaps and hotspots, provide the ability to compare research invest- ments to outputs, and support the comparison of projects that are funded with those that are not funded. Discussion Alexander first asked how the coding in STAR METRICS distinguishes between scientific and nonscientific grants. Kei Koizumi responded that it was not perfect but that the coding provides a reasonable taxonomy.6 ­ Alexander asked if they looked at the contracts, and the answer was not yet, but that topic modeling could ultimately be used on the contracts. William Bonvillian asked if the U.S. Department of Defense (DoD) or the National Aeronautics and Space Administration (NASA) are covered in STAR METRICS. King and Koizumi answered that with the exception of 6  The STAR METRICS algorithm to identify sectors of funds relies heavily on codes from the Catalog of Federal Domestic Assistance (CFDA), which identifies federal agencies and specific program sources of federal funds.

OCR for page 49
STATISTICAL MODELS AND ADMINISTRATIVE RECORDS 59 the Army Research Laboratory, DoD and NASA are not covered. The best way to cover them, he said, would be to bring in units that had uniform accounting frameworks, such as NASA. Some of the difficulty stems from the sensitivity of DoD to open up military data. Bonvillian thought that unclassified university research supported by DoD ought to be collectable. He said that given the size of DoD’s R&D budget and its role in innovation, it is important to try to bring them on board. Hill noted, first, that given text mining and related techniques, DoD has a reason not to make everything that is open available for every- one to see. Hill then asked whether, with topic modeling research, the t ­opics are mutually exclusive with regard to the grants and investigators or if the same projects show up under different topics. King responded ­ that grants can indeed show up under multiple topics. Hill pointed out that STAR METRICS is intended ultimately to measure the effects of research ­ on innovation, competitiveness, and science. The science part is done, but what about measuring the effects of innovation and competitiveness? King responded that STAR METRICS could be used to facilitate such research. He mentioned the work of Jason Owens-Smith at the University of Michigan, who is examining the labor market outcomes of the grants ­ that trained graduate students. Hill said that the number of jobs was not the same thing as innovation. King responded that they are also look- ing at tracking the collaborations among principal investigators through STAR METRICS and Bibliometrics to see whether certain patterns of col- laboration end up producing different kinds of science. David Newman then asked: What are the incentives of universities to participate in STAR M ­ ETRICS? King responded that they get a better view of their own insti- tution. They get back quarterly reports that are helpful for understanding what research is being carried out. Also, the researchers can find out who else is doing relevant work at their institution. The last topic raised was coverage. Bonvillian asked: Of the roughly 150 leading research universi- ties, how many are in STAR METRICS? King answered between one and two dozen. STAR METRICS for Edit and Imputation Cohen’s presentation considered using STAR METRICS to improve National Patterns. He began by noting that STAR METRICS is unlikely to cover industrial R&D statistics in the near future. It is also clear from King’s presentation that its coverage for the frame of the HERD survey is also unlikely to approach the level necessary for use in support of National Patterns in the near future. Therefore, STAR METRICS will not soon be able to replace the censuses and surveys that are used as input to National Patterns. However, if STAR METRICS values are subject to less measure-

OCR for page 49
60 NATIONAL PATTERNS OF R&D RESOURCES ment error than the analogous responses to the censuses and surveys, these alternative values might be useful for improving the quality of the census and survey data through editing or imputation. Either assumption—that censuses or surveys could provide preferred data to STAR METRICS or vice versa—can find support. Census and sur- vey responses might be preferable because administrative records systems often contain errors as a result of matching problems and other data link- age errors. But STAR METRICS values might be preferred to survey and census responses since the actual documentation of the grant is used for data entry, which may reduce various sources of measurement error, such as recall error. For the purposes of his presentation, Cohen said he assumed that the administrative sources were of superior quality in comparison with the infor- mation received from census and survey respondents, though he acknowl- edged that the assumption may ultimately prove false. Cohen noted that Chris Pece of NCSES was currently involved in an effort to determine the circumstances under which the census and survey data were and were not preferable to STAR METRICS data through a match study. One way in which STAR METRICS information could be used to improve the survey and census responses is through the use of editing routines. Assume, for example, that there is a census response and a STAR METRICS value for a response and that they differ by more than p percent, or that one amount is nonzero and the other is zero. In such situations, it would seem beneficial to contact the respondent to ask for a clarification. The likely result would be an improved dataset. Of course, investigating each alert of a potential discrepancy could be labor intensive, and respon- dents often do not like to be questioned, especially if the survey or census response was correct. So, if one wanted to implement such a procedure, one would want to keep such callbacks to a minimum without greatly increas- ing the number of true discrepancies that were missed. In order to do that, one would need to develop a much better understanding of the distribution of differences that arise in situations in which either the survey or census response was correct, or was in error, and similarly for the STAR METRICS value, in order to develop an effective editing routine. A second possibility would be to use STAR METRICS values for imputation. If one has both a census (or survey) response and a STAR METRICS value and the discrepancy between them results in a failed edit, and if for some reason one cannot resolve the discrepancy by contacting the respondent, an imputation model could provide a correction. For instance, one possibility would be to assume that the errors in census (or survey) responses and the errors in STAR METRICS were independent and nor- mally distributed. Then an imputation routine could be based on a linear combination of the two values weighted by their relative precision. But their

OCR for page 49
STATISTICAL MODELS AND ADMINISTRATIVE RECORDS 61 error distributions could be substantially different from this assumption. For example, the errors might be such that values are “changed to zero” with some probability (which could happen, for example, by attributing the wrong identification number to a grant). Correctly diagnosing such errors and providing good imputations for the census or survey responses would require a very different imputation model than one that would be effective in the first posited situation. In what could serve as a first step toward the development of impu- tation and editing routines, NCSES is undertaking an exploratory data analysis of the differences between STAR METRICS and census values. But even after this research is completed, Cohen noted, it may still be a very challenging task to design an effective editing or imputation routine. He noted that this kind of matching study is currently being conducted at the Census Bureau for a wide range of survey and census responses because of the increased availability of administrative records in a variety of contexts.7 In addition to editing and imputation when census and STAR ­ ETRICS M values are both available, Cohen suggested some additional though some- what speculative uses for STAR METRICS data that might serve to improve the responses to NCSES surveys and censuses in the future. For example, if one is missing the census (or survey) response for a quantity for which the STAR METRICS value is available, the STAR METRICS value could be used as a surrogate for the survey response. An important hypothetical example might be values for nonprofit R&D, if STAR METRICS in the future included a substantial fraction of nonprofit R&D activity. Absent a new census or survey of R&D funding to and from nonprofit institutions, one would be concerned that such STAR METRICS data were not validated for this purpose. However, given the lack of a recent survey and the likely poor performance of the current estimation approach, such an approach might still be an improvement in the data in National Patterns. In addition to using the STAR METRICS values only as survey sur- rogates, in situations in which data have been collected recently for a large number of census and STAR METRICS pairs, using such paired data in addition to the analogous STAR METRICS value for the missing census response might enable one to develop a model-based imputation for miss- ing census responses that would be preferable to using the analogous STAR METRICS value as a surrogate. Cohen added that, more broadly, even in situations in which the census response is available, again assuming that STAR METRICS data are shown to be more reliable, and further assuming that one discovered that STAR 7  For instance, the most recent Federal Committee on Statistical Methodology Research Conference, held in January 2012, included a presentation entitled “Evaluating Job Data in the Redesigned SIPP Using Administrative Records” by Martha Stinson of the Census Bureau.

OCR for page 49
62 NATIONAL PATTERNS OF R&D RESOURCES METRICS values had a strong relationship (including over time) with the census (or survey) values, one might model this relationship and then use such a model to “adjust” census responses. In this case, one is in effect estimating the census response using the combination of census and STAR METRICS responses for all institutions (possibly using not only data for the current time period, but also historical data). However, it is difficult to imagine that one would find such a stable relationship because the census responses are very likely to be correct, and so it is not easy to find an impu- tation model that would improve on an observed census or survey value. In any case, use of such a model would require a comprehensive validation effort, which would be very expensive to carry out. In addition, since STAR METRICS is a voluntary program, one should also be concerned that the institutions that choose not to participate are different than those for which the model was developed.