Click for next page ( 62


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 61
7 INTERNATIONAL PERSPECTIVES ON MEASURING RESEARCH IMPACTS Measuring the impacts of research ranges from studying broad changes in public policy to tracking the influence of a certain research paper on subsequent publications in that field. Some of the newest techniques marry on-line data collection and databases with analytic tools, yielding a nuanced picture of research outcomes and the influence of funding dollars. At the workshop, speakers from the United Kingdom (UK), the European Union (EU), and Brazil shared some of their thoughts on recent evaluation methods and future goals. Measuring the effectiveness of research is a growing field precisely because of the scarcity of resources and the need for policy makers to demonstrate returns on investments around the world. MEDICAL RESEARCH COUNCIL EVALUATION SYSTEM The United Kingdom Medical Research Council (MRC) provides government funding for public, private, and university research in the United Kingdom. Science funding in the UK comes from the government, the private sector, and charities, and universities function on a dual support system, where money for staff and infrastructure comes from higher education funding councils and research councils designate funds on a project and program basis. Ian Viney, Head of Evaluation for the London-based MRC, outlined the council’s efforts to measure and influence research impacts. The MRC is focused on collecting comprehensive evidence regarding the progress, productivity, and quality of research output; supporting studies along the lines of those funded by the National Science Foundation’s Science of Science and Innovation Policy Program 61

OCR for page 61
62 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH (SciSIP); encouraging researchers to maximize their “pathways to impact”; and adding the assessment of impact as a factor in allocating new funds to UK universities. In 2006 the MRC started using an online system called e-Val. The system, which replaces end-of-grant reporting, requires grant recipients to make online reports each year, resulting in structured feedback over the lifetime of a grant rather than a long report at the end summarizing years of progress. The evaluation is designed to track how scientists are influencing policy development and contributing to new products and interventions. In building the evaluation, the MRC asked questions intended to yield hard evidence of impacts, outcomes, and output, in addition to traditional tracking of papers and patents. In two years of data gathering, more than 3,000 researchers have participated. The system has collected 70,000 reports representing feedback on £2 billion of MRC funding, or 92 percent of MRC expenditures in the last four years. In 2010 the evaluation provided details on 5,000 active collaborations. Since 2006, MRC researchers reported over 130 citations in policy documents, 360 new products and interventions in development, 200 published patents, and 37,500 publications. The online evaluation system helps the MRC link research outputs with the social, economic, and academic impacts of research. For example, one study done by the Health Economics Research Group, the Office of Health Economics, and RAND Europe (2008) focused on the return on investment for research on cardiovascular disease and mental health. Combined with data from e-Val, the study built a strong quantitative argument for investment in medical research in time for the change of party control of government in 2009 and the subsequent review of all government spending. Monitoring policy citations and the influence of scientists in policy helps track progress over time and demonstrates how research translates to clinical practice, said Viney. The evaluations also have given context to case studies, which the council often uses to illustrate to the government the benefits of MRC funding. But it is not easy to encourage researchers to think about the ultimate objectives of their work and how to maximize their impact. Viney pointed out that the medical community is somewhat more accustomed to this, while other disciplines are more resistant. The Research Councils UK (RCUK), which is made up of seven UK research councils that together allocate £3 billion each year to research, is keen to maximize the economic, academic, and societal

OCR for page 61
63 INTERNATIONAL PERSPECTIVES ON MEASURING RESEARCH IMPACTS impacts of research, and the councils are including information on these impacts in all of their funding applications. They ask researchers not to predict what the impact will be but simply to consider enhancing the potential influence of their research. A peer review process, “Pathways to Impact,” is also designed to further this goal. The Higher Education Funding Councils in the UK, which allocate £2 billion to university research every year, have moved in a similar direction. Due partly to pressure encouraging the Higher Education Funding Councils to look more closely at impacts, they implemented the Research Excellence Framework, which assesses research outputs, impacts, and the research environment at each university. The framework splits disciplines into units of assessment, defined as substantive bodies of research in coherent discipline groups. There are roughly 30 units of assessment. A pilot using expert panels to assess impact at 29 universities, with each university submitting case studies for two units of assessment, was considered quite successful. The panels found ways to assess the validity and significance of impacts across diverse disciplines, including clinical medicine, physics, earth systems, social work, and English literature. The panels will contribute 20 percent to the overall REF assessment, with the goal of increasing that contribution after 2014. In the government’s 2010 Comprehensive Spending Review, the MRC’s evaluation helped protect the medical research budget in real terms until 2014 while the overall science budget received no inflation increase. For the MRC, this is a tangible example of evaluation influencing policy, Viney said. Other funding agencies are now looking at ways to imitate e-Val, and discussions are under way to harmonize and rationalize the data collection process with a view of generating a more UK-wide view of research output. Plans are also under way to commission more work on estimates of spillover benefits in the UK, rather than borrowing from U.S. estimates. Viney concluded that the government is focused on economic growth and leveraging investment, and the importance of describing, understanding, and assessing impact is becoming more widely accepted in the UK. MEASURING IMPACTS OF RESEARCH FUNDING IN THE EUROPEAN UNION When evaluating research, it is important to compare old and new approaches. Brian Sloan, Directorate-General for Research and

OCR for page 61
64 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH Innovation for the European Commission, discussed various forms of evaluation in the context of the European Union Framework Program, which supports European science, technology, and competitiveness. The program is designed to complement national programs, focusing on areas where national funding may not reach, and to encourage cooperation and coordination between countries. The program allocates funding for transnational research projects, and also for mobility so that researchers are able to travel from one country to another. The current Framework Program, the seventh since 1984, has a budget of 50 billion Euros, or approximately $70 billion, which is 7 to 8 percent of European R and D funding. There are four components: Cooperation, Ideas, People, and Capacities. The Cooperation piece funds transnational research consortia. Ideas funds national teams that compete across the European Union. People funds mobility. Capacities provides funding for infrastructure. Within each of these divisions is a range of different science and technology fields. Traditional methods that the Framework Program has used to evaluate the impacts of research include interviews, surveys of program participants, and expert panels. But Sloan pointed out several challenges inherent in these methods. Surveys can be a burden to participants, especially when long and detailed answers are required. This can influence the quality of their response. Response bias and partial responses are also a concern. In addition, because most research projects have various funding sources, it can be difficult to attribute specific findings directly to EU funding. While these methods are still quite valuable, it is worthwhile to look at new approaches. New methods include what is called linking and ex-ante modeling. Until recently, it was difficult to identify recipients of EU funding by linking into bibliometric databases, but in 2009 it became possible to search grant activity and funding acknowledgements in the Web of Science database and therefore accurately identify not only program participants but their affiliates. Using the database in this way allows for assessment of research output and comparison with other projects, national averages, and world averages. There is also a built-in control group, which is lacking in surveys or participant interviews. Using bibliometric data, it is possible to map co-publication or track which disciplines publish most within the various programs. This type of evaluation is particularly relevant for the Framework Program, as one of its goals is to measure the results of funding against other transnational endeavors. It is also possible to measure the effects of

OCR for page 61
65 INTERNATIONAL PERSPECTIVES ON MEASURING RESEARCH IMPACTS distance or language on collaboration, and evaluate whether the program is succeeding at connecting people and regions that would not otherwise be brought together. Another approach the program took was linking with the Community Innovation Survey, a harmonized questionnaire that surveys 40,000 firms across 30 European countries. The survey looks at innovative outputs and activity, R and D spending, patents, cooperation, and new products. Included in the survey were questions asking whether firms had received any EU funding from 2002 to 2004 and whether they had participated in the Framework Program. The responses provided crucial data that could then be used to compare Framework Program participants with other researchers, controlling for variables such as company size and sector, and discern whether the program increases collaboration and productivity. The commission also found ex-ante evaluation to be a useful tool when applied to the Framework Program. The European Commission produces an ex-ante impact assessment report each time it develops new funding programs, explaining what problem is being addressed, why the government and in particular the EU must intervene, the objectives of the program, and what policy options have already been considered. For each option, the assessment also includes predictions of economic, social, and environmental impacts. Using an econometric model, the commission used a similar approach to assess macroeconomic impacts of the seventh Framework Program up to 2030 under various scenarios. The model predicted effects of the program on exports, imports, research, GDP, employment, and a range of other indicators. Again, like bibliometric data, this approach allowed for comparisons and manipulation of data, as well as bringing up potentially interesting and important developments that may not otherwise have been recognized. Ex-ante evaluation and linking provide another angle on measuring research outcomes and impacts. Because official statistical surveys provide such a large amount of reliable data, sophisticated analyses can be done of networking effects that cannot be captured from participant surveys. Sloan emphasized the potential of such approaches to yield further progress in the future.

OCR for page 61
66 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH MEASURING IMPACTS OF SCIENCE, TECHNOLOGY, AND INNOVATION INVESTMENTS IN BRAZIL Brazil’s Marcio de Miranda Santos, Executive Director of the Center for Strategic Management and Studies in Science, Technology, and Innovation, explained why quality data and a good information gathering system are invaluable for evaluating research impacts and outcomes. A comprehensive information infrastructure that facilitates evaluation of research is difficult to build, since many types of information are necessary for a thorough evaluation, including data on individual researchers, projects, collaborations, R and D networks, research institutions, and public agencies. The system has to be adaptable and able to handle the complexity of a range of inputs. Santos described Brazil’s strategy for building such a system. Several principles are guiding the center’s work. One is to expand on what is already available. In Brazil, this means linking data from sources such as the National Council for Science and Technology, the National Agency for Industrial Development, various innovation agencies, projects, and dissertations. The data requirements must be designed not just for government needs but to provide access and functionality for science, technology, and innovation participants as well. An effective program will rely on traditional software engineering methods as well as knowledge engineering and e-government approaches. The Lattes platform, which Brazil has been using since 1999, holds program information in a database that currently contains over 2 million curricula vitae (CVs) and is updated every three months on average. In 2008 the Center used it successfully to do an ex-ante evaluation of networks that had submitted proposals to the National Institutes of Science and Technology program (INCT). The program aims to promote networks among research groups and individuals, internationally competitive research, high-quality S and T development, and joint use of laboratories by universities and companies. The program also will contribute to improving education standards at all levels. Using Lattes, the Center took snapshots of information from individual CVs and from the INCT program as a whole and analyzed that information to determine the success of the program. A snapshot of one project from 2008, with 25 people in the network, provided data on co- authorship of papers, researchers who shared advisors, and participation in other projects and committees. The Center then used Innovation

OCR for page 61
67 INTERNATIONAL PERSPECTIVES ON MEASURING RESEARCH IMPACTS Portal, an electronic service designed to link information from different data sources, to follow shifts in project networks and collaboration. For example, three scientists working on the first project were not co-authors at the time the proposal was submitted, but by 2011 they had begun to produce papers with other project participants. Another example comes from the Brazilian Academy of Mechanical Sciences and Engineering, which was interested in identifying the weaker departments in mechanical engineering in Brazil. Researchers used Lattes to examine the distribution of knowledge within mechanical engineering, based on the number of publications produced by each scientific domain. They broke down the field into smaller subdomains and pinpointed weaker areas where reinforcement would be useful. This methodology allowed public decision makers to not only identify weak spots but also track improvement, measure the impact of research investments, and make decisions on how to further improve the system. The advantages of an integrated national platform such as Lattes are substantial, said Santos. It allows efficiency in both ex-ante and ex-post evaluation processes, increased transparency, and increased community participation. Research institutions, individuals, and firms are able to access the Lattes platform as well, so it is an open system not limited to the government, and groups become aware of their own progress and that of other teams and programs. Some areas are still weak, but the center is currently developing a system to incorporate more information from the private sector in particular, which is one of the largest gaps. “[The platform] facilitates the participation of the scientific community,” said Santos. “If the scientific community knows what’s going on, it will be better for national federal agencies to interact and allow for the community to participate, because they know they have access to information.” DISCUSSION During the question period a participant asked Viney how the U.K. Medical Research Council (MRC) convinces grantees to participate in the e-Val system, since it is more time consuming than end-of-grant reporting. Viney explained that the MRC has been successful at getting increased government funding using data from the e-Val, which they can use to leverage participation since the research community is able to see the impact of providing such detailed reports. The e-Val is also

OCR for page 61
68 MEASURING THE IMPACTS OF FEDERAL INVESTMENTS IN RESEARCH mandatory for new grants, so participants must comply if they want to receive MRC funding in the future. Responding to questions about how the impact on policy is measured, Sloan explained that the Framework Program has attempted to study the impact of their projects on policymaking by questioning participants, but has not done citation analysis of policy documents. Viney said that the MRC has looked at where MRC research is cited, paying particular attention to which documents are more influential and tracking any resulting policy changes. A workshop participant asked about the European Commission’s guarantee fund, where some money is held back until participants fulfill the requirements of the grant, and whether surveys must be completed in order to receive that money. Sloan said that it depends on how strongly the requirements are enforced, but that much of what is asked is voluntary. In response to a question asking whether a clear policy is in place requiring researchers to acknowledge their funding when they publish a paper or develop a patent, Viney said that analysis of citations and publications is based on the most reliable data possible. He said that research councils in the UK do require a standard type of acknowledgement in publications, but that the MRC could potentially do a better job working with publishers and checking compliance. Santos added that in Brazil, there are policies for federal agency funding and some state funding, but there is room for improvement so that their system is able to capture exactly who funded what.