3

The Use of Research Knowledge:
Current Scholarship

With the arrival of big social science and the growth of the policy enterprise, the federal investment in social science brought attention to whether the knowledge being produced was being used. Research on what was labeled “knowledge utilization” got under way. We address that research under three headings: decisionism and its critique, the metaphor of two communities (researchers and policy makers), and the evidence-based policy and practice initiative.

As an introduction to these issues we take brief note of the characteristics of our three central topics—social science, policy, using science—that challenge any attempt at a comprehensive account for the when, how, and why of science use in policy.

A CHALLENGING LANDSCAPE

Scholarship on what happens at the interface of science and policy has to contend with two phenomena—policy making and use—that are particularly difficult to define. To begin with, investigations of these phenomena are launched in different disciplines, including anthropology, political science, psychology, and sociology and their myriad subfields and cross-fields, from science and technology studies to political psychology, from behavioral economics to historical sociology. Each of these fields has its own established principles of evidence and inference. They use different methods—experimental, analytic, quantitative, and qualitative. They work



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 35
3 The Use of Research Knowledge: Current Scholarship W ith the arrival of big social science and the growth of the policy enterprise, the federal investment in social science brought at- tention to whether the knowledge being produced was being used. Research on what was labeled "knowledge utilization" got under way. We address that research under three headings: decisionism and its critique, the metaphor of two communities (researchers and policy makers), and the evidence-based policy and practice initiative. As an introduction to these issues we take brief note of the characteris- tics of our three central topics--social science, policy, using science--that challenge any attempt at a comprehensive account for the when, how, and why of science use in policy. A CHALLENGING LANDSCAPE Scholarship on what happens at the interface of science and policy has to contend with two phenomena--policy making and use--that are particularly difficult to define. To begin with, investigations of these phenomena are launched in different disciplines, including anthropology, political science, psychology, and sociology and their myriad subfields and cross-fields, from science and technology studies to political psychology, from behavioral economics to historical sociology. Each of these fields has its own established principles of evidence and inference. They use different methods--experimental, analytic, quantitative, and qualitative. They work 35

OCR for page 35
36 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY at different levels of analysis--from individual behavioral decision theory to systems theory. They focus on different processes: from structural deter- minism and constrained probabilities at one end of a continuum to willful effort and chance happenings at the other. They draw on epistemologies as varied as positivism, critical realism, and postmodernism. Individual social scientists bring different motivations to their work--from expansion of theoretical knowledge to practical problem solving, from mapping policy options to advocacy of particular policies. Social scientists bring their exper- tise to universities, think tanks, the media, advocacy groups, corporations, and government agencies. This range--across fields of study and individual motivations and career lines--produces a lot of variability, which, of course, determines the way the science-policy nexus is framed. Complicating matters is the absence of a generally accepted explana- tory model of policy making. Instead, multiple descriptive policy process models offer ways to understand how policy is made and how science might enter into that process. There are, for example, rational models--including linear, cycle or stage, incrementalism, and interactive. There are models that question rational model assumptions, including behavioral economics, path dependency, and bureaucratic inertia. There are political models, including policy networks, agenda setting, policy narratives, advocacy coalition frame- works, punctuated equilibrium theory, and deliberative analysis models (see Baumgartner and Jones, 1993; Hajer and Wagenaar 2003; Kingdon, 1984; Lindblom, 1968; Neilson, 2001; Sabatier, 2007; Sabatier and Jenkins- Smith, 1993; Stone, Maxwell, and Keating, 2001). There are models that focus on different stages of the policy process and thus on different ways that social science can contribute, including: descriptive analyses that present conditions needing policy attention, such as a slowdown in small business start-ups; social indicators that document long-term trends, such as gender differences in pay scales; social experi- ments on alternative policy designs, such as school vouchers; and evaluation research on the effectiveness of a policy, such as neighborhood policing.1 Political science is the discipline that has devoted the most attention to the policy process. On the issue of use, it has reached a general conclusion (Henig, in press): 1 For a careful discussion of how evidence is used at different stages of the policy process, see McDonnell and Weatherford (2012).

OCR for page 35
THE USE OF RESEARCH KNOWLEDGE 37 [T]he main thrust of the political science literature serves as a warning against idealized visions of pure data being applied in depoliticized arenas. Although generalizations about an entire discipline inevitably are oversimplifications, the center of gravity within the field encourages skepticism about proposals for a ratio- nal, comprehensive, science of public policy making and regards data and information as sources of power first and foremost. It is difficult to assess how widely this characterization is accepted outside of political science, but it is clear that the various models and frameworks do not coalesce into anything remotely resembling a powerfully predictive, coherent theory of policy making. Lacking that, it is improb- able and perhaps impossible to reach a widely agreed-upon understanding of the use of science in policy making. "Use" itself, consequently, is elusive, seen differently depending on the perspectives brought to it and the policy and institutional arenas in which it is investigated (Neilson, 2001; Webber, 1991; Weiss, 1991). A political psychologist at the Central Intelligence Agency concerned with what transforms an angry, unemployed teenager into a terrorist uses research evidence very differently from an economist at the RAND Corporation designing a randomized controlled field trial (RCFT) on classroom size and school performance. Many researchers under score the conceptual confusion about use and conclude that different definitions of use are needed and appropriate for different purposes (e.g., Oh, 1997; Rich, 1997; Weiss, 1979). This conclusion is consistent with the fact that policy choices are context dependent. A school district deciding whether to establish charter schools is less interested in a comparative study of charter and public schools across the country than in wanting to know how well a charter school will perform under its conditions, which differ depending on whether the dis- trict is in the central city or suburb, with a homogenous or diverse popula- tion, with a historically competent or incompetent school administration. The usefulness of research is not assessed in terms of variance explained from a large sample of schools, but whether it is informative about a very specific choice. Given the context-dependent nature of the use of science, typologies are a common way of mapping the landscape (for a summary, see Nutley et al., 2007; see also Bogenschneider and Corbett, 2010; Renn, 1995). A frequently cited typology is that of Weiss (1979, 1998; see also Weiss et al., 2005):

OCR for page 35
38 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY Instrumental uses occur when research knowledge is directly ap- plied to decision making to address particular problems. Conceptual uses occur when research influences or informs how policy makers and practitioners think about issues, problems, or potential solutions. Tactical uses involve strategic and symbolic actions, such as call- ing on research evidence to support or challenge a specific idea or program, such as a legislative proposal or a reform effort. Imposed uses (which is perhaps a variant on instrumental uses) de- scribe mandates to apply research knowledge, such as a require- ment that government budgeting be based on whether agencies have adopted programs backed by evidence. Other scholars add a fifth category, symbolic or ritualistic use--that is, the organizational practice of collecting information with no real intent to take it seriously, except to persuade others of a predetermined position or even to delay action (Leviton and Hughes, 1981; Shulha and Cousins, 1997). It is a frequent complaint among scientists that policy makers use scientific evidence as confirmation of prior beliefs. This complaint, however, overlooks the fact that, when policy makers argue on the basis of evidence, it is more difficult for their opponents to ignore that evidence, or to leave it unchallenged. "My science versus your science" has the merit of putting science in play, and over time opens more space for policy arguments that include scientific evidence. Weiss emphasizes that each of the four uses--which also applies to the fifth use noted--can be found in particular situations, but that no one of them offers a complete picture. Scholars who debate typologies of use generally conclude that, although typologies are heuristically valuable, they are not easily applied empirically. Boundaries are blurred, and access to users' cognitive processes is unattainable. In fact, it is unlikely that us- ers themselves can make sharp distinctions in explaining how they use knowledge (Contandriopoulos et al., 2010). The empirical application of typologies in research is difficult because use is "a dynamic, complex and mediated process, which is shaped by formal and informal structures, by multiple actors and bodies of knowledge, and by the relationships and play of politics and power that run through the wider policy context" (Nutley et al., 2007, p. 111). Typologies of use fail to meet the standard criteria of scientific typolo- gies in which each category consists of an internally coherent set of variables,

OCR for page 35
THE USE OF RESEARCH KNOWLEDGE 39 with the value of each variable predictably correlating with the values of each of the other variables in that particular category. In the periodic table of chemical elements, for example, hydrogen is distinguished from other chemical elements by its atomic weight, its specific gravity, its bonding properties, the temperature at which it freezes and boils, and other traits. Each of these traits differs consistently and predictably from those same traits in helium or in any other chemical element (see Stinchcombe, 1987). In the social world it is impossible, in any practical sense, to construct typologies that meet this standard. Typologies of social conflict, ethnic or racial groups, or government corruption are never going to have categories with internally coherent variables whose values covary in completely pre- dictable ways. It is unrealistic to expect a clear and unambiguous typology for a phenomenon as complex as the use of science in policy. To address the charge given to this committee--to understand the use of science in policy--is thus to simultaneously deal with three elusive phenomena: Scientific findings from multiple sources and that are at times contradictory; A policy-making process, that is variable along many dimensions; and A phenomenon, "use," that changes its meaning depending on the perspective brought to it and one's location in the complex space where policy is made. With this challenging landscape in mind, we turn to the recent scholar- ship on knowledge utilization. DECISIONISM AND ITS CRITIQUE The scholarship on knowledge utilization has, virtually from its begin- nings, been skeptical of rational models of the relationship between research and policy. Rational models assume that decisions unfold through five stages (Nutley and Webb, 2000, p. 25): 1. A policy problem requiring action is identified and goals, values, and objectives are clearly set forth; 2. All significant ways of addressing the problem and achieving the goals or objectives are enumerated;

OCR for page 35
40 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY 3. The consequences of each alternative are predicted; 4. The consequences are then compared with the goals and ob- jectives; and 5. A strategy is selected in which consequences most closely match the goals and objectives. Weiss and Bucuvalas (1980, p. 263) summarized the essence of this model: "a decision is pending, research provides information that is lacking, and with the information in hand the decision maker makes a decision." Ra- tional models have also been characterized as "decisionism"--"a limited number of political actors engaged in making calculated choices among clearly conceived alternatives" (Majone, 1989, p. 12; see also Rein and White, 1977; Rich, 1997). Criticisms of this model have focused on several significant defects; for example, that decisions made are optimal, that is, based on complete information and an examination of all possible alternative courses of action (see the work of Simon [1957], who introduced satisficing as a replace- ment for maximizing); or, that the model is a normative account of policy making (see the work of Braybrooke and Lindblom [1963] and Lindblom [1959], authors who substitute incrementalism for rational models). Other critics argue that rational models underemphasize or ignore the important role that value judgments play in policy arguments (Brewer and deLeon, 1983); or that linear problem solving is "wildly optimistic," because it "takes an extraordinary concatenation of circumstances for research to influence policy decisions directly" (Weiss, 1979, p. 428). More recent examinations of the relationship between research and policy making echo these concerns. For example, Gormley (2011, pp. 978- 979) notes: A hypodermic needle theory of scientific impact on policy, which anticipates direct, immediate, and powerful effects, is flawed for several reasons. First, scientific research is one of many inputs into the policy process. . . . Second, scientific knowledge accumulates through multiple studies, some of which reach different conclu- sions. . . . Third, the applicability of a given study to a particular policy choice is a matter of judgment. . . . Fourth, scientific research is translated, condensed, repackaged, and reinterpreted before it is used. Fifth, the use of scientific information by public officials, when it is occurs, is more likely to involve justification

OCR for page 35
THE USE OF RESEARCH KNOWLEDGE 41 (reinforcement of a prior opinion) than persuasion (conversion to a new opinion). Although we share Gormley's view, there are situations in which dis- crete decisions are directly triggered by the use of some specific scientific knowledge--for example, the direct, even formulaic translation of census results into congressional apportionment or formula-based fund allocations that are legislatively required. There also are situations in which a user is considered sovereign in her or his capacity to mobilize evidence and, con- sequently, to modify her or his behavior on the basis of that evidence--for example, the choice of a preferred clinical treatment (Contandriopoulos et al., 2010). But these examples are exceptions to the rule, and uncommon at that. It is estimated that evidence-based programs accounted for less than 0.2 percent of nonmilitary discretionary spending in fiscal 2011.2 In almost all decision-making situations, the use of science takes place in "systems characterized by high levels of interdependency and intercon- nectedness among participants" (Contandriopoulos et al., 2010, p. 447). No single decision maker has the independent power to translate and apply research knowledge. Rather, multiple decision makers are embedded in sys- temic relations in which use not only depends on the available information, but also involves coalition building, rhetoric and persuasion, accommoda- tion of conflicting values, and others' expectations. In criticizing rational models and decisionist thinking, Weiss and others suggest that use is less a matter of straightforward application of scientific findings to discrete decisions and more a matter of framing issues or influencing debate (Weiss, 1978, p. 77): Social science research does not so much solve problems as pro- vide an intellectual setting of concepts, propositions, orientations and empirical generalizations. . . . Over a span of time and much research, ideas . . . filter into the consciousness of policy-making officials and attentive publics. They come to play a part in how policy makers define problems and the options they examine for coping with them. 2 The George W. Bush administration piloted a program linking federal financing to clear demonstration of program effectiveness. These evidence-based programs "accounted for about $1.2 billion out of a $670 billion budget for nonmilitary discretionary programs in the 2011 fiscal year" (Lowrey, 2011).

OCR for page 35
42 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY Although Weiss suggested that this enlightenment model is perhaps the way science is most frequently used in policy making, she did not claim it was the way it ought to happen. "Many of the social science understandings that gain currency are partial, oversimplified, inadequate, or wrong. . . . The indirect diffusion process is vulnerable to oversimplification and distortion, and it may come to resemble `endarkenment' as much as enlightenment" (Weiss, 1979, p. 430). In sum, the research on knowledge utilization reflects a consensus about what should be ruled out: (1) that the science/policy nexus can be uniformly understood in terms of rational decision-making models; (2) the assumption of a specified single actor with freedom to achieve goals formu- lated through a careful process of rational analysis characterized by a com- plete, objective study of all relevant information and options; and (3) the definition of use as problem solving in the sense of a direct application of evidence from a specific set of studies to a pending decision. Although evidence may occasionally be used in such narrow ways, these depictions of "use" do not accurately reflect the full realities of policy making. Knowledge utilization research, in agreement about what is ruled out, is less clear about what should be ruled in. It has, however, pointed to the importance of closing the distance between the "two communities" of sci- entists and policy makers. THE TWO COMMUNITIES METAPHOR Viewing use from the perspective of two communities has been a recur- ring motif in knowledge utilization studies (see Caplan, 1979). The basic idea is refreshingly simple. Scientists and policy makers are separated by their languages, values, norms, reward systems, and social and professional affiliations. The primary goal of scientists is the systematic search for a reli- able and accurate understanding of the world; the primary goal of policy makers is a practical response to a particular public policy issue. Like any binary distinction, this one oversimplifies, though there is a crude truth to several distinctions rooted in the different tasks facing researchers and policy makers. They differ in the outcomes they value-- knowledge about the world in all its complexities versus knowledge helpful in reaching feasible solutions to pressing problems--and in the incentives, rewards, and cultural assumptions associated with these different out- comes. They also differ in habits of expression--probabilistic versus certain statements about conditions or people. And they differ even in modes of

OCR for page 35
THE USE OF RESEARCH KNOWLEDGE 43 thought--deductive and general versus inductive and particular (Szanton, 2001, p. 64). This is described as "research think" and "political think." The "culture of the researcher tends to add complexity and resist closure. The culture of the political actor tends to demand straightforward and easily communicated lessons that will lead to some kind of action" (Henig, 2009, p. 144). Differences between the two communities are associated with a con- trasting list of supply-side and demand-side problems (Bogenschneider and Corbett, 2010; Furhman, 1994; Nutley et al., 2007; Rosenblatt and Tseng, 2010). On the supply side are researchers who fail to focus on policy- relevant issues and problems, cannot deliver research in the time frame generally necessary for effective policy making, do not relate findings from specific studies to the broad context of a policy issue, ineffectively commu- nicate their findings, depend on technical arguments that are inaccessible to policy makers, and lack credibility because of perceived career interests or even partisan biases. On the demand side are policy makers who fail to spell out objectives in researchable terms, have few incentives to use science, and do not take time to understand research findings relevant to pending policy choices. This framing of the use problem offers little guidance as to which of the long list of factors, from either side, best explains variance in use, let alone how the factors interact and whether they apply only in specific settings or have general applicability (Bogenschneider and Corbett, 2010; Johnson et al., 2009). Although the two communities framework has been helpful in understanding the differing expectations of researchers and policy makers and problems of communication between them, it has not been able to offer a systematic explanation of use. Thinking about how best to bridge the gap between the two communities has, however, led to practices of translation and brokering and to more intensive interactions between researchers and policy makers. Translation Translation is a supply-side solution to the use problem. It was devel- oped in clinical diagnostic, preventive, and therapeutic practices. The idea is simple: basic science is translated into clinical efficacy, efficacy is translated into clinical effectiveness, and effectiveness is translated into everyday health care delivery (Drolet and Lorenzi, 2011). The oft-invoked catchphrase is "bench to bedside." One important sign of the seriousness with which

OCR for page 35
44 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY translation is taken is the U.S. Department of Health and Human Services initiative, Translating Research into Practice (TRIP) Program, that focuses on implementation techniques and factors associated with successfully translating research findings into diverse applied health care settings (see Agency for Healthcare Research and Quality, 2012). Translational strategies have now moved beyond health care, intro- ducing additional and somewhat differently focused activities. One is evi- dence-based registries, a compilation of scientifically proven interventions. They are considered tools to improve practice in various fields, including social services, criminal justice, and education. A different initiative is the Campbell Collaboration,3 an international organization conducting system- atic reviews of the effects of social interventions. The translation strategy is well institutionalized in education. The U.S. Department of Education's Institute of Education Sciences (IES) was estab- lished in part to develop the science that could be translated into strategies to change education practice in public schools. The What Works Clearing- house of the IES aims to provide educators, policy makers, and the public with an independent, and trusted source of scientific knowledge relevant to education policies and practices.4 IES also supports 10 regional educational laboratories, the role of which is similar to that of extension agents in the agricultural field: taking research results and putting them into practice in school districts and classrooms (see U.S. Department of Education, 2012). The movement toward evidence-based approaches in practice settings began more than 40 years ago in medical practice. Archibald Cochrane (1972) railed against ineffective and sometimes harmful therapies despite randomized clinical trials showing that better treatments were available. In response to his call for systematic reviews of such trials, the Cochrane Collaboration5 was established. Its rigorous model of research synthesis has been adopted in other fields, including the above-noted Campbell Collaboration and the What Works Clearinghouse. Although translation strategies have largely been applied to practices, the logic of translation is applicable to questions of using science in policy. Begin with a dependable, valid scientific base that provides evidence about 3 See the Campbell Collaboration: What Helps? What Harms? Based on What Evidence?, available: http://www.campbellcollaboration.org/ [August 2012]. 4 For example, see the IES guides in education, such as "Turning Around Chronically Low-Performing Schools" (May 2008): available: http://ies.ed.gov/ncee/wwc/practiceguide. aspx?sid=7 [July 2012]. 5 See the Cochrane Collaboration: available: http://www.cochrane.org/index.htm [August 2012].

OCR for page 35
THE USE OF RESEARCH KNOWLEDGE 45 what works so that policy makers can readily grasp its relevance to the deci- sion or task at hand, and make that science available in the form of research summaries or lists of demonstrably effective social interventions. The re- search record, however, is far from clear on whether translation (of either social or medical science research) works and is an effective strategy for enhancing use (see, e.g., Glasgow and Emmons, 2007; Green and Seifert, 2005; Lavis, 2006; Slavin, 2006). Brokering While translation is primarily a matter of repackaging technical find- ings in terms more readily consumable by policy makers, brokering is a two-way conversation aided or mediated by a third party. Brokering involves filtering, synthesizing, summarizing, and disseminating research findings in user-friendly packages. It is generally seen as the task of intermediary orga- nizations, such as think tanks, evaluation firms, and policy-oriented organi- zations, including those focusing on specific target populations or specific social issues as well as those organized around particular political persuasions. These organizations (Bogenschneider and Corbett, 2010, p. 94): do research and evaluation, but they also have one foot in the policy world. They see policymakers as their primary clients. In addition to producing knowledge, they also see their role as trans- lating extant research and analysis in ways that enhance their utility for those doing public policy. . . . To greater and lesser degrees, these firms bridge the knowledge-producing and knowledge- consuming worlds. Science and technology studies describe brokering as occurring in boundary organizations occupying a territory between research and policy making (Guston, 2000).6 In contrast to translation strategies that gener- ally are one-way efforts in dissemination, brokering involves interaction and two-way communication. Intermediary organizations and knowledge brokers are increasingly being viewed as critical in promoting the capacity for evidence-based, or evidence-informed, decision making (e.g., Dobbins et al., 2009a). 6 In this view, the National Research Council can be viewed as a brokering organization, synthesizing research in a consensus-based process and then presenting it in a form intended to contribute to improved policy making.

OCR for page 35
46 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY If brokering occurs, use is not something that happens when experts "here" hand off research to policy makers "there." A brokering model views use as emerging from multidirectional communication and ongoing negotiation among researchers, policy makers, planners, managers, service providers, and even the public. Often this interactive process will involve consideration of more than one stream of research as relevant to a given policy (e.g., Sudsawad, 2007). To bridge the gap between the differing cultures of the producers and consumers of scientific knowledge will require, according to some s cholars, cultural changes in each community. Bogenschneider and Corbett (2010, pp. 299 ff.) write that the culture of research should change, perhaps through education and training on how to do more policy-relevant research, devel- oping incentives for doing such research and developing opportunities to work with policy makers. The user or consumer culture should also change, perhaps by institutional innovations that improve policy makers' access to research, helping them communicate their policy needs to researchers, and providing forums to discuss research agendas. In more ambitious formula- tions, research literacy of the general public should be improved through education (see also Carr et al., 2007; Gigerenzer et al., 2008). An Interaction Model Closing the distance between the two communities has taken an ad- ditional step in what is labeled the interaction model (Contandriopoulos et al., 2010; Greenhalgh et al., 2004). This model goes beyond transfer, diffusion, and dissemination and even beyond translation and brokering. The interaction label covers a family of ideas directed to systemic changes in the means and opportunities for relationships between researchers and policy makers (Bogenschneider and Corbett, 2010). It holds that the rela- tion between researchers and users is not only not linear it is iterative and even "disorderly" (Landry et al., 2001, p. 335). One source for an interest in interaction is science and technology studies documenting the co-evolution of social and technological systems (Jasanoff, 2004; Jasanoff et al., 1995). Another source is the use of systems thinking to better understand the complex adaptive systems involved in diagnosing and solving public health problems and the interactions among the design of prevention interventions, testing their efficacy and effectiveness, and dis- seminating innovations in community practices. A third is the emphasis on practical reasoning, the argumentative turn in policy analysis discussed in

OCR for page 35
THE USE OF RESEARCH KNOWLEDGE 47 the next chapter (Fischer and Forester, 1993; Hajer and Wagenaar, 2003; Hoppe, 1999). Research that works in close proximity to practice settings illustrates the interaction framework. First noted in corporate research (Pelz and Andrews, 1976), and later in the life sciences (Louis et al., 1989), the publication of Pasteur's Quadrant (Stokes, 1997), with its emphasis on use-inspired research, increased its visibility. This research influenced how the National Academy of Education (1999) set research priorities and its interest in how to hold policy specialists, researchers, and professional educators, program developers, and curriculum specialists collectively accountable for educational outcomes. Collaborations of this kind formed the basic design concept for the Strategic Education Research Partnership. These involved connecting researchers to teachers, bringing in research communities, school administrations, and edu- cational policy makers (see National Research Council, 1999a; Smith and Smith, 2009). The Carnegie Foundation for the Advancement of Teaching and Learning is also promoting a framework for research and development labeled improvement research (Bryk et al., 2011), which synthesizes the work of researchers and practitioners. In this spirit, the Institute of Medicine (IOM) created a Roundtable on Evidence-Based Medicine, which then became the Roundtable on Value & Science-Driven Health Care, to foster interaction among stakeholders interested in building a continuously learning health care system in which science, information technology, incentives, and culture are aligned to bring together evidence-based practice and practice-based evidence (see Green, 2006). This effort and its attendant workshops (Institute of Medicine, 2007, 2010b, 2011a, 2011b) stress the importance of rigorous science and applying the best evidence available. The goal is understanding how health care can be restructured to develop knowledge from science and from the health care process and to then apply it on many fronts: health care delivery and health improvement, patient and public engagement, health profes- sional training, infrastructure development, measurement, costs and incen- tives, and policy. The IOM's reports on these activities draw attention to active collaboration, exchange, and appraisal of research and policy and to what is known by researchers and users of research about practice--drawn from the life-cycle of therapies, their development, testing, introduction, and evaluation. As attractive as these initiatives are, there are cautionary voices. There is a difference across political time, policy time, and research time. One should take care not to mistake one for another (Henig 2009, p. 153):

OCR for page 35
48 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY The pressure for fast, simple, and confident conclusions, however, is generated by the needs of politicians--not necessarily the needs of the policy. Political time is defined by election cycles, scheduled reauthorization debates, and the need to respond to short-term crises or sudden shifts in public attention. But a consideration of the history of public policy suggests that societal learning about complex problems and large-scale policy responses takes place on a much more gradual curve. Interaction models offer an insight into what the use of science means in practice. Evidence from science is not simply there for the taking. It emerges and is made sense of in the particular circumstances that give rise to a policy argument (see Chapter 4 for discussion of policy argument). "Mak- ing sense" is iterative. It involves negotiating what kind of situation-specific knowledge is relevant to a policy choice, whether it is firmly established and available under the constraints of time and budget, and what political consequences might follow from using it. In this framework, formal link- ages and frequent exchanges among researchers, policy makers, and service providers occur at all steps between knowledge production and knowledge use (Huberman and Cox, 1990). What emerges is a social as well as a techni- cal exercise. Conklin et al. (2008, p. 7) explain this framework: Strategic interactions (between human actors within and between organizations) therefore address both sides of the research-policy interface. On the one hand, decision-makers highlight policy relevant research priorities; on the other hand, researchers can in- terpret research findings in local contexts. In so doing, a common understanding of a policy problem, and its possible solutions, is built between different actors in the two communities. . . . Spillane and Miele (2007) underscore the point in observing that what information is noticed in a particular decision-making environment, whether it is understood as evidence pertaining to some problem, and how it is eventually used all depend on the cognitions of the individuals operat- ing in that environment. Furthermore, what these actors notice and make sense of is determined in part by the circumstances of their practice environ- ment. Examining use, then, also requires examining "the practice of sense making, viewing it as distributed across an interactive web of actors and key aspects of their situation--including tools and organizational routines"

OCR for page 35
THE USE OF RESEARCH KNOWLEDGE 49 (p. 49). It also introduces the idea that research might "be interpreted and reconstructed--alongside other forms of knowledge--in the process of its use" (Nutley et al., 2007, p. 304). Focusing on understanding institutional arrangements--how the agencies, departments, and political institutions involved in policy making operate and relate to one another--may be what matters most in improving the connection between science and policy making. For example, a study of drug misuse in government agencies in Scotland and England (Nutley et al., 2002) suggests that three aspects of microinstitutional arrangements within and between the agencies mattered a great deal in understanding how research evidence was (or was not) used: 1. How different agencies integrated research with other forms of evidence, 2. How agencies collectively dealt with the fragmentation of research evidence resulting from different agencies produc- ing different types of evidence given their respective research cultures, and 3. What mechanisms were in place to integrate evidence and policy making (co-location of research and policy staff, cross- government work groups, establishment of quasi-policy bodies that specialize in the substance of a policy domain, etc.)? Nutley et al. (2007, pp. 319-320) conclude [T]here is now at least some credible evidence to underpin [their view] . . . that interactive, social, and interpretive models of re- search use--models that acknowledge and engage with context, models that admit roles for other types of knowledge, and models that see research use being more than just about individual behav- ior--are more likely to help us when it comes to understanding how research actually gets used, and to assist us in intervening to get research used more. . . . If this conclusion holds up, it is a step toward accumulating what the committee believes is lacking: understanding institutional arrangements that facilitate the use of science in policy. There is an important cautionary observation about efforts to overcome the "two communities" challenge. There are tensions between scientific

OCR for page 35
50 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY engagement with practical policy problems and the long-standing assump- tion that science maintains its authority by virtue of its independence from politics (Jasanoff, 1990; Jasanoff et al., 1995). Persons working to bring sci- entists and policy makers closer need to be mindful that this tension is never far from how scientists think about and engage the policy uses of their work. EVIDENCE-BASED POLICY AND PRACTICE Current discussions about the use of research knowledge are heavily influenced by "evidence-based policy and practice." The goal is realizing better and more defensible policy decisions by grounding them in the conscientious, explicit, and judicious use of the best available scientific evi- dence ( Davies et al., 2000). The initiative explicitly rejects habit, tradition, ideology, and personal experience as a basis for policy choices: they are to be replaced with a more dependable foundation of "what works," that is, what the evidence shows about the consequences of a proposed policy or practice. With access to an evidence base, argue the proponents, policy makers will make better decisions about the direction, adoption, continuation, modi- fication, or termination of policies and practices. Dunworth et al. (2008, p. 7) note: [W]hile scientific evidence cannot help solve every problem or fix every program, it can illuminate the path to more effective public policy. . . . [T]he costs and lost opportunities of running public programs without rigorous monitoring and disinterested evalua- tion are high . . . without objective measurements of reach, impact, cost effectiveness, and unplanned side effects, how can government know when it's time to pull the plug, regroup, or, in business lingo, "ramp up?" The use of science is, of course, not a logical or inevitable outcome of having the science. In fact, the normative claim that policy should be grounded in an evidence base "is itself based on surprisingly weak evidence" (Sutherland et al., 2012, p. 4). The approach of evidence-based policy and practice assumes that there is an agreement among policy makers and researchers on what the desired ends of policy should be. "The main contribution of social science research is to help identify and select the appropriate means to reach the goal" (Weiss 1979, p. 427). This, in turn, depends on the quality of the science providing

OCR for page 35
THE USE OF RESEARCH KNOWLEDGE 51 evidence to the policy maker, and thus the evidence-based approach places a premium on improving policy-relevant research, often through the use of RCFTs. In the settings in which they are carried out, RCFTs provide a strong, if not the strongest, form of scientific evidence of cause and effect. Circum- stances may permit such experiments in a desired setting, such as when scarce resources are allocated by lottery, for example with admission to magnet schools or charter schools or the allocation of health care resources. An example of the latter is the Oregon Health Insurance Experiment in which names were drawn by lottery for the state's Medicaid program for low-income, uninsured adults (Finkelstein et al., 2012). Even when RCFTs are conducted in one setting, inference from them may be applied to other settings or contexts with concurrent collection of information on other variables or factors that differ in different settings and that may influence the results. So-called substitutes for randomized trials, however, such as "natural" experiments and "quasi-experiments," as Sims (2010) argues, are not actually experiments. They are often invoked as a way to avoid confronting "the complexities and ambiguities that inevitably arise in nonexperimental inference." For these situations and even in conjunc- tion with randomized experiments, there are nonexperimental methods of drawing causal inferences and model-based methods for adjusting experi- mental results for inherent biases. Appendix A provides a review of some of these research methods and sets them in the context of the varied statistical methods for research and evaluation. The active debate regarding the appropriate methodology for a given research question promotes attention in the policy community to the desirability of producing the best possible evidence under a given set of circumstances, especially the strongest evidence that bears on policy imple- mentation and policy consequences. Bringing attention to the importance of strong evidence in policy making advances the goal of using science even though the specific formulation of an evidence-based policy approach offers little insight into the conditions that bring about its use. CONCLUSION Despite their considerable value in other respects, studies of knowledge utilization have not advanced understanding of the use of evidence in the policy process much beyond the decades-old National Research Council (1978) report. The family of suggestive concepts, typologies, and frame-

OCR for page 35
52 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY works has yet to show with any reasonable certainty what changes have oc- curred in the nature, scope, and magnitude of the use of science as a result of different communication strategies or different forms of researcher-user collaborations (Dobbins et al., 2009b; Mitton et al., 2007). There is little assessment of whether innovations said to increase the use of science in policy have had or are having their desired effects. A recent study reporting the results of a collaborative procedure among 52 participants covering a range of experiences in both science and policy identified 40 (!) key unanswered questions on the relationship between sci- ence and policy--this despite nearly four decades of research on the ques- tion of "use" (Sutherland et al., 2012). One extensive review of the literature reaches the striking conclusion that knowledge use is "so deeply embedded in organizational, policy, and institutional contexts that externally valid evidence pertaining to the efficacy of specific knowledge exchange strate- gies is unlikely to be forthcoming" (Contandriopoulos et al., 2010, p. 468 [italics added]). Our conclusion is not that pessimistic. If "use" is broadly understood to mean that science--or, more specifically, in the language of evidence- based policy and practice, scientific evidence of the effectiveness of interventions--is incorporated into policy arguments, we agree that there probably will never be a definitive explanation of what strategies best facili- tate or ensure that incorporation. But this conclusion does not rule out that the possibility that new approaches in the study of the science-policy nexus might reveal factors or conditions that have thus far been missed. Perhaps the preoccupation with defining use, identifying factors that influence it, and determining how to increase it has detracted from the search for alter- native ways in which social science can contribute to understanding the use of science in policy. That possibility is the subject of Chapter 4.