Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 65
5
The Next Generation of
Researchers and Practitioners
T
he three actors central to advancing and applying the research
framework now outlined are (1) established scholars in the fields
and specialties identified in Chapter 4; (2) Ph.D. candidates in
those fields and specialties; and (3) administrators and faculty responsible
for curricula in schools and programs summarized below by the term
"policy education." For the first two of these actors, there are historical and
contemporary models we briefly note; the third will involve fresh thinking.
ESTABLISHED SCHOLARS
In 1923, the Social Science Research Council (SSRC) was established
to promote "co-operative research among the several disciplines" (Fosdick,
1952, p. 198),1 as a necessary foundation for creating entirely new research
fields and specialties. Later the term "field development" was coined. We
use that term to describe a coordinated and well-funded effort to attract es-
tablished scholars to important but under-researched issues for which their
theories and methods are appropriate.
Citing metaphors favored in that more naïve time, the director of the
Laura Spelman Rockefeller Memorial, Beardsly Ruml, who funneled mil-
lions of dollars to research universities and the SSRC in the 1920s, lamented
that "All who work toward the general end of social welfare are embarrassed
1
This early plea for interdisciplinary research did not use the term, which did not appear (as
interdiscipline inquiries) until SSRC's Sixth Annual Report, 1929-1930 (noted by Sills, 1986).
65
OCR for page 66
66 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY
by the lack of that knowledge which the social sciences must provide."
Ruml offered what for him was the clinching argument (cited in Fosdick,
1952, p. 194):
It is as though engineers were at work without an adequate devel-
opment of physics and chemistry, or as though physicians were
practicing in the absence of the medical sciences. The direction of
work in the social field is largely controlled by tradition, inspira-
tion and expediency.
Ruml and the SSRC leadership had a clear goal: to professionalize the
social sciences, provide them methodological tools necessary for rigorous
research, and point them toward important fields of investigation.
Ruml was not naïve about the challenges: data were meager; research
was based on second-hand observations and anecdotal material; classroom
instruction isolated students from social conditions; and, especially, the so-
cial sciences were challenged to investigate topics that could not "be brought
into the laboratory for study," but "must be observed if, when, and as op-
erative." Difficulties notwithstanding, "unless means are found for meeting
the complex social problems that are so rapidly developing, our increasing
control of physical forces may be increasingly destructive of human values"
(cited in Fosdick, 1952, p. 195).
We bring this early philanthropic initiative to mind to draw a lesson
still applicable. Targeted funds can help develop new research specialties.
The well-funded SSRC emphasized interdisciplinary research and a strong
commitment to empirical methods. Social science researchers responded
not only to the SSRC, but also to the program priorities announced by oth-
er philanthropic foundations, to the Russell Sage Foundation in more than
a century of social science funding and to the larger foundations--Ford,
Carnegie, Hewlett, and MacArthur, among others--in the second half of
the 20th century. The label field development, for example, was attached
to area studies, a Cold War era success story. Coordinated conferences,
workshops, research monographs, and edited volumes advanced research
focused on showing how recently decolonized countries could engage in
"nation building" and how western democracies should meet the threat of
Communism, which in turn spawned a generation of research on the Soviet
Union and China watchers (so-called because lacking access to the Chinese
mainland, they "watched" from Hong Kong). There are many examples
of new research fields promoted by private foundations and government
OCR for page 67
THE NEXT GENERATION OF RESEARCHERS AND PRACTITIONERS 67
funders--behavioral economics, human dimensions of climate change,
population studies, life course development, aging, and race, ethnic, and
gender studies.
An important example directly related to understanding the use of
social science across a broad array of public policies is the ambitious effort
pioneered by neoconservative social scientists who were skeptical about the
effectiveness of many Great Society programs. With the Olin Foundation in
the lead, private funds subsidized books, endowed university professorships,
offered student fellowships, and established think tanks--similar to strate-
gies earlier pioneered by the Spelman Rockefeller Memorial Fund--"all
with the intent of changing the prevailing terms of debate" and advancing
market-sympathetic policy (Rodgers, 2011, p. 7). This effort shaped the
ongoing debates about the respective merits of the state and the market
with respect to a long list of social policies--including whether poverty was
better reduced by social welfare government programs or market forces, and
whether school reform was better advanced through vouchers and school
choice than leaving many educational practices under the influence of
teacher unions.
Field development is not limited to foundations, though they have
been particularly adept at it. The U.S. National Science Foundation (NSF)
is attentive to how its funds might shape fields of inquiry, noting that in
addition to its reliance on a well-established peer review process to guide its
grant making, program officers should "identify promising research that re-
sponds to national priorities identified by Congress and the Administration"
and to "incorporate agency or programmatic priorities" in NSF funding
(Marrett, 2011, p. 3).2 Particularly important to our purpose, the portfolio
of grants funded by the NSF is expected to achieve "special program objec-
tives and initiatives" and to build "capacity in a new and promising research
area" (Marrett, 2011, p. 5).
A highly visible and in some quarters sharply criticized foray is the
recent Science of Science Policy Initiative (Fealing et al., 2011). The broad
purpose is to develop an evidentiary base for policy decisions on invest-
ments in basic and applied scientific research. New federal programs associ-
ated with this purpose include the Science of Science and Innovation Policy
at NSF and an interagency task force sponsored by the National Science and
Technology Council. A virtual community of practice has been organized,
2
For a technical description of NSF's merit review policy, see http://www.nsf.gov/bfa/dias/
policy/meritreview/ [February 2012].
OCR for page 68
68 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY
facilitated by the establishment of a website hosted by the Office of Science
and Technology Policy in the Executive Office of the President.3 We cite
this example not to endorse it (for a critical analysis, see Feller, 2011) but
to illustrate how federal funding is used to develop new fields of inquiry:
in this example, it is an attempt to build a community of practice among
researchers and between researchers and policy makers.
Two sponsors of our study, the William T. Grant Foundation and the
Spencer Foundation, have specifically targeted funding to better understand
the use of research in policy and practice with respect to children and youth
(W.T. Grant) and data and information to improve education (Spencer).
The W.T. Grant Foundation sponsors research on the acquisition, interpre-
tation, and use of research evidence to develop "strong theory and empirical
evidence on when, how, and under what conditions research is used." Its
request for proposals notes that "[r]esearch acquisition, interpretation, and
use occurs within a social ecology" and that the foundation seeks "to under-
stand how organizational, social, economic, and political contexts matter"
(William T. Grant Foundation, 2012).4
The Spencer Foundation's Evidence for the Classroom Project, part
of its broader Data Use and Educational Improvement Initiative, sponsors
research on the assumptions behind data-based educational reforms "by
investigating whether, when, and how student performance data informs
instruction in K-8 classrooms." The goal is "to learn more about how K-8
teachers use student performance data for instructional decisions and how
organizational and individual factors affect that use." Included in this
initiative is research on how organizations learn and improve (Spencer
Foundation, 2012).
In a review of Spencer-funded research papers published in the Ameri-
can Journal of Education, Goren (2012) notes that the papers "call for a
deeper and better understanding of data, their use, the conditions that are
most conducive for using data well, how individuals and groups of practitio-
ners make sense of the data before them, and the intended and unintended
consequences of data use for school improvement" (p. 233). The summary
conclusion laments "that our understanding of how data lead to improve-
ment in education is tremendously underdeveloped" (p. 234).
These Grant and Spencer examples are consonant with the research
agenda described in Chapter 4. As valuable as they are, however, they
3
For details, see http://www.scienceofsciencepolicy.net [February 2012].
4
For a description of this research program and early lessons from its funded research, see
Tseng (2012) and the accompanying commentary.
OCR for page 69
THE NEXT GENERATION OF RESEARCHERS AND PRACTITIONERS 69
touch on a small subset of the issues that need study in order to develop a
deeper and wider understanding of the use of science in the policy context.
The Grant Foundation initiative is limited to children and youth, and the
Spencer Foundation initiative is limited to data use as a particular feature of
educational practice. Similarly, the NSF example noted above is limited to
science policy, and it is narrow in its selection of research methods.
If these initiatives are joined by sponsored research on how science
is used as evidence in many other policy areas--international security,
economic growth, renewable energy, transportation efficiency, agricultural
productivity, etc.--and are targeted to methods and approaches described
in Chapter 4, a new research field on the scale of area studies or behavioral
economics would take shape. Of course, established scholars have already
worked out their future research, and we cannot expect more than a small
percent to shift their interests to the framework in Chapter 4 (though we
welcome being proven wrong). Science funders have long accepted this re-
ality, and have often focused on entry-level researchers as better candidates
for launching new fields and specialties. With this in mind, we turn next
to Ph.D. candidates.
PH.D. TRAINING: AN ENTRY POINT
A well-tested strategy for establishing new research fields provides
incentives early in a person's research career, especially at the dissertation
phase. The SSRC pioneered this approach in the 1920s, eventually offer-
ing hundreds of fellowships in the social sciences that produced leaders in
the academically based departments and in the steadily growing array of
policy institutions (Fosdick, 1952, pp. 230-231). Another major chapter
in the history of philanthropic leadership was the substantial, decades-long
funding of graduate training in languages and area studies by the Ford and
Mellon Foundations, in service of enlightened foreign policy. The 1958 Na-
tional Defense Education Act (Title VI) added federal funds to this effort.
In more recent decades, dissertation grants provided by the MacArthur
Foundation added depth to international security education, successfully
reorienting the field from a 1960s focus on a limited array of issues, primar-
ily arms control, to a broader consideration of how international economics,
global immigration, climate change, and other "nonsecurity" issues were,
in fact, deeply implicated in how the nation should approach its security
challenges in the 21st century. The predoctoral research training program
in the neurosciences, sponsored by the National Institutes of Health (NIH),
OCR for page 70
70 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY
encouraged broad, early-stage training in the neurosciences. This program
was targeted to basic and disease-related research of importance to the
participating institutes.5 A successful current effort is NSF's Integrative
Graduate Education and Research Traineeship (IGERT) Program. Initiated
in 1997, the program is intended "to establish new models for graduate
education and training in a fertile environment for collaborative research
that transcends traditional disciplinary boundaries" (National Science
Foundation, 2012c).
These funding initiatives, and there are others, have in common a
determination to establish new fields by starting with researchers in the
earliest stages of their professional training. The strategy rests on a simple
assumption. Ph.D. candidates searching for a dissertation topic are attracted
to new areas, where a single study can quickly be influential. The disserta-
tion is the basis for their early publications, which, if cited, keeps them on
this track. Enough young scholars on a similar track begin to establish a
new field. This time-tested strategy fits with a central point of this report:
attracting a fresh generation of researchers to studies of the use of science
in policy should not be difficult in this period of heightened political (and,
we expect, funder) attention to whether the substantial public investment
in science--social sciences included--results in science that is used. The list
of research topics is long--this is a small sample:
· Challenges in linking the natural and social sciences in the policy
context;
· How variability in the quality of scientific evidence affects its use;
· The role of intermediaries in promoting evidence use;
· The responsiveness of policy makers to commissioned research;
· The interaction of scientific claims and value claims in policy ar-
gument; and
· Comparative research that considers how different government
systems produce and use scientific evidence for policy and how
this relates to differing political systems and beliefs about the role
of government.
Based on their disciplinary training--in systems analysis, studies of
complex organizations, science and technology studies, social psychology,
5
For details, see http://grants.nih.gov/grants/guide/pa-files/PAR-00-037.html [February
2012].
OCR for page 71
THE NEXT GENERATION OF RESEARCHERS AND PRACTITIONERS 71
behavioral economics, political science, statistics, cognitive sciences, and the
history of science--Ph.D. candidates can start with the substantial research
literature on how scientific knowledge is produced and proceed quickly to
what is not known about how science is used as evidence in policy making,
and then apply methods and theories, already available from their disci-
plinary training, best suited to remedying the gaps in knowledge. These
beginning scholars are guaranteed two attentive audiences for their work.
There is an influential audience of public and private science funders, gov-
ernment agencies, institutes, think tanks, lobbyists, and others with a stake
in whether relevant scientific knowledge is brought to bear in policy. The
second audience is faculty responsible for what is being taught to students
en route to careers in the policy enterprise, to which we now turn.
POLICY EDUCATION: WHAT IS NEEDED
Training beyond the bachelor's degree is a minimum job requirement
for almost all public policy positions. Perhaps mentoring and on-the-job
learning worked in an earlier period, when policy challenges slowly made
their way to the public agenda and arrived as fairly straightforward ques-
tions of whether X leads to Y. That world, if it ever really existed, is clearly
not today's policy world. A nation dependent on policy analysts and policy
makers who learn as they go is put at risk when policy challenges (as well
as information, both helpful and unhelpful) arrive at bewildering speed,
from unexpected directions, and in ever more complex forms. Professional
preparation is the norm today, and university-based programs are where
that preparation occurs.
Senior policy positions often require (or assume) Ph.D.-level training,
but a significant number of positions in the policy enterprise recruit from
programs leading to a master of public administration (M.P.A.), the degree
traditionally offered in schools of public policy, though now more likely to
be labeled master of public policy (M.P.P.). This relabeling reflects the shift
from careers in the civil service to those in the policy enterprise. Related
training takes place in other professional schools, especially law, business,
public health, social work, and education. There are also programs focused
on particular policy arenas, such as environmental policy, security policy,
and urban policy. Some of these have become stand-alone master's degrees,
an increasing practice in higher education (Radin, 2000). Although the
United States leads the world in establishing public policy schools and pro-
grams, similar initiatives are now found on every continent and in steadily
OCR for page 72
72 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY
growing numbers. We use the generic term "policy education" to cover the
M.P.A., M.P.P., topical master's degrees, and related certificate programs.
This array of programs presents an obvious entry point for introducing
fresh ways of thinking among those who will practice policy analysis and
program design. Their education should be based on two priorities. One is
now being taught--acquiring the competencies relevant to assessing policy-
relevant research knowledge. One is not--developing a clear understanding
of the factors that influence the conditions under which that knowledge is
likely to be used.
These joint priorities distinguish policy education from what is pro-
vided in academic departments, where the priority is primarily the discovery
of new knowledge--even recognizing that academic social scientists increas-
ingly hope that their research will be used. Policy education also differs from
what aspiring political consultants and policy advocates seek (though many
looking for such careers earn a M.P.A. or M.P.P.), which are skills relevant to
advancing a political cause or winning a policy battle. The academic social
sciences adequately attend to the education of advanced students whose
vocation is the discovery and dissemination of knowledge. The political
world adequately supplies on-the-job training for those whose vocation is
winning through bargaining and compromising, media campaigns, mo-
bilization of support, and using science evidence selectively and tactically.
Neither the academically oriented nor the politically motivated student
is the audience we have in mind. Rather, it is the student whose priority is
bringing scientific evidence to bear on policy choices, and wanting this not
for tactical reasons but because it is a core professional principle. As Majone
(1989, p. 7) writes:
The job of analysts consists in large part of producing evidence
and arguments to be used in the course of public debate. Its
crucial argumentative aspect is what distinguishes policy analysis
from academic social science on the one hand, and from problem-
solving methodologies such as operations research on the other.
The arguments that analysts produce may be more or less techni-
cal, more or less sophisticated, but they must persuade if they are
to be taken seriously in the forums of public deliberation.
The statement of task guiding this report did not direct the committee
to conduct a comprehensive investigation of what is being taught in policy
programs and schools. Deliberations of the committee, however, led to the
OCR for page 73
THE NEXT GENERATION OF RESEARCHERS AND PRACTITIONERS 73
firm belief that it is timely to examine policy education in the same spirit
that the famed Flexner Report (Flexner, 1910) examined medical education
a century ago and the Ford Foundation (Gordon and Howell, 1959) and
the Carnegie Corporation of New York (Pierson, 1959) examined business
education in the 1950s. The Flexner Report, commissioned in 1908, stands
out in this list; it is widely credited with initiating reforms that professional-
ized medical and health training appropriate for 20th century challenges,
and from which the nation continues to benefit.
An analogous effort directed to policy education could determine if
schools and programs are suitably aligned with the challenges that have
emerged over the past half-century: decolonization; democratization;
globalization; mass communication and the emergence of the Internet;
economic and technological development; the international diffusion
of science and technology; the rise of knowledge elites; and the growing
influence of the private sector in information production and knowledge
management, in addition to the host of specific competencies associated
with evidence-based policy, performance metrics, cost-benefit analysis, and
evaluation research. A Flexner-like effort could determine whether policy
schools are providing the knowledge and skills relevant to assuring that poli-
cies responding to these broad challenges are influenced by science.
In the absence of such a study, we turn to a research literature offering
partial though important insights into policy school objectives and the im-
plementation of those objectives. In addition, the committee conducted its
own cursory examination of the curricula of nearly 100 policy schools and
programs in the United States. We acknowledge that what is readily avail-
able allows only best-guess estimates about what is being taught every year
to the thousands of students enrolled in public policy courses. Although we
would prefer to have a Flexner-like exhaustive study at hand, our immediate
question can be adequately answered with what is available: how much of
what we endorse as a policy education curriculum is already in place?
We are confident that practically all public policy education includes
courses on the "politics of policy making." These courses draw on a large
political science literature that examines how political considerations affect
policy outcomes. There is also attention to the role of values, a topic appear-
ing in any number of topical courses on the assumption that value tradeoffs
appear in practically all policy choices. Examples include intergenerational
choices, such as abundant energy for current generations versus the risk of
sea-level rise that will inundate coastal communities of future generations;
allocating public funds between competing public goods, such as repairing
OCR for page 74
74 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY
roads versus lower student-teacher ratios; deciding who should pay for pol-
icy failures, whether the costs of the collapse in the housing market should
be borne by those who borrowed above their means or by those who pack-
aged the mortgages in ways that hid the risks. More generally, students are
taught that the complexity in policy making results not just from weighing
counterarguments about effectiveness and efficiency, but also from facing
questions about what is right, just, or fair.
If political and value considerations are being routinely taught, so are
methods. In these courses there is a decided emphasis on quantitative skills.
Morçöl and Ivanova (2010) document this, and categorize the quantitative
methods courses into three groups: (1) research design courses, in which
experimental and quasi-experimental designs are favored; (2) data collec-
tion methods, in which surveys are favored; and (3) analytic approaches,
in which regression analysis is favored. That is, it is clear that methods
associated with the "evidence-based policy" framework (see Chapter 3) are
strongly represented in policy education curriculum. This is to be expected,
and policy education should continue to emphasize the quantitative meth-
ods relevant to analyzing social conditions, designing responsive policy
interventions, and evaluating the consequences of interventions.
However, as detailed in Chapter 4, other competencies are needed to
navigate the policy world. These competencies include attention to the
properties of reasoning about scientific knowledge (Grozer, 2009) and
to understanding the assumptions underlying divergent policy framings,
expert judgments, consensus-building techniques, and analytic meth-
ods or approaches. This knowledge will help prepare students to cope
with the realistic, everyday problems encountered in applying existing
knowledge--with its gaps, imperfections, and disciplinary constraints--to
policy problems. Without such understanding, students may overestimate
the persuasive power of scientific reasoning, and overlook the substantial
barriers of institutional and cultural resistance to new research knowledge,
unfamiliar policy framings, or solutions that challenge deeply held moral
or ethical beliefs. Internships and case studies can help students learn about
these and other complexities of the policy-making process.
Because the case study method is widely used in policy education,
we reviewed a large number of case studies from the perspective of our
report. Consistent with the observation above, cases used in policy schools
routinely cover how political considerations influence policy outcomes and
value tradeoffs. They draw student attention to the distribution of benefits
and costs and how the "rules of the game" condition policy choices. They
OCR for page 75
THE NEXT GENERATION OF RESEARCHERS AND PRACTITIONERS 75
use to advantage a large number of key concepts and processes--from bu-
reaucratic inertia to unintended consequences, from negotiation strategies
to using the media.
What the examined cases rarely attend to is how scientific knowledge
is used in policy making. There is little discussion of the quality or quantity
of research available to the policy makers, even less discussion of whether
that research is used as evidence, and still less about why science is ignored.
Except incidentally, the cases do not explore the role of knowledge brokers
or whether the ideas of evidence-based policy come into play. The pro-
cesses and institutions through which policy makers gain access to relevant
knowledge, such as expert advisory committees, receive little notice. There
certainly is no attention to whether variation in cognitive biases of policy
makers or variation in cultures of decision making tell them what to expect
when science enters the policy argument. In summary, practically nothing
of what is emphasized in Chapter 4 as ways to better understand the use of
science is reflected in the case studies we examined.
An additional suggestive finding comes from Great Britain, where the
current government has established the Behavioral Insights Team, a small
office led by a social psychologist. Thaler (2012) describes how this office
used a randomized control trial to test behavioral theory on when people
conform to social norms. The issue was tax compliance; the treatment was
a letter to late payers stating that others in their community pay their taxes
promptly. There was a sharp increase in compliance in the treatment group,
and not in the control group, whose message made no mention of neigh-
bor's behavior. British tax authorities estimate that the reinforcing message
could generate extra annual revenue of £30 million ($46.5m) nationwide.
We cite this small study because the government (Thaler, 2012, p. 4) "is suf-
ficiently convinced of the value of these activities [of the Behavioral Insights
Team] that it announced last week that behavioral science is to be included
in the required curriculum for civil servants." Behavioral science had not
been taught in Britain's civil service training but now will be.
Though it would take a Flexner-style investigation to offer a thorough
account of what is today being taught to thousands of M.P.A. and M.P.P.
students in U.S. universities, our cursory review points to what is absent.
Our review found few courses that draw on social psychology and cognitive
science to provide public policy students with an understanding of human
decision-making processes--including biases, heuristics, and probabilistic
errors--as they pertain to reasoning about policy. Nor did we find many
courses in which an anthropological, sociological, or humanistic approach
OCR for page 76
76 USING SCIENCE AS EVIDENCE IN PUBLIC POLICY
to policy making is used to help students make sense of the interconnected-
ness of actors and institutions and the frameworks that shape policy choices.
Nor did we find policy education to be self-conscious about the issue one
might expect it to be most attentive to: what do students need to understand
about the use of scientific evidence in public policy?
The social sciences have the opportunity to influence the competen-
cies and perspectives that today's students in master's-level policy programs
carry with them into positions across the policy enterprise. We hope that
this report will spur self-examination across policy schools. One outcome
might be differentiation, with some programs providing ever more rigor-
ous training in methods and theories that strengthen research about "what
works" and other programs emphasizing rigorous training in methods and
theories that strengthen understanding of the conditions needed to put that
research to policy use. Such a division of labor would result in a broad array
of perspectives and skills available to think tanks, legislative staffs, policy
units in executive branches, and other settings in the policy enterprise--
from local government to international agencies, in both the public and
the private sector.
There is no better way to summarize this chapter than repeating a
truism--effective public policy is dependent on a steady supply of well-
prepared graduates prepared for public service and associated careers in the
policy enterprise. Our report advocates a broad definition of well prepared,
certainly to include technical competencies in evaluation research, program
design, measurement, and the like--but to include as well an understanding
of how science can be used to inform public policy.
A CONCLUDING THOUGHT
The committee writes this report mindful that the American public's
willingness to invest in science education and research is not unlimited,
and that the immediate times emphasize scrutiny of the investment. But
these times are also witness to a steadily growing policy enterprise--a broad
effort to make "better" policy through the application of science. We have
not taken a position on "better" policy, but have certainly taken a position
on the value of, to return to our title, Using Science as Evidence in Public
Policy. Moreover, we have written that it is within the competency of and is
therefore an obligation of the social sciences to advance our understanding
of "using science."
