Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 9
1
The Importance of Measuring
Productivity in Higher Education
This study has two major objectives: to present an analytically well-defined
concept of productivity in higher education and to recommend empirically valid
and operationally practical guidelines for measuring it. In addition to its obvi-
ous policy and research value, improved measures of productivity may generate
insights that potentially lead to enhanced departmental, institutional, or system
educational processes. In pursuit of these objectives, we address a series of ques-
tions: What is productivity and how can the concept of productivity be applied
to higher education? What limitations and complexities are confronted when
attempting to do so? Why is the measurement of productivity important to educa-
tion policy? Who should care about measuring productivity? And, how can the
measurement of productivity be improved?
These questions are not new. Indeed, 2010 marked the 100th anniversary of
the Carnegie Foundation Report (Cooke, 1910), which developed a time-use ac-
counting formula to estimate the costs and outputs of higher education for both
teaching and research. Essentially, the Carnegie Foundation Report sought "to
measure the efficiency and productivity of educational institutions in a manner
similar to that of industrial factories" (Barrow, 1990:67). One goal of this earlier
effort was to create a method for measuring productivity so that higher education
would be subject to and benefit from competitive market pressures akin to those
in private industry. To accomplish this, the Carnegie Foundation Report created
a key unit of measure called the student hour, defined as "one hour of lectures,
of lab work, or recitation room work, for a single pupil" (Barrow, 1990:70). The
motivation behind the initiative was to facilitate calculation of relative faculty
workloads, the cost of instruction per student hour, and, ultimately, the rate of
educational efficiency for individual professors, fields, departments, and univer-
sities (Shedd, 2003). These are the essentially the same things we want to know
9
OCR for page 10
10 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
today and which this report again addresses. Additionally, the difficult measure-
ment issues limiting completeness of the analysis 100 years ago are still very
much in play, as we detail in Chapter 3.
While productivity measurement in many service sectors is fraught with
conceptual and data difficulties, nowhere are the challenges--such as accounting
for input differences, wide quality variation of outputs, and opaque or regulated
pricing--more imposing than for higher education. Compounding the challenge
is that many members of the panel (and many reading the report) are being asked
to consider the same measurement tools to analyze their own industry as they
would use in analyzing any other. And, from up close, the complexities are much
more apparent than when dissecting productivity from a distance.
One lesson drawn from this effort is that we may be too sanguine about the
accuracy or relevance of measures of productivity in other sectors, having seen
how daunting they can be in a setting with which we are more intimately familiar.
The conceptual and practical problems surrounding this effort raise additional
concerns because it is known that measurements create incentives, incentives
change practices, and those practices have the potential to affect people and in-
stitutions we care deeply about. Yet the current higher education environment is
not without incentives, many of which have flaws that are at least as profound and
distorting as those associated with economic measurement, and are sometimes
much worse. Readers of the report will have to make the up their minds whether
the potential disadvantages of this approach, as well as the costs of implementing
the specific recommendations, are worth the potential benefit. While we under-
stand how some might come to a different conclusion, we believe the advantages
outweigh the disadvantages.
1.1. SOCIAL AND POLICY CONTEXT
Not everything that counts can be counted, and not everything that can be
counted counts.--William Bruce Cameron
While this observation is broadly profound, it seems exceptionally applicable
to the case of higher education. At the same time, a better understanding of the
workings and nature of the sector is necessary, given its prominent role in the
economy and impact on the future of our society. Higher education is part of
the essential fabric of American experience, one in which many citizens spend a
significant fraction of their adult lives. For many individuals, higher education is
the largest or second-largest consumer decision.
On an aggregate level, colleges and universities employ around 3.6 million
individuals, 2.6 million of those in professional positions.1 The sector accounts
1From Bureau of Labor Statistics, see http://www.bls.gov/spotlight/2010/college/ [June 2012]. This
source also includes data on teacher salaries by field, earnings by graduates, etc.
OCR for page 11
THE IMPORTANCE OF MEASURING PRODUCTIVITY IN HIGHER EDUCATION 11
(directly) for about 3.3 percent of gross domestic product (Soete, Guy, and Praest
Knudsen, 2009), which makes it larger than a number of industries for which
productivity data are routinely collected. It also accounts for about 10 percent of
state budgets in recent fiscal years (National Association of State Budget Officers
State Expenditure Report, 2011).
Beyond the production of credentialed citizens, academic institutions also
perform much of the nation's research and development. In 2008, colleges and
universities spent $52 billion on research and development, with 60 percent of this
funding derived from the federal government. Academic institutions performed
55 percent of basic research and 31 percent of total research (basic plus applied)
in the United States (National Science Board, 2010:5-4). Although nonacademic
organizations conduct research in select functional fields such as health, defense,
space, energy, and agriculture, the general prominence of academic research and
the established funding patterns reflect a postWorld War II political consensus
that federally funded basic research is most effectively performed in academic
institutions. This contrasts with patterns observed elsewhere in the world, where
there is greater reliance on government-operated laboratories, other forms of
public research organizations, or industry to conduct research.
In the current global economic and fiscal climate, the attention being paid
by policy makers to the competitiveness and general state of higher education in
the United States continues to heighten. Recent research (e.g., Carnevale, Smith,
and Strohl, 2010) indicates that the economy's expanding sectors and industries
rely disproportionately on workers with higher education credentials. During the
current recession, characterized by high and persistent unemployment, analyses
of evidence such as online job postings and real-time jobs data reveal a mismatch
between job openings and the educational credentials of the workforce. Higher
education institutions themselves have become increasingly concerned about
improving their own performance, competing with peer institutions on cost and
quality, and providing a degree of public accountability.
In this environment of strong policy maker and institutional interest in the
performance of higher education, stakeholders have used whatever data and mea-
sures are available in an attempt to understand trends and perceived problems; for
better or worse, some version of productivity will be measured. Therefore, it is
crucial to develop coherent measurement tools that make the best possible use of
available and potentially available data. Failure to do so will keep the door open
for an ever-expanding profusion of measures, many of them unnecessarily distor-
tive, and endless debates about measurement as opposed to productivity itself.
Currently in policy debates, administration discussions, and media coverage,
attention tends to focus on the soaring sticker price of college (overall costs have
remained more or less in line with general inflation). Cost per degree, graduation
rates, and retention metrics have been used as though they measured efficiency
or overall productivity. What is often ignored in these discussions is the quality
of higher education instruction. When attention is overwhelmingly focused on
OCR for page 12
12 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
completions or similar metrics, the risk is heightened that the stated goal will
be pursued at the expense of quality.2 If the aim is to know whether increased
spending is resulting in commensurate returns, the quantity and quality of the
sector's inputs and outputs must be reliably tracked, which, for the latter, requires
developing assessment tools for quantifying the outcomes of higher education.
Used without a solid understanding of their meaning in divergent contexts,
simple metrics such as graduation rates and costs per degree can distort and
confuse as much as they inform. In the absence of more rigorous alternatives,
however, they will continue to be used--and, at times, misused. In this report,
we take a closer look at some of the unidimensional performance metrics to
understand better what exactly they reveal. We then develop a more appropriate
approach to productivity measurement--one that can serve as a key component
in the set of information from which to base resource and other policy decisions.
However, even the productivity measure developed in this report, which expresses
outputs in terms of quantities of credits and degrees, cannot explicitly take ac-
count of quality variation and change. As detailed in Chapter 4, an effect will be
captured by the proposed measure to the extent that higher quality inputs, such
as better teachers, lead to higher percentages of students completing degrees; but
this effect is indirect. Thus, a single metric--even a well-conceived productivity
measure--will rarely be sufficient, on its own, to adequately serve as a compre-
hensive assessment of institutional, system, or even sector-wide performance.
Other factors--most notably the quality dimension--must be monitored through
parallel tracking of information that will often have to be processed indepen-
dently from the productivity metric.
Finally, there are aspects of human and, more narrowly, productive enterprise
that create social value but that statisical measures do not and indeed do not pre-
sume to capture. From a societal perspective, investment in citizens' work careers
is not the only motivation for supporting and participating in higher education.
Nonpecuniary components of the sector's output assoicated with instruction, re-
search, and other public goods are also important. Like a policeman who brings
extraordinary passion to protection of fellow-citizens, a technology entrepreneur
whose vision ultimately changes the way people live, or an artist who is appreci-
ated long after creating the art, the passion and dynamism of a master teacher who
is truly interested in a student who, in turn, is truly interested in learning cannot
be richly portrayed in a number. In this context, some very real elements of the
value of experiencing life-changing learning cannot be fully quantified within a
(still very important) statistical infrastructure.
2Similar tendencies to focus on the easily quantifiable hamper discussions of medical care. The
increase in costs is known; the value gained from these expenditures, in terms of health benefits to
the population, frequently is not.
OCR for page 13
THE IMPORTANCE OF MEASURING PRODUCTIVITY IN HIGHER EDUCATION 13
1.2. CHARGE TO THE PANEL
The statement of task for this project--co-developed by the Lumina Founda-
tion for Education and the National Research Council's Committee on National
Statistics at a planning meeting held February 20, 2009--reads as follows:
The Panel on Improving the Measurement of Productivity in Higher Education
will develop a conceptual framework for measuring higher education productiv-
ity and describe the data needs for that framework. The framework will address
productivity at different levels of aggregation, including the institution, system,
and sector levels.
An overarching goal of the study is to catalogue the complexities of measuring
productivity and monitoring accountability in higher education. In particular, the
study will take into account the great variety of types and missions of higher
education institutions in the United States, ranging from open admission col-
leges to major research universities that compete on an international scale. The
study will also address the necessity to consider quality issues when attempting
to measure productivity. Since the quality of inputs to and outputs from higher
education varies greatly across institution types and, indeed, within them, the
study will highlight the pitfalls of using simplistic metrics based on insufficient
data for evaluating the performance of higher education.
One objective of the study will be to provide guidance to institutions and policy
makers about practical measures that can be developed for the purposes of insti
tutional improvement and accountability. However, to the extent that the differ-
ences in inputs, outputs, and institution types within higher education (along
with inadequate data) make the development of comprehensive productivity
measures impossible, the panel will assess the strengths and weaknesses of the
various alternatives in providing evidence on different aspects of the input-
output relationship.
At the conclusion of its study, the panel will issue a report with findings and rec-
ommendations for developing the conceptual framework and data infrastructure
and that provides an assessment of the strengths and limitations of alternative
approaches to productivity measurement in higher education. The report will
be written for a broad audience that includes national and state policy makers,
system and institution administrators, and higher education faculty.
An important aspect of this report is to highlight the complexity of measur-
ing productivity in higher education. A deeper understanding of this complexity
reduces the chances that decision makers will misuse measures--for example,
by incentivizing "diploma mills" through overemphasis of graduation rate or
time-to-degree statistics in accountability policies. While attempting to provide
novel insights into productivity measurement, we are cognizant that it is easy to
find fault with approaches that policy makers, administrators, and other practitio-
ners have relied upon to do their jobs. It is much more difficult to envision and
OCR for page 14
14 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
implement new methods that could become broadly endorsed. Recognizing that
funding and personnel decisions, as well as plans to improve resource allocation
are sometimes based at least in part on these measures, our intent is to encourage
those attempting to improve and apply them in real policy settings.
Due to the sheer breadth of activities associated with higher education in the
United States, this report cannot be exhaustive. The scope of the study and the
recommendations herein reflect policy information needs as well as feasibility-
of-measurement constraints. The report's purview includes all types of higher
education institutions (public, private, for-profit), but not all missions. Our mea-
surement prescriptions focus on instruction, which includes all taught programs,
regardless of level (e.g., associate, bachelors, taught terminal masters). 3 Joint
production of instruction, research, and public service is discussed in detail,
though it is recognized that measurement of the latter two is largely beyond the
scope of the panel's charge. Other missions, such as health care and athletics,
which sometimes are budgeted separately, are also excluded from our measure-
ment proposals, which mean that any synergies that exist between these activities
and conventional resident instruction programs are missed. To include them at
this point in the development of productivity measurement for the sector would
hopelessly complicate the task.
In developing a model of productivity (Chapter 4), the panel recognizes
that this is only a starting point for what promises to be a long-term research
agenda. It is worth pointing out that no industry is without its complexities, and
no productivity measure currently in use is permanently fixed. The extensive
and impressive research by the Bureau of Labor Statistics (BLS) into the con-
cepts and techniques of productivity measurement is indicative of the ongoing
process and continuing progress but also of the fact that measurement and con-
ceptual barriers remain.4 Additionally, as described in the next chapter, more
than one paradigm exists for constructing productivity models. 5 It is especially
worth distinguishing between aggregate models of the kind developed here,
which are designed to measure long-term trends, and structural models aimed
more specifically at operational improvement and accountability concerns. Ag-
gregate and sector-level productivity models have proved to be important for
economic and policy analysis. In higher education, for example, they reveal
whether resource usage per unit of output in particular institutional segments
3Application of the model developed in Chapter 4 uses IPEDS data that do not exclude Ph.D. and
research degrees (though they clearly have a quite different teaching production function). Due to
the way universities categorize instructional expenses, it is not possible to subdivide these activities
on the input side and, therefore, these degrees are not excluded from the output side either (they are
also included in the student credit-hour production figures). However, it is doubtful that the small
number of degrees and enrollments involved will have much effect on the actual productivity statistics.
4For details of the BLS work, see http://www.bls.gov/lpc/lprarch.htm#Concepts_and_Techniques_
of_Productivity [June 2012].
5OECD (2001) provides a thorough overview of aggregate and industry-level productivity measures.
OCR for page 15
THE IMPORTANCE OF MEASURING PRODUCTIVITY IN HIGHER EDUCATION 15
has been increasing or declining. The model may not reveal why this is so,
but at the very least it pushes us to ask additional, useful questions. However,
these kinds of models are not typically intended to be used for accountability
or incentivizing purposes--especially for applications such as higher education
where output prices do not necessarily reflect quality. In contrast, the structural
models involve a fairly detailed representation of an entity's internal structure,
and thus require more granular data. Such models also generally focus on mar-
ginal revenues and marginal costs, as opposed to the average revenues and costs
considered in the aggregate models. As noted above, the panel was not charged
with developing a structural model and has not attempted to do so.
At a conceptual level, this report dedicates considerable attention to pro-
ductivity measurement at different levels of aggregation, including the institu-
tion, system, and sector levels. For most purposes, it is necessary to segment
the sector by institution type to avoid inappropriate comparisons. However, the
measure developed in Chapter 4 is focused on productivity of the sort typically
applied to aggregate economic sectors (e.g., autos, steel, higher education), which
rests on the methodology used by the BLS. While one can imagine aggregating
institution-level data to produce a macro productivity measure, such an approach
is not practical at the present time for the higher education sector. As a technical
matter, there is nothing to prevent the model developed here from being applied
at the level of a state, system, or individual institution, but this opens the way for
it to be exploited for performance measurement without the proper support of
additional quality measures. The panel generally believes that this risk associated
with pushing forward with productivity measurment is worth taking, and that to
maintain the "know-nothing" status quo would perpetuate dysfunctional behavior.
It is noteworthy that the panel was not charged with recommending processes
to improve productivity, for example, through innovative new methods for de-
signing courses or through online education. Similarly, the panel was not asked
to develop models for monitoring departmental, institutional, or system activity;
these are applications. One stumbling block to productivity measurement--and
indeed, to productivity improvement--has been the widely-held view that, be-
cause learning is a service and its production is labor-intensive, colleges and uni-
versities suffer from a condition known as Baumol's cost disease. The underlying
theory, which breaks from the notion in classical economics that wage changes
are closely tied to labor productivity changes, is that labor costs in some sectors
of the economy are affected by productivity gains in other unrelated sectors.
Those productivity gains drive an increase in wages across the entire economy.
Sectors without productivity gains are nonetheless faced with a higher wage bill,
making them appear less efficient.6 Archibald and Feldman (2011) subscribe to
6In their landmark book, Performing Arts: The Economic Dilemma, Baumol and Bowen (1966)
use as an example a Mozart string quintet composed in 1787. More than two centuries later, it still
requires five musicians and the same amount of time to perform the piece.
OCR for page 16
16 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
this view, noting that the production processes for colleges and universities rely
on human interaction (at least traditionally), nearly fixed amounts of time inputs
from faculty and students, and a key role for highly educated, highly compensated
employees.
Even when steps can be taken to increase throughput, questions rightfully
arise about the effect of the changes on quality. Archibald and Feldman write
(2011:40):
An institution can increase class size to raise measured output (students taught
per faculty per year) or it can use an increasing number of less expensive adjunct
teachers to deliver the service, but these examples of productivity gain are likely
to be perceived as decreases in quality, both in the quality rankings and in the
minds of the students.
However, the evidence on the potential of higher education to benefit from
new models of production, such as online courses, is not conclusive. Harris and
Goldrick-Rab (2011) argue that "researchers and institutions themselves have
rarely paid much attention to whether policies and practices are cost-effective.
How would you know whether you're spending money effectively if you've never
even asked the question?" They conclude that colleges "can conceivably become
more productive by leveraging technology, reallocating resources, and searching
for cost-effective policies that promote student success." Indeed, many industries
that formerly were believed to be stagnant have been able to improve productivity
dramatically. Even in the quintessential example of Baumol's cost disease (noted
above), string quartets have improved "productivity" dramatically through the
capability to simulcast a performance to theaters or, more obviously, by record-
ing their music and earning money while the musicians sleep (Massy, 2010:39).
Other examples can be found in medical care, legal services, and elsewhere.
Work by the National Center for Academic Transformation (NCAT) on
course redesign provides a contemporary example of what can be accomplished
in this area (see Chapter 2 for a description of some of this work; see also Ap-
pendix B on NCAT's methods). The organization's clients analyze measures to
determine new ways to combine inputs so as to produce student credit hours of
the same or better quality than with traditional methods. Quality assurance also
enters the process. Indeed, the changes that have been made following such analy-
ses are the classic ones used in essentially all industries: shifts from high-cost to
lower-cost labor, more intensive use of and better technology, and elimination of
waste in resource utilization.
The idea that instructional productivity may potentially be increased by alter-
ing the way inputs in the production function are combined highlights why im-
proved measurement is so important. Potential improvement in productivity also
justifies requirements that colleges and universities systematize collection of data
OCR for page 17
THE IMPORTANCE OF MEASURING PRODUCTIVITY IN HIGHER EDUCATION 17
on expenditures and the volume and quality of inputs and outputs. Routine gen-
eration and collection of such data is a prerequisite for wider efforts to improve
productivity and enable external stakeholders to hold institutions accountable.
1.3. AUDIENCE AND REPORT STRUCTURE
In the face of the observations laid out above, we take the following premises
as the starting point for our assertion that improved information regarding the
functioning of higher education is needed: (1) Those who fund higher education
have a legitimate interest in meaningfully measuring productivity, both in order
to make the best possible allocations and spending decisions within the sector,
and to assess the value of higher education against other compelling demands on
scarce resources; (2) Institutions, individuals, and communities whose economic
well-being is most directly at stake when funding decisions are made have a le-
gitimate interest in ensuring that measurements of productivity are accurate and
appropriate. The analysis and recommendations in this study attempt to balance
these interests.
This report has been written for a broad audience including national and state
policy makers, system and institution administrators, higher education faculty,
and the general public.
· State and federal legislators: Policy makers benefit from discussion
that identifies important questions, explains the need for particular data
programs, and clarifies the meaning of different performance metrics.
· College and university administrators: These decision makers are under
increasing pressure to address accountability and productivity concerns.
This report may provide authoritative backing to resist pressure to im-
pose inadequate assessment systems just so as to be seen to be doing
something. These groups may also benefit from guidance about what
data to collect to support proposed evaluations of programs.
· Faculty: College and university professors need to understand the inter-
action between their own interests and society's interests in the educa-
tion enterprise. They need to be informed about innovative approaches
to increasing mission efficiency through use of technology and other
means. And they need quality information to guide them in the context
of shared governance that prevails in most colleges and universities.
· General public: We hope that this report will promote a greater under-
standing of societal interests in higher education and of how the inter-
ests of stakeholders (students, faculty, administrators, trustees, parents,
taxpayers) fit into that broader picture. The arguments herein may also
promote a fuller understanding of the complexity of colleges and uni-
versities and how they benefit the economy and society.
OCR for page 18
18 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
The remainder of the report is organized as follows: In Chapter 2, we define
productivity and then characterize the activities of higher education in terms of
inputs or outputs. We pay particular attention to the heterogeneity of the sector,
including the great range of its products and the changes and variation in the
quality of its inputs and outputs. Accounting for all outputs of higher education
is particularly daunting, as they range from research findings and production of
credentialed citizens to community services and entertainment. Although the
panel's recommendations focus on degree output, research and other scholarly
and creative activities must be acknowledged because they are part of the joint
product generated by universities, and because they may affect the quality and
quantity of teaching. We also contrast productivity with other measurements that
have been used as proxies for it and discuss the merits and limitations of proxies
currently in use.
In Chapter 3, we articulate why measurement of higher education productiv-
ity is uniquely difficult. Colleges and universities produce a variety of services si-
multaneously. Additionally, the inputs and outputs of higher education production
processes are heterogeneous, mix market prices and intangibles, vary in quality,
and change over time. Measurement is further impeded by conceptual uncertain-
ties and data gaps. While none of these difficulties is unique to higher education,
their severity and number may be. We detail the complexities--not to argue that
productivity measurement in higher education is impossible, but rather to indicate
the problems that must be overcome or mitigated to make accurate measurements.
This report will be instructive to the extent that it charts a way forward for
productivity measurement. Toward this end, in Chapter 4, we provide a prototype
productivity measure intended to advance the conceptual framework. The objec-
tive here is not to claim a fully specified, ideal measure of productivity, for such
does not exist. Rather, we aim to provide a starting point to which wrinkles and
qualifications can be added to reflect the complexity of the task, and to suggest a
set of factors for analysts and policy makers to consider when using productivity
measures or other metrics to inform policy.
In Chapter 5, we offer practical recommendations designed to advance mea-
surement tools and the dialogue surrounding their use. We provide guidance
for developing the basic productivity measure proposed in Chapter 4, targeting
specific recommendations for the measurement of inputs and outputs of higher
education, and discuss how changes in the quality of the range of variables could
be better detected. A major requirement for improved measurement is better data.
Thus, identifying data needs demanded by the conceptual framework, with due
attention to what is practical, is a key part of the panel's charge. This is addressed
in Chapter 6. In some cases, the most useful measures would require data that do
not now exist but that could feasibly be collected.
