Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 107
6
Implementation and Data Recommendations
6.1. GENERAL STRATEGIES
This report contains two types of recommendation for improving productiv-
ity measurement: those that are conceptual and those that address issues related
to implementation of new measures and data development. The model presented
in Chapter 4 requires specific data inputs and, while considerable progress can
be made using existing sources such as the Integrated Postsecondary Education
Data System (IPEDS) of the National Center for Education Statistics (NCES),
an ideal productivity measure will require new or improved capabilities as well.
In moving forward with plans to implement productivity measurement, pro-
gram administrators will not be able to do everything suggested by this panel (and
by others1) all at once. It is helpful, initially, to simply take stock of information
that is available on input and output trends at various units of analysis, and then
consider how far--with that data--one can get in constructing measures. This
type of demonstration was a major motivation for working through the model
presented in Chapter 4 using real data examples.
More generally, both with and beyond our model, we want to know how
well--with current data and approaches--we can address questions of policy
interest that arise:
· If measurable outputs have increased while resources have been stable
or declining, has quality suffered?
1See recommendations by the National Postsecondary Education Cooperative (2010).
107
OCR for page 108
108 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
· If outputs have declined while resources have increased or remained
stable, has quality changed correspondingly?
· How do productivity trends in comparable states, institutions, or depart-
ments compare?
· Have changes in education delivery mode, student characteristics or
important contextual variables (economic, demographic, political, insti-
tutional) had a measurable bearing on the trends?
· Are there clear indicators (spikes, dives, or other anomalies) that sug-
gest data problems to be cleaned (as opposed to sudden changes in
performance)?
· What evidence or further research could be brought to bear to answer
these questions in the next step of the conversation?
The more accustomed administrators, researchers, and policy makers become to
conversations that incorporate these kinds of questions, the better the selection of
metrics and the potential to understand them are likely to become, and the more
evident the need for high-quality data.
A general strategy of implementing improved metrics for monitoring higher
education begins with the following assertions:
· Productivity should be a central concern of higher education.
· Policy discussions of higher education performance will lack coherence
in the absence of a well-vetted and agreed upon set of metrics.
· Quality of inputs and outputs--and particularly changes in them--
should always be a core part of productivity measurement discussions,
even when it is not fully or explicitly captured by the metrics. Alterna-
tively put, productivity measurement is not meaningful without at least
a parallel assessment of quality trends and, at best, a quality dimension
built directly into the metric.
· Some elements relevant to measuring productivity are difficult to quan-
tify. This should not be used as an excuse to ignore these elements or to
avoid discussions of productivity. In other words, current methodologi-
cal defects and data shortcomings should not stunt the discussion.
Additionally, in devising measures to evaluate performance and guide resource al-
location decisions, it is important to anticipate and limit opportunities to manipu-
late systems. The potential for gaming is one reason many common performance
metrics should not be relied on, at least not exclusively, in funding and other
decision-making processes. Simply rewarding throughput can create distortions
and counterproductive strategies in admissions policies, grading policies, or
standards of rigor. Clearly, a single high-stakes measure is a flawed approach;
a range of measures will almost always be preferable for weighing overall per-
formance. We note, however, that monitoring productivity trends would not be
OCR for page 109
IMPLEMENTATION AND DATA RECOMMENDATIONS 109
introducing incentives to a world without them. Among the major incentives now
in place are to enroll students, acquire research grants, improve standing in rank-
ings, raise money, and win athletic competitions. The recommendations in this
report are merely adding another incentive (and one more worthy than a number
already in play) that will help round out the current set.
6.2. RECOMMENDATIONS FOR IMPROVING
THE DATA INFRASTRUCTURE
A major element of the prescription for improving productivity measures for
higher education involves developing more accurate and new kinds of informa-
tion. Thus, identifying data needs is a key part of the panel's charge. Much has
been implied in this regard throughout the report and some specific data needs
were highlighted in Chapter 4. Here, we reemphasize these in the form of ad-
ditional recommendations. In thinking about new approaches, the panel had an
advantage over practitioners and administrators in being able to think in terms of
future goals by recommending changes to IPEDS, coordination of other existing
data sources, or development of new data approaches altogether.
6.2.1. Data Demanded by the Conceptual Framework
The categories of data demanded by the modeling framework are, broadly:
· Output/Benefit information. Basic institutional data on credits and de-
grees can be enhanced through linkages to longitudinal student data-
bases. In addition to their role in sharpening graduation rate statistics,
longitudinal student surveys are needed to more accurately estimate
degree costs, degree earnings value, and input/output quality.2
· Input/Cost information. Sources include institution and state-based ex-
penditure accounting data; basic information about faculty time alloca-
tions3; and student unit records.
Ideally, for higher education, underlying data for productivity measurement would
be as detailed and accurate as it is for the best-measured sectors of the economy.
Information requirements--including the need for data at more granular levels,
2Data Quality Campaign, 10 Essential Elements of a State Longitudinal Data System, see http://
www.dataqualitycampaign.org/survey/elements [January 2012].
3IPEDS provides some data on teaching loads; the Delta Cost Project (2009) includes some data on
staffing. One could push for time use surveys of faculty time allocations, including hourly accounts
of research activity, instruction, and service. This would be difficult though as faculty do not bill
by the minute or hour, and much research is done off-campus; furthermore, how would conceptual
breakthroughs or mental crafting of language be accounted for when they occur during the course of
another activity, such as teaching or hiking?
OCR for page 110
110 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
and input and output quality indicators--extend beyond what is currently avail-
able in IPEDS and other sources, though these provide an excellent start.
Another logical source of information to support productivity measurement
is the quinquennial economic census surveys, a reliable source of expenditure and
other data--including total labor costs and total hours--for most sectors of the
economy, including many nonprofits. Statistics for other service industries have
improved a great deal in recent years, in part as a result of periodic enhancements
to the economic census. However, Bosworth (2005:69) notes that "the higher
education community successfully lobbied to be exempted from these reporting
requirements; thus the Census Bureau is blocked from gathering data and we lack
even the most basic information about the education industry. For example, we
do not know with any degree of detail who is employed in higher education, how
much capital is being spent, or how many computers are being used."
The higher education sector has not been covered in the economic census
since 1977, when it was introduced there (it only appeared once). This decision
to omit this sector should be revisited, specifically to evaluate the costs of re-
introducing it to the census and the benefits in terms of value added to existing
data sources.
Recommendation (12): Every effort should be made to include col-
leges and universities in the economic census, with due regard for the
adequacy of alternative data sources and for the overall value and costs
added, as well as difficulties in implementation.
The Department of Education could require that institutions file the census forms
in order to maintain eligibility to participate in Title IV programs.
For purposes of constructing the National Income and Product Accounts
(NIPAs) and productivity measures, BEA and BLS may also benefit from such
a reversal of policy. Census data would provide details on employment by broad
categories (it is now difficult to find statistics on employment in higher education
industry), along with a range of other operational and performance information.
It would facilitate construction of value-added statistics for most of the inputs
(including capital goods) that are useful for broad measures of productivity. Ad-
ditionally, participation in the economic census would harmonize data reporting
formats used in other industries. An alternative to this recommendation would be
to take advantage of the established IPEDS framework, which includes institution
identifiers, and then import economic census style questions into it.
6.2.2. Envisioning the Next Generation IPEDS
IPEDS provides valuable information annually (or biannually) on institu-
tional characteristics, degree completions, 12-month enrollment, human resources
(employees by assigned position, fall semester staff, salaries), fall semester
OCR for page 111
IMPLEMENTATION AND DATA RECOMMENDATIONS 111
enroll ment, finance, financial aid, and graduation rates for all private and public,
nonprofit and for-profit postsecondary institutions that are open to the general
public and that provide academic, vocational, or continuing education for credit.4
Even so, fully specified productivity measurement of the type envisioned by
the panel requires more complete and different kinds of information than what
is currently available in IPEDS. For some purposes, greater data disaggrega-
tion, such as at departmental levels, and quality dimensions are needed. More
comprehensive longitudinal student databases are essential for calculating better
tailored and more clearly defined graduation rates and for estimating full degree
costs and values.
Box 6.1 summarizes IPEDS data that may be useful in the measurement of
higher education productivity, enumerating its significant advantages and remain-
ing challenges for its improvement.
For the model proposed in Chapter 4, institutional data submission require-
ments are not exceedingly onerous. As noted above, many of the needed vari-
ables--credit hours, enrollments, degrees--are already reported to IPEDS. The
most significant change required to fully implement output differentiation is to
link credit hours to degree or field.
Recommendation (13): Institutions should collect credit-hour data that
track both individual students and the departments of record. The nec-
essary information exists in most institutions' student registration files.
IPEDS should report these data along with the numbers of degrees
awarded.
Chapter 4 provides details about how exactly the credit-hour data should be
structured and the statistics extracted.5
A side benefit of following students is that it creates new opportunities for
calculating more sophisticated graduation rates as well. Cohort-based statistics
such as those produced through IPEDS Graduation Rate Survey typically re-
strict the denominator to first-time, full-time students. For institutions that enroll
large numbers of part-time students and beginning transfers, this will not yield
a meaningful number.6 Including these students in the cohort allows for more
4See http://nces.ed.gov/ipeds/ [July 2012].
5In applying this approach to, among other things, discover the course components that go into
producing a certain kind of degree, credit hours could be calculated by degree program and institu-
tion one time and the results applied in subsequent cohorts for that institution. The calculation may
not have to be done every year, though that might not be such an onerous task once systematized.
"One-time" benchmarks that are refreshed periodically may be adequate.
6NPEC (2010) offers recommendations for (1) counting and defining the composition of an initial
cohort of students; (2) counting and defining who is a completer; (3) understanding the length or time
of completion; and (4) incorporating students who transfer out of an institution. Available: http://nces.
ed.gov/pubs2010/2010832.pdf [July 2012]. The panel agrees with most of these recommendations.
OCR for page 112
112 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
BOX 6.1
Integrated Postsecondary Education Data System (IPEDS)
IPEDS describes itself as "the primary source for data on colleges, uni-
versities, and technical and vocational postsecondary institutions in the United
States." Elements of IPEDS will be essential to most attempts to measure pro-
ductivity in a consistent manner across state lines, within states that do not have
their own consistent statewide reporting processes for all public institutions, and
across both public and private institutions. IPEDS collects detailed descriptive data
from all public and private institutions in the United States that wish to be eligible
for federal student financial aid, which almost all do.
Productivity-Related Content
Completions
Degrees and certificates awarded are the only elements of the IPEDS sys-
tem that are broken down by discipline. They are reported for every institution, with
field of study and degree level (e.g., associate, bachelor, > one-year certificate,
etc.), first/second major, and student demographic characteristics.
Finances
Expenditures are reported by the purpose of the expense (e.g., instruc-
tion, research, public service, student services, academic support, etc.) and by
the type of expense (i.e., salaries, benefits, plant operations, depreciation, and
other). For measurement of instructional productivity using IPEDS data, the direct
instructional and student services expenses, plus a prorated share of academic
support, institutional support, and plant operations constitute the relevant portion
of the total expenditure.
Enrollment
Student credit hours are a core measure for productivity analysis. IPEDS
collects only aggregate data on undergraduate and graduate enrollments for each
institution. Definitions provided to reporting institutions allow for translation among
different calendar systems and for calculation of full-time equivalent enrollments
based on numbers of semester/quarter/clock or other units.
Institutional Characteristics
This file contains elements that could be used to group like institutions for
productivity analysis. Relevant groups include Carnegie classification, public or
private control, Historically Black College and Universities, geographical details,
etc.
Human Resources
For degree-granting institutions, the number of employees and their com-
pensation are reported by faculty and tenure status, and equal employment op-
portunity (EEO) category:
OCR for page 113
IMPLEMENTATION AND DATA RECOMMENDATIONS 113
Staff whose primary responsibility is instruction, research, and/or public
service:
· xecutive/administrative/managerial
E
· Other professionals (support/service)
· Technical and paraprofessionals
· Clerical and secretarial
· Skilled crafts
· Service/maintenance
Advantages
· I
PEDS is the primary national data source for cross-sector and cross-state
comparisons.
· Terms are defined independently of state or system-level categories.
· Completions data are reported at a high level of detail.
· IPEDS does not change quickly. Most data fields have been consistently de-
fined over many years, allowing for reliable analysis of trends.
Challenges
· C
redit-hour enrollments, staffing patterns, and finance data are available as
aggregate numbers only, with no discipline-level detail.
· Incoming student data are limited. Other than a few average data points in
the Institutional Characteristics file, IPEDS does not have information about
students' level of academic preparation (including remediation or advanced
placement), credits transferred in, socioeconomic background, or academic
objectives at time of entry.
· IPEDS does not change quickly. Changes typically require strong consensus
among institutions, Congressional action, or both, and take many years to
implement fully.
· Institutional interpretations of human resources categories, and to a lesser
extent finance and degree discipline classifications, vary widely in practice.
Required human resources files are designed primarily for equal opportunity
reporting and auditing, rather than productivity analysis.
· For cross-institution comparisons of costs and outcomes, researchers need
data on discipline mix. Many state systems already identify enrollment by
major in their data sources. Classification structures in IPEDS are complicated
by varying levels of interdisciplinarity among institutions, different types of
academic organization, evolution of categories over time, and external factors
such as financial incentives to encourage students to major in STEM fields or
to award more degrees in those areas. Weights assigned to disciplines for the
purpose of assessing productivity would also risk creating external pressure to
adjust discipline classification. Data on distribution of degrees granted across
majors has been shown to be an important predictor of six-year graduation
rates in many educational production function studies. Given this coverage, it
is not clear that collecting data on departmental level progression of students
makes sense, especially since there is so much movement of students across
fields of study during their period of college enrollment.
OCR for page 114
114 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
completeness; however it can also create new problems because part-time stu-
dents have differing credit loads and transfer students bring in varying numbers
of previously earned credits. This renders fair comparisons difficult because, un-
like the first-time full-time population, not all subpopulations are starting from
the same baseline.
On the labor input side of the production function, it would be helpful to have
data that clarifies more detailed categories for employees. IPEDS does not distrib-
ute labor hours into all categories; it does include the proportion of FTE allocated
to instructional time. A complete dataset underlying productivity measurement
would identify major job categories and how time is allocated on average at a
given level of aggregation. In Chapter 5, we recommended that institutions be
charged with collecting data on employees by personnel category, making time
allocation approximations--focusing on instruction, research, and service--and
reporting the results in a revised IPEDS submission that would be subject to the
same kind of audit used by other agencies in data collection.
A number of international data efforts are already heading in these new
directions, developing microdata and quantitative indicators for higher educa-
tion institutions, including time in research and other activities. The European
Commission, for example, appointed the EUMIDA Consortium to "explore the
feasibility of building a consistent and transparent European statistical infrastruc-
ture at the level of individual higher education institutions" (Bonaccorsi, Daraio,
and Simar, 2006). The goal of the project is to "provide institutions and policy
makers with relevant information for the benchmarking and monitoring of trends
for modernisation in higher education institutions. . . . [M]ost European countries
collect data on their universities, either as part of R&D and higher education
statistics or as part of budgeting/auditing systems."7 U.S. educational institutions
and the Department of Education may benefit from assessing these efforts.
6.2.3. Administrative Data Sources
Beyond IPEDS, a number of existing administrative databases can be tapped
in constructing useful performance measures. The potential of the kinds of ad-
ministrative data sources described below depends heavily on the ability of re-
searchers and policy analysts to link records across state boundaries and across
elementary, secondary, postsecondary, and workforce boundaries (Prescott and
Ewell, 2009). This, in turn, depends upon the presence of secure unique identi-
fiers, such as social security numbers, in all the databases of interest. At present,
use of such identifiers is limited (or, more importantly, is perceived to be limited)
by the provisions of Family Educational Rights and Privacy Act (FERPA) regu-
lations. Clarification or re-regulation of FERPA might considerably enhance the
usefulness of these administrative databases.
7See http://isi.fraunhofer.de/isi-de/p/projekte/us_eumida.php [July 2012].
OCR for page 115
IMPLEMENTATION AND DATA RECOMMENDATIONS 115
Much of the potential of source linking involves data collected and main-
tained at various levels, ranging from local to federal. For example, administrative
data such as that maintained by the IRS's Statistics of Income program and by
the Social Security Administration (SSA) can enhance the robustness of studies
of the economic benefits associated with postsecondary education (Dale and
Krueger, 2002). In principle (and with due attention to legal confidentiality),
longitudinal files linking individuals' earning and their educational attainment can
be created. BLS data and states' Unemployment Insurance Wage Records could
be substitutes for this as well; typically, this kind of research must take place in
census data centers.
Institutional and System-Level Data
In many instances, information on higher education inputs and outputs can be
obtained inexpensively from institutions' accounting and management systems. In
fact, much of the detailed data on human resources and finances needed for ideal
analysis of productivity is maintained only by institutions. Many systems and
institutions have studies that distinguish expenditures and staffing by discipline
and course level in ways that are well suited to the type of analyses recommended
in this report.
Micro-level data related to learning outcomes and program quality--for
example, exam results and assignments, course evaluations, student surveys, and
faculty and staff credentials--are also available in some cases. As has been noted
throughout this report, longitudinal student data is especially valuable for tracking
the quality of incoming students and the value of higher education attainment.
An example of this kind of resource is the longitudinal study recently developed
for the University of Virginia that collects data on students following them from
kindergarten through college, and then adds the capacity to link to unemployment
insurance job records thereafter.8
Among the advantages of institution and state-level data are their high levels
of detail and accuracy, which are necessary to support evaluation of quality within
programs, departments, institutions, and systems. Among the challenges found
in institution and state-level data are that methods developed to support day-to-
day business operations are often not well configured for research and analysis.
Further, data are often not comparable across institutions or across time since
they have been built up historically using different practices, and tend to focus
more on financial information than on the physical data needed for productivity
measurement (see Figure 6.1).
8The project is being developed by the State Council of Higher Education for Virginia, which
makes higher education public policy recommendations, and it will be available to researchers with
appropriate safeguards to ensure confidentiality of records.
OCR for page 116
116 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
FIGURE 6.1 "Person-years" reported for Engineering at one university, by activity type,
2009-2010.
SOURCE: State University System of Florida Expenditure Reports. Available: http://www.
flbog.org/about/budget/expendanalysis.php [October 2012].
State Student Unit-Record Databases
All but five states have constructed databases that cover enrollments and
degrees granted in higher education (Garcia and L'Orange, 2010). At minimum,
student-level databases contain one record for every student enrolled in a given
term or year in all of the state's public institutions. Many have expanded data
content that includes detailed academic activities such as specific course enroll-
ments, placement test scores and remedial work, and financial aid sources. Many
of these state sources have also begun including data from nonpublic institutions
(both proprietary and not-for-profit). Records can, in principle, be linked together
to create the kinds of longitudinal databases required to analyze retention, de-
gree completion, and patterns of student flow among institutions within a state.
Increasingly, they are being linked with similar databases addressing elementary
and secondary schooling and entry into the workforce to enable large-scale hu-
man capital studies.
The primary challenges to effective use of these databases are (a) the lack
of standardized definitions across them, (b) incomplete institutional and content
coverage, and (c) relative inexperience in linking methodologies to create com-
prehensive longitudinal databases. Federal action to standardize data definitions
and to mandate institutional participation in such databases as a condition of
receiving Title IV funding would help stimulate productive use.
OCR for page 117
IMPLEMENTATION AND DATA RECOMMENDATIONS 117
Recommendation (14): Standardization and coordination of states' stu-
dent record databases should be a priority. Ideally, NCES should revive
its proposal to organize a national unit record database.
Such a data system would be extremely valuable for both policy and research
purposes. This is a politically difficult recommendation that may take years to
realize. In the meantime, progress can be made through the adoption of standard
definitions, based on IPEDS and the Common Data Set, for all data elements in
state longitudinal databases, plus linkages among them.9
Valuable administrative data are also collected and maintained by state and
federal agencies. Wage record and employment data are among the most relevant
for estimating productivity and efficiency measures.
Unemployment Insurance Wage Record Data
Under federal guidance, all states maintain databases of employed personnel
and wages paid for the purpose of administering unemployment compensation
programs. These employment and earnings data can be linked via Social Secu-
rity Numbers (SSNs) to educational data by researchers interested in estimating
such things as return on investment for various kinds of postsecondary training
programs and placement rates in related occupations.
The comparative advantage of Unemployment Insurance (UI) data is in its
capacity to provide aggregate estimates of outcomes--earnings by industry--for
graduates and nongraduates. Additionally, such data are not easily subject to
manipulation. The value of UI data for productivity measurement is most obvi-
ous for comparatively sophisticated models wherein outputs (i.e., degrees) are
weighted in proportion to their contribution to students' short-term or lifetime
earnings potential or other career-related outcomes; or studies that make use of
employment records to track and compare the earnings of graduates in different
disciplines to establish weights. There may also be uses for these data in assessing
compensation of higher education personnel and in establishing the quantities or
weights of inputs in calculations of productivity.
Though a powerful national resource, the potential of UI data is limited by
several factors. First, most UI systems contain data only at the industry level for
the reported employee, and not the actual occupation held within that industry.
Industry is not always a good proxy for occupation. Universities, for example,
employ police officers, landscapers, and social workers. Some employees may
thus be misclassified when conducting placement studies--for example, a nurse
working in a lumber camp being classified as a forest products worker. A simple
9This kind of work is being piloted by the Western Interstate Commission for Higher Education,
Gates Foundation data sharing project. The panel also applauds work ongoing by the Common Data
Standards Project, currently under way at the State Higher Education Executive Officers association.
OCR for page 118
118 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
fix that federal authorities could make would be to mandate inclusion of the
Standard Occupational Classification (SOC) code in all state UI wage records.
Second, each state maintains its own UI wage record file independently under
the auspices of its labor department or workforce agency. And, whether for post-
secondary education or employment, it is difficult to coordinate multi-state or
national-level individual unit record databases. For this reason, state databases
cannot track cohorts of graduates or nongraduates across state lines without some
agreement to do so.
Recommendation (15): The Bureau of Labor Statistics should continue
its efforts to establish a national entity such as a clearinghouse to facili-
tate multi-state links of UI wage records and education data. This would
allow for research on issues such as return on investment from postsec-
ondary training or placement rates in various occupations.
Such an entity could also begin to address a third major limitation of UI wage re-
cords--the fact that they do not contain data on individuals who are self-employed
or employed in the public sector. Linkages to public employee and military per-
sonnel databases could be established on a national basis and self-employment
earnings and occupation data, with the appropriate privacy safeguards, might be
obtained from the IRS. Additionally, economic analyses of earnings often require
looking many years after graduation to see the economic value of some degrees
(some biology majors, for example, may have low reported earnings during medi-
cal school and residency programs, but very high wages afterward).
State Longitudinal Databases
Many states have databases that allow for longitudinal studies of students
from K-12 education through postsecondary enrollment, degree completion, and
beyond. The U.S. government has contributed significant resources in an effort
to create or expand these databases over the last several years. For example,
late in 2010, the U.S. Department of Labor, through its Workforce Data Quality
Initiative, awarded more than $12 million to 13 states to build or expand longi-
tudinal databases of workforce data that could also be linked to education data.
As described in Chapter 2, these kinds of data are essential for research into and
policy analysis of the link between employment and education, and the long-term
success of individuals with varying levels and kinds of education.
Databases vary in age, depth, and breadth, all of which affect how they can
be used for productivity measurement. The state-maintained databases all contain
student-level enrollment and degree completion information, usually with ag-
gregate numbers of credit hours--all useful for productivity measurement. They
vary in the extent to which they include specific course-level detail. Many contain
data about K-12 preparation, which affects productivity in higher education in the
OCR for page 119
IMPLEMENTATION AND DATA RECOMMENDATIONS 119
sense that similar degrees represent different quantities of achievement (levels of
preparedness) for different students. Many also include at least some data from
private postsecondary institutions. Another asset of these kinds of databases is
that they allow for analysis that crosses institutional boundaries. They also pres-
ent many challenges: (1) the nature of the gaps in data vary by state; (2) access
to records can be subject to state political considerations; (3) they are less useful
in states with substantial mobility across state lines; and (4) they include no or
only limited amounts of financial or human resources data.
National Student Clearinghouse
The National Student Clearinghouse (NSC) is a national postsecondary
enrollment and degree warehouse that was established about fifteen years ago
to house data on student loan recipients. It has since been expanded to include
enrollment records for more than 94 percent of the nation's postsecondary enroll-
ments and almost 90 percent of postsecondary degrees, essentially rendering it a
national database.10 Its primary function is administrative, verifying attendance
and financial aid eligibility for the Department of Education, among other things.
It also provides limited research services.
Proof of concept studies have demonstrated that the data contained in NSC
records are capable of grounding accurate longitudinal tracking studies of col-
lege entrance and completion regardless of place of enrollment, thus overcoming
one of the major limitations of state SUR databases (Ewell and L'Orange, 2009).
Often when a student fails to re-enroll at an institution or within a state system of
higher education, there is no record to indicate whether the student transferred or
re-enrolled elsewhere, and therefore no way to know whether the credits earned
prior to departure are part of an eventual credential. Similarly, when a degree is
awarded, there is often imperfect information about the different institutions a
student may have attended prior to the degree award from the graduating insti-
tution. NSC's matching service can fill in some of these gaps for institutional
or state-level cohorts of students. The major advantage is that it can provide
information about cross-institution, cross-state, and cross-sector enrollments and
degrees awarded that would not otherwise be available.
The major drawback of harnessing these data for such purposes, however, is
the fact that reporting to the NSC is voluntary: institutions provide data because
they get value in return in the form of the ability to track, and therefore account
for, students who have left their institutions with no forwarding information. As
a result, some safeguards on the use of NSC data for research purposes would
need to be established and enforced. Additionally, only the enrollment and degree
award events are recorded. Although 94 percent of college enrollments are repre-
sented, states and institutions with a disproportionate share of the nonparticipants
10Available: http://www.studentclearinghouse.org/ [July 2012].
OCR for page 120
120 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
may not be able to benefit. Some researchers are also reporting problems with the
matching algorithms that make the data difficult to use and interpret correctly. 11
6.2.4. Survey-Based Data Sources
In this section, we describe several survey-based data sources that have value
to developers of productivity and other performance metrics. A fuller accounting
of data sources is provided in Appendix C.
NCES Postsecondary Sample Surveys
The National Center for Education Statistics conducts a number of relevant
surveys: Baccalaureate and Beyond (B&B), Beginning Postsecondary Students
(BPS), National Postsecondary Student Aid Survey (NPSAS), National Survey
of Postsecondary Faculty (NSOPF), and the Postsecondary Education Transcript
Study (PETS).12 These surveys use large national samples to provide research
and policy answers to questions that cannot be addressed from IPEDS. They
include information about students' level of academic preparation, transfer pat-
terns, socioeconomic status, financial resources, aid received, academic and non
academic experiences during college, and persistence and degree attainment.
NSOPF includes data about faculty activities, but have not been administered
since 2003-2004. The samples are structured to provide reliable samples at the
level of institutional sector (public four-year, two-year, private four-year, etc.)
across the United States. Only limited additional disaggregation is possible with-
out losing statistical significance.
This set of surveys includes content in a number of areas that is relevant to
productivity measurement. What an institution contributes to a student's educa-
tion should ideally be separated from what a student brought with him or her, in
terms of credits earned elsewhere, level of preparation in earlier levels of educa-
tion, and other experiences and aptitudes. NPSAS and its offshoots can help to
assign the contribution share for degrees to multiple institutions or sectors when
students transfer (as a significant percentage do), and to distinguish what institu-
tions are contributing from what students bring with them at entry.
The BPS survey, conducted on a subset of NPSAS participants, is especially
important for understanding sector-level productivity nationally, and has typically
been started with a new cohort every eight years.
11See http://www.spencer.org/resources/content/3/documents/NSC-Dear-colleagues-letter.pdf [July
2012].
12Baccalaureate and Beyond: http://nces.ed.gov/surveys/b%26b/; Beginning Postsecondary Stu-
dents: http://nces.ed.gov/surveys/bps/; National Postsecondary Student Aid Survey: http://nces.
ed.gov/surveys/npsas/; National Survey of Postsecondary Faculty: http://nces.ed.gov/surveys/nsopf/;
and Postsecondary Education Transcript Study: https://surveys.nces.ed.gov/pets/ [July 2012].
OCR for page 121
IMPLEMENTATION AND DATA RECOMMENDATIONS 121
These datasets have several positive attributes. Samples attempt to represent
all students in U.S. higher education; longitudinal follow-ups track attainment
across institutions, sectors, state lines; and students entering with different levels
of preparation can be distinguished. They also present a number of challenges.
National surveys are, by design, not useful as state- or institution-level resources;
surveys are administered infrequently; and surveys are costly to implement and
to scale up.
National Science Foundation
The National Science Foundation (NSF) conducts surveys on sciences and
engineering graduate and postgraduate students.13 Information is collected on
type of degree, degree field, and graduation date. Data items collected in NSF
surveys are more attuned toward understanding the demographic characteristics,
source of financial support and posteducation employment situation of graduates
from particular fields of science, health, and engineering. These data are useful
in understanding trends in salaries of science, technology, engineering, and math-
ematics (STEM) graduates.
NSF surveys concentrate on degree holders only. The sampling frame does
not include individuals who have not graduated from a higher educational insti-
tution. There is no information on credits completed by degree and nondegree
holders. Although the sampling frame is limited for purposes of calculating in-
stitutional productivity for undergraduate programs, it does collect information
on post-bachelor degree holders, postdoctoral appointees, and doctorate-holding
nonfaculty researchers. Sampling techniques and data items collected also make
the NSF data useful for calculating department level (within STEM fields) output
in terms of research and academic opportunities available to graduates.
The NSF survey of academic R&D expenditures is valuable to federal, state,
and academic planners for assessing trends in and setting priorities for R&D
expenditures across fields of science and engineering, though it is not directly
related to instructional productivity.14 It has potential for indirect use in estimat-
ing the volume of research expenditure, by discipline, at different institutions in
order to better untangle joint products. In contrast, IPEDS provides only aggre-
gate research expenditure, with no disciplinary detail. This NSF survey may be
the only national source for this limited purpose.
13See http://nsf.gov/statistics/survey.cfm [July 2012].
14Survey description: http://www.nsf.gov/statistics/srvyrdexpenditures/ and WebCaspar data access:
https://webcaspar.nsf.gov/ [July 2012].
OCR for page 122
122 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
Census and the American Community Survey
The American Community Survey (ACS), which replaced the Census long
form, is a national sample survey of 2 million households that became fully
implemented in 2005. ACS data are used to produce estimates for one-year,
three-year, and five-year periods. Each month, the ACS questionnaire is mailed to
250,000 housing units across the nation that have been sampled from the Census
Bureau's Master Address File.15 As with the long-form of the Census, response
to the ACS is currently required by law. The questionnaire has items on sex,
age, race, ethnicity, and household relationship. Each observation is weighted
to produce estimates. Weighting is done via a ratio estimation procedure that
results in the assignment of two sets of weights: a weight to each sample person
record, both household and group quarters persons, and a weight to each sample
housing unit record. There are three education-related variables in the ACS: col-
lege or school enrollment in the three months preceding the survey date, current
grade level, and educational attainment, including field of bachelor's degree. ACS
collects data from households and group quarters. Group quarters include institu-
tions such as prisons and nursing homes but also college dormitories.
In terms of value for productivity measurement, no information is collected
on credit hours and colleges or universities attended or completed by survey
respondents enrolled in college. The questionnaire has items on various sources
and amounts of income, and details on occupation and work in the year preceding
date of survey. ACS data thus provides descriptive statistics of educational status
of various population groups (even in small geographic areas like census tracts),
but it lacks relevant information to calculate institutional productivity. In the ACS,
survey respondents change from year to year. No household or person is followed
over time. Therefore it is difficult to understand educational pathways which
other education-related data sources address. At best, ACS provides a snapshot
of educational status of the U.S. population based on a sample size larger than
that of other data sources.
The major attraction of the ACS, for the purposes here, is its comprehensive
population coverage. Its limitations are that it is a relatively new survey, with data
available from 2006; only three education-related variables are present and the
data are not longitudinal.
Bureau of Labor Statistics
The Bureau of Labor Statistics (BLS) conducts two longitudinal surveys, the
National Longitudinal Survey of Youth 1979 (NLSY79) and National Longitudi-
nal Survey of Youth 1997 (NLSY97). These gather information on education and
employment history of young individuals. The survey begun in 1979 is still ac-
15See http://www.census.gov/acs/www/ [June 2012].
OCR for page 123
IMPLEMENTATION AND DATA RECOMMENDATIONS 123
tive, over three decades later.16 The schooling survey section collects information
on the highest grade attended or completed, earning of GED/high school diploma,
ACT and SAT scores, Advanced Placement (AP) test (grades, test taken date,
subject of test, highest AP score received), range of colleges applied to, college
enrollment status, field of major and type of college degree (bachelor or first pro-
fessional), number and types (two-year or four-year) of colleges attended, credits
received, major choice, college GPA, tuition and fees, sources and amounts of
financial aid. Survey respondents were administered Armed Services Vocational
Aptitude Battery test and Armed Forces Qualifications Test (only for NLSY79
respondents and Peabody Individual Achievement Test for NLSY97 respondents)
and the respective scores are available in the dataset. The employment section has
items on types of occupations, education requirements and income in different
occupations and pension plans. Along with sections on employment and school-
ing, both the surveys cover areas such as health, family formation, relationships,
crime and substance abuse, program participation, etc.
The National Longitudinal Surveys of Youth are important. They can track
a survey respondent over time across more than one educational institution. For
each institution attended by the respondent, information on credit hours, degree
attained and other associated variables are collected. Information about each in-
stitution attended (IPEDS code) is available in restricted files. Using the IPEDS
code a researcher can access the institutional information available in IPEDS
survey files. The database is also helpful in looking at multiple enrollment pat-
terns, kinds of jobs held, and information on graduates' salaries. The National
Longitudinal Surveys of Youth are among the longest-running longitudinal sur-
veys in the country.
Even though the NLSY samples are representative of the U.S. youth popula-
tion, one cannot calculate institutional productivity for single colleges or universi-
ties, unless a sufficient number of observations is available. The survey collects
information on institutions attended by survey respondents. Therefore it is not
comprehensive because data covering a reasonable period of time are not avail-
able for all institutions.
Student and Faculty Engagement Surveys
Student and Faculty Engagement surveys gather information on learning
gains and are available in different formats for different types of institutions.
For two-year and four-year undergraduate institutions, both students and faculty
16See http://www.bls.gov/nls/ [June 2012].
OCR for page 124
124 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
are surveyed. For law schools, only students are surveyed. Participation in these
surveys is optional.17
The National Survey of Student Engagement and the Law School Survey do
not gather information on final output such as degrees and credit hours. Rather,
their focus is on intermediate outputs, such as learning during enrollment. The
community college engagement survey asks students to report the range of credit
hours completed rather than the exact number of credit hours. The faculty surveys
collect information on full-time/part-time status, number of credit hours taught
and rank. There is no survey of nonfaculty staff. No information is collected on
faculty or staff salaries. Information gathered in the surveys can be supplemented
by individual institutions by linking student responses to other institutional data.
Results from the survey can be used to estimate sheepskin effects. As students
report a field of major, the results can be used to deduce learning gains in vari-
ous fields.
The most useful attribute of student and faculty engagement surveys is that
they collect data at the individual institution level and can be tailored to specific
needs. Among their drawbacks is that participation at institutional, faculty, and
student level is optional; and facility to convert survey results to productivity
measures has not been developed.
17Beginning College Survey of Student Engagement: http://bcsse.iub.edu/; Community College
Faculty Survey of Student Engagement: http://www.ccsse.org/CCFSSE/CCFSSE.cfm/; Community
College Survey of Student Engagement: http://www.ccsse.org/aboutsurvey/aboutsurvey.cfm/; Faculty
Survey of Student Engagement: http://fsse.iub.edu/; Law School Survey of Student Engagement:
http://lssse.iub.edu/about.cfm/; and National Survey of Student Engagement: http://nsse.iub.edu/html/
about.cfm/ [July 2012]. The Community College Leadership Program at the University of Texas, Aus-
tin, conducts the community college survey. Other surveys are conducted by the Indiana University
Center for Survey Research in cooperation with its Center for Survey Research.
