Click for next page ( 62


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 61
4 Advancing the Conceptual Framework This chapter presents a framework intended to provide a starting point for measuring productivity in higher education. Chapters 2 and 3 presented argu- ments for why productivity measurement in higher education is exceedingly dif- ficult and why, in turn, the panel cannot simply prescribe a fully defined metric. Nonetheless, because governments and many other stakeholders insist on, and in fact need, an aggregate measure of productivity change, it is important to begin developing the best measure possible. The measure proposed involves a number of important assumptions and ap- proximations, which are elaborated below. Chief among these is the lack of an agreed-upon measure of educational quality. Productivity should be defined as the ratio of quality-adjusted outputs to quality-adjusted inputs, but the needed quality adjustments are not currently possible in higher education and are not likely to become possible any time soon. We recognize the problem, but believe it is important to extract as much information as possible from the (quantitative) data that can be measured. We will describe later how the risks associated with the lacuna of measures of quality can be minimized including, for example, how entities can use university and third-party quality assurance methods to ensure that focusing on the quantitative inputs and outputs does not trigger a "race to the bottom" in terms of quality. 4.1. CHAPTER OVERVIEW The productivity measure proposed here is consistent with the methodology practiced by the Bureau of Labor Statistics (BLS), and offers several significant advantages over the ad hoc approaches that have been used to date. In particular: 61

OCR for page 61
62 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION The measure is a multi-factor productivity index. It captures output in physical units (credit hours, degrees) and, unlike cost studies, measures direct labor inputs in terms of full-time equivalents (FTEs). Labor pro- ductivity can be derived from the multi-factor results if desired. Outputs include credit hour production and degree attainment, both of which have been shown to be important in labor market studies. Most if not all the measures currently in use (e.g., credit hour production alone or graduation rates) depend on one or the other but not both, and there- fore miss a critical output dimension. The measure does not vary along with the proportion of part-time stu- dents, except to the extent that being part-time might require different student services or contributes to wasting credits or dropping out. This feature sidesteps the problem of comparing graduation rates and aver- age times to degree among schools with different numbers of part-time students. Credits not on the mainline path to a degree, including those due to changes in major and dropouts, are counted and thus dilute the degree completion effect. In other words, programs with a heavy dropout rate will have more enrollments per completion, which in turn will boost resource usage without commensurate increases in degrees. Productiv- ity could thus increase with the same number of credit hours if more students actually complete their degrees. Credit earned, however, is not treated as entirely wasted just because a degree was not awarded. The measure allows differentiation of the labor and output categories, although doing this in a refined way will require significant new data. The measure readily lends itself to segmentation by institutional type, which is important given the heterogeneity of the higher education sector. The measure can in principle be computed for institutions within a state, or even single institutions. However, the incentives associated with low-aggregation level analyses carry the risk of serious accuracy deg- radation and misuse unless it is coupled with robust quality assurance procedures. Until quality adjustment measures are developed, the panel advises against using the productivity metric described in this chapter for institution-to-institution comparisons (as opposed to more aggregate level, time series, or perhaps state-by-state or segment analyses). Data collection, including data beyond the Integrated Postsecondary Education Data System (IPEDS) and the proposed special studies, ap- pears to be feasible. We emphasize again that the proposed measure follows the paradigm of ag- gregate productivity measurement, not the paradigm for provision of institution- level incentives and accountability. As stressed in Chapter 3, institutions should

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 63 be prepared to resist inappropriate initiatives to improve productivity as measured by applying the formula below to their particular data, and to buttress their resis- tance with their own internal data about quality. Section 4.2 presents our base model. It is a "multi-factor productivity model" in that it uses output and input quantities and includes all categories of inputs. Section 4.3 proposes a segmentation scheme, which is important because of the heterogeneity of higher education. The section also discusses how the model can be computed at the state and single-institution level but, again, we stress that this will be dangerous without a robust quality assurance system. Section 4.4 enhances the base model by differentiating among labor categories. This is important because of the fundamental difference between academic and nonaca- demic labor, and the difference between tenure-track and adjunct faculty. Section 4.5 differentiates among output categories, which again is important because of institutional heterogeneity and the fact that production of degrees at different levels and in different fields involves different production functions. Finally, Section 4.6 presents the rationale for using the model in conjunction with quality assurance procedures. Nearly all the data required for calculating values using the model sketched out here can be obtained from the U.S. Department of Education's IPEDS or other standard public sources (though this would not be the case for the fully specified "ideal"). Adding the model refinements outlined in Section 4.3 requires a modest amount of additional information. Data requirements for the enhance- ments described in Section 4.4 can be approximated from IPEDS, but proper implementation will require additional data collection. The panel's recommended changes to IPEDS are discussed in detail in Chapter 6. The new data that are called for would break useful ground not only for productivity analysis, but also for institutional planning and resource allocation. This is important because an institution's use of data for its own purposes makes data collection more palatable and improves accuracy. 4.2. A BASELINE MULTI-FACTOR PRODUCTIVITY MODEL FOR HIGHER EDUCATION Following the concepts defined in Chapter 2, the model calculates the ratio of changes in outputs (credit hours and degrees) to inputs (labor, purchased ma- terials, and capital). The focus is on instructional productivity, with inputs being apportioned among instruction, research, and public services prior to calculating the productivity ratio. As emphasized throughout this report, our model involves only quantitative factors. It will be reliable only to the extent that input and output quality remains approximately constant, or at least does not decline materially. Currently, quality measurement--of both inputs and outputs--is largely beyond the capacity of quantitative modeling; but, because quality should never be taken for granted, we return to the issue at the end of the chapter.

OCR for page 61
64 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION 4.2.1. Multi-Factor Productivity Indices Multi-factor indices relate output to a bundle of inputs; that is, they show how much of an industry's or firm's output growth can be explained by the combined changes in its inputs. The panel has concluded that a multi-factor pro- ductivity index is appropriate for measuring higher education productivity at the segment and sectoral levels. Other kinds of productivity models--for example, those which estimate educational production functions--are of course possible and worthwhile. However, the panel was not charged with recommending such models. Nor was it charged with developing strategies for improving productivity. Our proposed productivity model is based on the methodology for multi- factor productivity indices used by BLS, the OECD, and other U.S. and foreign agencies that produce sectoral productivity statistics (Bureau of Labor Statistics, 2007; Schreyer, 2001). For example, BLS uses this methodology to calculate productivity indices for aggregate manufacturing and some eighteen manufactur- ing industries. The BLS method uses what is known as a Trnqvist index. This differs in important ways from the method of simply calculating weighted averages of the variables in the numerator and denominator and then taking the ratio of the two averages. The key ideas behind the Trnqvist index are as follows (from Bureau of Labor Statistics [2007:6-7]): 1. The figures for input and output are calculated as weighted averages of the growth rates of their respective components. Weighting average growth rates avoids the assumption, implicit in directly averaging the variables, that the inputs are freely substitutable for one another. It also removes issues having to do with the components' dimensionality. Both attributes are important when comparing variables like adjusted credit hours with labor and other inputs. 2. The weights are allowed to vary for each time period in which the index is calculated. This means the index always represents current informa- tion about the relative importance of the variable in question while maintaining the requirement (discussed in Chapter 2) that the weights move more slowly than the variables themselves. 3. The weights are defined as the means of the relative expenditure or revenue shares of the components for the two data periods on which the current index is based. This method brings relative wages and prices into the equation because they affect total expenditures. The Trnqvist scheme has often been the indexing structure of choice for describing multi-factor productivity change under fairly broad and representative assumptions about the nature of production: specifically, that the production func- tion can be represented by a translog generalization of the familiar Cobb-Douglas

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 65 function with mild regularity conditions on the parameters (Caves et al., 1982). A technical description of the Trnqvist methodology is provided in the appendix to this chapter. Determining appropriate indices embodying the general ideas put forward in Chapters 2 and 3 remains a task for future work. As such, it would be premature for the panel to commit to a specific approach. However, because of its wide- spread use in other applications--a Trnqvist index is used here for expository purposes. The denominator of our baseline higher education productivity index uses a Trnqvist structure to represent the composite growth rates of labor and capital inputs. The numerator also takes the form of a Trnqvist index, though in this case with only one element. Section 4.4 extends the index in the denomina- tor to include more than one labor category, and Section 4.5 uses multiple output categories in the numerator. The final productivity index is the ratio of the indices in the numerator and the denominator. 4.2.2. Outputs On the output side, the model uses two data elements that can be obtained from IPEDS: 1. Credit Hours: 12-month instructional activity credit hours summed over undergraduates, first-year professional students, and graduate students; 2. Completions: awards or degrees conferred, summed over programs, levels, race or ethnicity, and gender.1 Illustrative data for a four-year private university are shown in Table 4.1. 2 The data cover three years: 2003, 2006, and 2009 (it is best to aggregate over multi- year periods to reduce volatility associated with noise in the data, but the illustra- tion ignores that refinement). For reasons explained earlier, both credit hours and degrees (or completions) are included as outputs. Whatever their flaws, these are the standard unit measures of instruction in American higher education. The model uses adjusted credit hours as its measure of output, defined as follows: Adjusted credit hours = Credit hours + (Sheepskin effect Completions) 1For broader use, definitions become more complicated. For example, as discussed in the next chapter and elsewhere in the report, "completions" defined as certificates and successful transfers become relevant in the community college context. Nondegree seekers (e.g., summer transients) also come into play at many kinds of institutions. 2The data are based on an actual institution, but certain adjustments were made to make the illustration more coherent.

OCR for page 61
66 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION TABLE 4.1 Illustrative Data from IPEDS for the Base Model Period 1 Period 2 Period 3 Enrollments and Completions Credit hours 578,815 574,176 602,000 Completions 2,154 2,310 2,500 Adjusted credit hours 638,435 643,476 677,000 Total Number of Staff Full time 6,265 6,656 6,826 Part time 683 4,949 2,250 Labor FTEs 6,493 8,306 7,576 Finance: Core Expenditures Wages and Fringe Benefits Instruction $421,534 $525,496 $641,749 Research 295,814 531,759 424,075 Public service 5,339 5,500 5,700 Student services 39,178 50,113 62,626 Administration and support services 488,969 563,969 534,924 Intermediate Expenditures Instruction $161,142 $328,987 $427,833 Research 436,824 332,909 424,075 Public service 463 450 450 Student services 19,643 31,374 62,626 Administration and support services 491,953 366,841 534,924 Total Cost Instruction $582,676 $854,483 $1,069,582 Research 732,638 864,668 848,149 Public service 5,802 5,950 6,150 Student services 58,821 81,487 125,251 Administration and support services 980,921 930,810 1,069,847 Finance Balance Sheet Items Land improvements; ending balance $233,698 $238,269 $269,551 Buildings; ending balance 2,370,981 2,455,427 2,940,552 Equipment, including art and library; 1,150,228 1,191,801 1,372,257 ending balance Total Capital $3,754,907 $3,885,497 $4,582,360 NOTE: FTE = full-time equivalent, IPEDS = Integrated Postsecondary Education Data System. SOURCE: This, and all other tables in Chapter 4, were calculated by the panel and staff. The "sheepskin effect" represents the additional value that credit hours have when they are accumulated and organized into a completed degree. The panel believes that a value equal to a year's worth of credits is a reasonable figure to use as a placeholder for undergraduate degrees.3 Additional research will be needed to determine the sheepskin effect for graduate and first professional programs. 3Jaegerand Page (1996) suggest something more than an additional year for the sheepskin effect. They conclude "Sheepskin effects explain approximately a quarter of the total return to completing 16

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 67 4.2.3. Inputs Inputs consist of the following variables, which can be calculated mainly from IPEDS data as shown in Section 4.2.4. a) Expenditures on Labor (LE): nominal value of salaries and wages plus fringe benefits, used as the weight of L when aggregating the input. b)Labor (L): the quantity measure for labor input, approximated by full- time equivalent (FTE) employees. Both academic and nonacademic employees are included in the calculation (this assumption, driven by the limitations in IPEDS data categorization, is relaxed in Section 4.4). FTE figures are calculated from total full- and part-time employees, with a part-time employee counting as one-third of a full-time employee, as assigned in IPEDS (this, too, could be adjusted with empirical justifica- tion). Labor is the biggest input into higher education instruction. c) Expenditures on Intermediate Inputs (IE): nominal cost of materials and other inputs acquired through purchasing, outsourcing, etc. (the sum of the IPEDS "operations & maintenance" [O&M] and "all other" categories). These nominal values are used in calculating weights for intermediate inputs. d) Intermediate Inputs (I): Deflated nominal expenditures (IE) are used to represent the physical quantities. e) Expenditures on Capital (KE): opportunity cost for the use of physical capital; also called rental value of capital. Expenditures equal the IPEDS book value of capital stock times an estimated national rate of return on assets, where book value of capital stock equals the sum of land, buildings, and equipment.4 Overall, the book value reported in IPEDS is likely too low; however, it does include buildings that may not be specifically allocated to teaching, which offsets the total to an unknown degree.5 These nominal capital values are used in calculating the capital weights. years of education and more than half of the return to completing 16 years relative to 12 years. . . . The marginal effect of completing a Bachelor's degree over attending `some college' is 33%, conditional on attending school for 16 years." Park (1999) found the sheepskin effect to be somewhat lower. Wood (2009) provides a review of the literature. See also Section 5.1.1. 4Book value is typically defined as the original cost of an asset adjusted for depreciation and amortization. 5An alternative option was considered: current replacement value, the cost to replace an asset or a utility at current prices. This figure is available in IPEDS estimates of current replacement value for educational institutions and also calculated by Sightlines, a private company (see http://www. sightlines.com/Colleges-Universities_Facilities.html [June 2012]). Sightlines calculates current re- placement value based upon the age, function, and technical complexity of each building. Current replacement value is defined as the average cost per gross square foot of replacing a building in kind in today's current dollar value. The Sightlines figures reflect the total project cost, including soft

OCR for page 61
68 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION f)Capital (K): For the quantity of capital input, the book value is deflated by the Bureau of Economic Analysis's investment deflator for gross private domestic investment. The deflators for intermediate expenditures and capital are, respectively, the Producer Price Index (PPI) and the index for Gross Private Domestic Investment: Chain-Type Price Index (GPDICTPI). These figures cannot be obtained from IPEDS but they are available from standard sources.6 4.2.4. Allocations to Education The first step in the allocation process is to isolate inputs for the institutions' educational function from those attributable to the research and public service functions. Because IPEDS does not break out the FTE and capital variables by function,7 our approach is to allocate these variables proportionally to expen- ditures by function--which are available in IPEDS. Here, "education" means "Education and Related Cost" (E&R), as defined by the Delta Project on Post- secondary Education Costs as "Instruction plus Student Services" (Delta Cost Project, 2009). The allocation formulas are: L = FTE(EdShAllLE + EdShDirLE AdShAllLE) LE = DLEI + DLES + DLEA EdShDirLE I = IE/PPI IE = DIE + DIS + DIA EdShDirIE K = KE/GPDICTPI KE = Stock ROR(EdShAllTot + EdShDirTot AdShAllTot) costs, and are adjusted for architectural significance and region. Of course there is scope for further exploration and refinements in this estimate which is best left to the judgment of college/university authorities. As per e-mail exchange between one panel member and Jim Kadamus (vice president of Sightlines), the company's staff conducted some preliminary comparisons between current replace- ment values as calculated by IPEDS and Sightlines. The Sightlines estimates (for comparable space) are 70-100 percent greater than the value reported in IPEDS. The entity charged with implementing the productivity model will have to decide which estimate to use. 6 The PPI is available at http://www.bls.gov/ppi/ and the GPDICTPI is available from http://www. bea.gov via the GDP and personal income interactive data link. 7The IPEDS Human Resources section provides a functional breakdown for direct teaching, research, and public service staff, but only an occupational breakdown for nonteaching staff. This scheme does not map into our model, and in any case the functional breakdown may be unstable due to inconsistencies in institutional classification schemes.

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 69 Additional variables (beyond those on p. 67) are defined as: DLEI = "Direct labor expenditures for instruction" as given by IPEDS; no- tations follows this format for student services (DLES) and administration. EdShAllLE is "Education's share of all labor expenditures"; AdShAllLE is "Administration and support services' share of all labor expenditures"; notation follows this format for intermediate expenditures and capital. Finally, EdShDirLE is "Education's share of direct labor expenditures," and similarly for all the other shares. The difference between EdShAllLE and EdShDirLE is that the former's denominator includes labor expen- ditures for administration and support services whereas the latter's does not. IE is expenditures on intermediate inputs; DIE is direct nonlabor ex- penditures on instruction; DIS is direct nonlabor expenditures on stu- dent services; DIA is direct nonlabor expenditures on administration; EdShDirIE is education share of direct nonlabor expenditures. Stock is capital stock as shown on institutional balance sheets, ROR is the national rate of return on capital, PPI is the producer price index, and CPDICIPI is the price index for gross private domestic investment (both price indices are suitably normalized).8 Faculty time that is separately budgeted for institutional service is included in administration and support services, and unbudgeted faculty service time (e.g., departmental administration) is included in instruction. 4.2.5. Illustrative Productivity Calculations Table 4.2 shows the productivity calculation for the institution referred to above. The calculation can be broken down into four steps: 1. Allocate the quantity and expenditure data to the Education function: Apply the formulas above. For example, adjusted credit hours in the three periods equal 638,435, 643,476, and 677,000. 2. Calculate the change in the quantity data from period to period: The change for adjusted credit hours equals the current value divided by the 8See Hodge et al. (2011:25, Table 2). The rate of return to the net stock of produced assets for other nonfinancial industries is used as a proxy for the rate of return to higher education land, buildings and equipment. Other industries includes agriculture, forestry, fishing and hunting; transportation and warehousing; information; rental and leasing services and lessors of intangible assets; profes- sional, scientific, and technical services; administrative and waste management services; educational services; health care and social assistance; arts, entertainment, and recreation; accommodation and food services; and other services, except government.

OCR for page 61
70 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION TABLE 4.2 Base Model Productivity Calculations Period 1 Period 2 Period 3 Step 1: Allocations to Education Outputs Adjusted credit hours (ACHs) 638,435 643,476 677,000 Input Quantities Labor FTEs (L) 3,926 4,296 4,705 Intermediate expenditures $324,680 $486,147 $643,599 Rental value of capital (K) $261,834 $267,507 $348,033 Input Expenditures Wages and fringe benefits $756,399 $867,311 $1,036,594 Intermediate expenditures $324,680 $550,921 $777,193 Retail value of capital (K) $261,834 $301,954 $400,791 Total Cost $1,342,913 $1,720,186 $2,214,578 Step 2: Quantity Changes Period 1 Period 1 2 Period 2 3 Output Change Adjusted credit hours 1.000 1.008 1.052 Input Change Labor FTEs 1.000 1.094 1.095 Real intermediate expenditures 1.000 1.497 1.324 Real capital stock 1.000 1.022 1.301 Step 3: Input Index Weights (average) Wages and fringe benefits 53.4% 48.6% Normal intermediate expenditures 28.1% 33.6% Real capital stock 18.5% 17.8% Weighted geometric average 1.180 1.204 Step 4: Multi-Factor Productivity Productivity index 0.854 0.874 Productivity change 2.3% NOTE: FTE = full-time equivalent. previous one, with the first value being initialized at one. The ratios for Periods 2 and 3 are 1.008 and 1.052, for example, which indicate growth rates of 0.8 percent and 5.2 percent. 3. Calculate the input index: For inputs, the composite index ("weighted geometric average") is equal to the geometric average of the indices for the individual variables using the arithmetic average of the successive periods' nominal expenditure shares as weights (no averaging is needed for outputs because there is only one output measure).9 This calculation 9The index for the more complicated models presented later is based on the geometric average of output changes, the same as for inputs.

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 71 BOX 4.1 Productivity and Quality The productivity measure here does not take account of quality changes. Instead, it depends on the market to police quality erosion. Normally such polic- ing is done through the price mechanism, although sometimes products such as computers whose quality is increasing over time do become cheaper. As argued earlier in this report, higher education prices generally are not set in competitive markets. Hence the conclusion, "Productivity has increased by `x' percent" must be taken as tentative until the constancy of quality has been verified, for example, through a separate quality assurance procedure. A master artist produces 10 paintings in a month; her student also produces 10 paintings in the same period, for example. We sense that the quality of the artists is different, and that this dif- ference should be reflected in the final product. But we cannot tell the difference in productivity just by counting the hours and the paintings (nor can we quantify the quality difference just by looking at them)--though we may eventually get some evidence by tracking the price that the paintings sell for or whether they sell at all. We return to the question of output quality in Section 4.6. comes from the third equation in the Technical Appendix. The results are 1.180 and 1.204, which indicate average input growths of 18.0 percent and 20.4 percent. 4. Calculate the productivity index: This is expressed as the ratio of the index for the change in outputs to the index for the change in inputs-- i.e., 1.008/1.180 = 0.854 for the first period and 1.052/1.204 = 0.874 for the second. The last line, "productivity change," is the ratio of the productivity indexes for the two periods: (0.854/0.874 1), or 2.3 percent. An alternative but equivalent calculation illuminates Step 4. Notice that the out- put index grew by 1.052/1.008 = 1.044 (4.4 percent) and the input index grew by 1.204/1.180 = 1.020 (2.0 percent) between the second and third periods. Dividing the output growth by the input growth yields 1.044/1.020 = 1.023, or 2.3 percent, the same as in the table. Put another way, the output index grew 2.3 percent faster than the input index--which represents productivity improvement (see Box 4.1). 4.3. INSTITUTIONAL SEGMENTATION AND DISAGGREGATIVE INDICES Having established the basic productivity index, we now present refine- ments intended to make it more useful. This section describes how indices can

OCR for page 61
76 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION TABLE 4.3 Additional IPEDS Data for the Differential Labor Market Period 1 Period 2 Period 3 Quantities Number of FTEs--instruction, research, and public 1,564 1,601 1,1601 service staff with faculty status Number of PT/PI staff 2 2 2 Number of other staff 4,928 6,704 5,974 Expenditures Expenditures on wages and salaries for instruction, $529,572 $701,791 $874,051 research, and public service Expenditures on fringe benefits for instruction, 193,203 361,064 193,373 research, and public service Expenditures on wages and salaries for other 380,610 410,296 482,139 functions Expenditures on fringe benefits for other functions 147,449 203,686 119,510 Average salary for FT instructional, staff 108,200 114,464 122,508 NOTE: FT = full-time, FTE = full-time equivalent, PT/PI = part-time/primarily instruction. come directly from IPEDS, with those for other functions obtained by subtrac- tion. Average salary for full-time instructional staff comes from the corresponding table in IPEDS. The very small PT/PI share may be an artifact of the particular data used in the example, but because the number is growing across the sector we believe this variable remains worthy of consideration. Only one required data element is unavailable in IPEDS: the ratio of PT/PI salaries per FTE to those of regular faculty. It is possible (though perhaps not cost-effective) that IPEDS could be expanded to get this information; absent the change it may be adequate to assume the compensation ratio or determine it by special study.11 Finally, the expenditures for tenure-track and adjunct faculty expenditures are subtracted from total labor expenditures to get the figure for other labor. Table 4.4 illustrates the calculations. The first step is to allocate the quantities and expenditures to the Education function. The figures for full-time tenure-track faculty and other staff are portioned using the education share variable computed in the base model. PT/PI staff needs no allocation because they are "primarily instruction" to start with. The FTE figures for Period 3 have been adjusted to demonstrate the effect of substituting PT/PI staff for full-time tenure-track fac- ulty (discussed below): specifically, we subtracted 100 instructional FTEs from tenure-track faculty and added them to PT/PI. 11One-third may be a reasonable approximation. We obtained this figure by assuming that (a) PT/ PI staff get about $5,000 per course with no benefits, (b) average annual faculty salary plus benefits is $90,000, and (c) the average full-time faculty member teaches 6 courses a year. In this case the calculation is $5,000/($90,000/6) = 1/3.

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 77 TABLE 4.4 Differentiated Labor Index Calculations Period 1 Period 2 Period 3 Allocations to Education Quantities Regular faculty FTEs 946 828 828 PT/PI staff FTEs 2 2 2 Other staff FTEs 2,978 3,466 3,875 Expenditures Effective fringe benefit rate 36.5% 51.4% 22.1% Regular faculty $139,667 $143,552 $123,891 PT/PI staff $28 $32 $30 Other staff $616,704 $723,726 $912,673 Total expenditures $756,399 $867,311 $1,036,594 Index Calculations Quantities FT regular faculty 1.000 0.876 1.000 PT/PI staff 1.000 1.000 1.000 Other staff 1.000 1.164 1.118 Quantity Change Period 1 2 Period 2 3 FT regular faculty 0.876 1.142 PT/PI staff 1.000 1.000 Other staff 1.164 1.118 Weights FT regular faculty 17.5% 14.3% PT/PI staff 0.00% 0.00% Other staff 82.5% 85.7% Labor Index Geometric average 1.107 0.985 Change in the average (0.110) NOTE: FT = full-time, FTE = full-time equivalent, PT/PI = part-time/primarily instruction. Calculating the expenditures requires the effective fringe benefits rate to be computed as a preliminary step. (It equals the ratio of fringe benefits to salaries and wages in Table 4.3.) The expenditure figures can be computed as follows: Expenditures for regular faculty = (1+ Fringe benefits rate) Average salary for FTE instructional staff Regular faculty FTEs; Expenditures for PT/PI staff = (1+ Fringe benefits rate/2) PT/PI salary ratio Average salary for FTE instructional staff PT/PI FTEs; and Expenditures for other staff = Total wages + fringe benefits (Expenditures for regular faculty and PT/PI staff).

OCR for page 61
78 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION Total "Wages & fringe benefits" is taken from Table 4.1, and readers may notice that PT/PI staff receive only half the fringe benefits rate. The final step, calculating the change indices and their weights, proceeds as in the base model. As shown in Table 4.4, resulting geometric averages show more variation than did the labor indices in Table 4.2. This produces larger multi- factor productivity indices, and also a larger change in the index, as shown at the bottom of the table. Which approach is more accurate may become more clear during implementation. 4.5. DIFFERENTIATING OUTPUTS Another potentially key model enhancement is to control for the heteroge- neity of educational outputs. Institution-level cost data indicate clearly that the resources required for producing an undergraduate degree vary across fields. Likewise, the resource requirements for producing bachelor degrees differ sys- tematically from those for associate, graduate, and first professional degrees. Fail- ure to control for these differences would risk the kinds of distortions described earlier. For example, a shift of outputs from the more expensive disciplines of science, technology, engineering, and mathematics (STEM) to non-STEM disci- plines would falsely boost the productivity index. IPEDS provides data on degrees by field and award level. A difficulty arises only in differentiating the credit hours associated with degree production. It is not unusual for institutions to track credit hours by the department or broad dis- cipline in which a course is taught (as required to apply the so-called Delaware cost benchmarks for example), but these data cannot be mapped directly to degree production because students take many courses outside their matriculated area. Researchers have made the necessary correspondences by creating course-taking profiles for particular degrees, but these matrices are difficult to manage and maintain on an institution-wide basis. There is a better way, which we outline below--one that feeds directly into the productivity statistics and produces the course-taking profiles as by-products.12 For the long run, credit-hour data for productivity analysis should be col- lected in a way that follows the students, not only the departments that teach them. The necessary information exists in most institutions' student registration files and the needed statistics can be extracted as follows: Identify the students matriculated in a given degree program ("output category") as defined by the IPEDS fields and degree levels. Undeclared students and students not matriculated for a degree would be placed in separate ("nonattributable") output categories. 12Simply applying a sheepskin effect to each degree category and summing the result before add- ing to aggregate credit hours is insufficient because shifts in degree production will induce shifts in credit-hour production, which will produce the kinds of distortion described in the text.

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 79 For each output category, accumulate the credit hours earned by the stu- dents in that category, regardless of the department in which the course was offered or the year in which it was taken. Allocate credits earned by matriculated but undeclared students in pro- portion to the credit-hour fractions of declared students for the given degree. Retain nonmatriculated students in their own separate category, one that has no sheepskin effect but in other respects is treated the same as other categories. The question arises as to whether these data need to be collected by each institution, or whether generalized weights based on special studies (e.g., using the Postsecondary Educational Transcripts Study, PETS) could be used. The use of generalized weights is not inconceivable, but we worry that the heterogeneity of programs from school to school means that much information would be lost. Further, institutions may find the data on credit hours by degree program useful for internal purposes as well as for reporting--for example, in studying student profiles of course-taking behavior and benchmarking costs per degree program. It seems likely that, once in hand, these data will open significant new opportunities for institutional research. It may be some time before data on student-based credit hour accumulations can be obtained, but there is a simple interim procedure that can be computed from the available IPEDS data. It is to allocate total enrollments to fields and levels based on fractions of completion. While ignoring differences in the course- taking profiles, the procedure does allow at least some differentiation among output categories. Aggregation to a single output index is best accomplished by taking a geo- metric average of the output category indices using their net student revenue shares as weights. A geometric average with weights equal to revenue shares reflects the BLS methodology described earlier. Use of net as opposed to gross shares appears reasonable because revenue based on net prices (which is what the institution can spend on operations) is consistent with the underlying Trnqvist model as presented by Caves, Christensen, and Diewert (1982). We recognize that getting these data may be problematic for institutions, in which case an alternative based on reported tuition and financial aid rates might well suffice. We also should note that it is not necessary to allocate any inputs across output categories. Such a requirement would disrupt data collection. Like most sectoral productivity indices, ours simply divides the aggregate output index for a segment by its aggregate input index. 4.6. VARIATIONS IN OUTPUT QUALITY The quality of education is the elephant in the room in all discussions about instructional productivity, and the issue has been raised repeatedly in this report.

OCR for page 61
80 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION The panel would have liked nothing better than to propose an explicit quality adjustment factor for weighted credit hours as part of our conceptual framework. In our model, an effect will be captured to the extent that higher quality inputs lead to higher graduation rates (and, in turn, a larger degree bonus in the numera- tor), but this effect is indirect. For example, if small classes or better teaching (inputs of different quality) lead to higher graduation rates, this will appear in the output numerator as a greater sheepskin effect. Similarly, high student and teacher quality at selective private institutions may offset high input costs by creating an environment conducive to high throughput rates. This modest step notwithstanding, full (or even adequate) integration of qual- ity adjustment into a productivity measure will not be possible any time soon. Significant progress on quality assessment has been made, but there is a long way to go before a generally accepted cardinal measure--one that can be used reliably to adjust weighted credit hours--can be agreed upon. It is possible, and perhaps even likely, that critics will call for a moratorium on all efforts to measure instructional productivity until a valid and reliable output quality index can be developed. We believe this would be unwise, for two reasons. First, the kind of productivity measures proposed in this report is intended to deal primarily with changes over time rather than comparisons across institutions. It is true that an increasing focus on the quantitative aspects of productivity might trigger a "race for the bottom" in educational quality as competing institutions make increasingly larger concessions, seeking to boost the numerator and cut the denominator of the productivity fraction. Pressures on enrollments relative to budgets make this a danger whether quantitative productivity is properly mea- sured or not, but there is no doubt that an increasing emphasis on the quantita- tive elements of productivity could exacerbate the problem. Normalizing the productivity index for each segment to 1.0, as is done in Section 4.3.1, will help alleviate this danger--though of course it is always possible to manipulate data to achieve desired results. Attention to the limitations of the metric for measuring at low levels of aggregation is also important. Second, failure to agree on an economically valid and technically robust quantitative productivity measure will only increase proliferation of the weaker measures described in Chapter 2. These are even more susceptible to missing differences in quality than the method proposed here. Furthermore, failure to implement a good measure would indefinitely defer the benefits achievable from a better understanding of quantitative productivity even in the absence of quality adjustment. These considerations suggest a strategy of simply assuming that, ab- sent evidence to the contrary, the quality of outputs is not declining significantly over time. The panel believes, albeit reluctantly given our desire for evidence, that the general approach proposed has much to recommend it. To broaden the applicability of the measure developed here, additional steps should be taken. The essential idea is that effective and transparent quality assur- ance systems should be maintained to insure that output quality does not race for

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 81 the bottom (Massy, 2010). These could be based on extant accreditation systems, on the methods of academic audit being used effectively in Tennessee and certain overseas venues (Massy, Graham, and Short, 2007), or on the other quality- reviewing initiatives now being conducted at the state level. From a modeler's perspective, the approach converts a quantity that would be included in the objec- tive function if a cardinal measure were available to a "yes-no" constraint that needs only binary measurement. The binary constraint amounts to what might be called a "watchdog evaluation": remaining silent if all is well but sounding the alarm if it is not. The watchdog evaluation would be at root subjective, but based upon evidence; we return to this idea in Section 5.3.2. The approach is by no means perfect. However, it allows progress to be made in measuring the quantitative aspects of productivity while containing the risk of triggering institutional competition that results in lowering educational quality. Progress on the development of quantitative productivity measures may boost the priority for developing a serviceable quality adjustment index.

OCR for page 61
82 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION TECHNICAL APPENDIX The Trnqvist Productivity Index This Appendix briefly describes the theoretical basis and calculations for the Trnqvist productivity index used in Chapter 4. The argument follows Caves, Christensen, and Diewert (1982:1393). As noted in the text, the Trnqvist index is used by the Bureau of Labor Statistics in calculating multi-factor productivity change (cf. Bureau of Labor Statistics, 2007). The appendix text is adapted from Section 1 of Massy (2012). Where applicable, the equations are keyed to Steps 2-4 of Table 4.2. Input Indices, Distance Functions, and Productivity We define "Firm k" and "Firm l" as two enterprises whose productivity is to be compared. Standard usage takes the two to be the same organization at different time periods, but this is not a requirement of the Trnqvist theory. For example, the two could be separate enterprises operating in the same or different periods. At their Eq. 6, Caves and colleagues (1982) define the Malmquist input index for Firm k with respect to the inputs of Firm l as: (1) Dk ( yk , x l ) Qk (x l , x k ) = , Dk ( yk , x k ) where yk is an m-element vector of outputs and xk is an n-element vector of inputs for firm k. The numerator of the right-hand side, called "Firm k's input distance function with respect to the inputs of Firm l," is defined at Eq. 7 as: xl (2) D k ( y k , x l ) = max { : F k y k ( ) y }, k l k k k where Fk(yk,xk) = 0 is k's production function, y k is the vector {y2 , y3 ,... yn }, and y1 = F ( y k k , x ). Maximizing determines the "minimum Firm l input mix k required to produce Firm k's output using k's production function"--i.e., it "de- flates" kl onto k's production function. From this it can be seen that Dk (yk, xk), the denominator in (1), always must equal one because any firm can produce its own output using its own inputs. This means Q k ( x l , x k ) = D k ( y k , x l ) 1, which implies that "the input vector of firm l, xl, is `bigger' than the input vector of firm k (i.e., k, xk) from the perspective of Firm k's technology" (Caves, Christensen, and Diewert, 1982:1396). A similar argument applies to Ql (xl,xk). Suppose now that Firms k and l are in fact the same entity, observed at times t1 and t2 respectively, and Qk (xl,xk) 1. This means the technology used at time

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 83 would have required more inputs to produce the t2 outputs than does the technol- ogy at t2 (this conclusion does not require the outputs at t1 and t2 to be the same). In other words, there has been a productivity improvement. A Practical Method of Computation This is a powerful result, but the calculation cannot be done without detailed knowledge of the production function and its parameters--knowledge that is rarely if ever available. Caves and his co-authors surmount this difficulty by in- voking classic profit maximization--an assumption that is dubious when applied to higher education institutions (see below)--together with a modest technical simplification. In their words: By making use of a specialized functional form and the assumption of cost- minimizing behavior, it is possible to compute a geometric average of the two Malmquist indices and Qk(xl,xk), using only observed information on input pro- cesses and quantities. We demonstrate this fact for the case in which each firm has a translog distance function, but the properties of the two translog functions are allowed to differ substantially. In this case the geometric average of the two Malmquist indices turns out to be a Trnqvist index. (Caves, Christensen, and Diewert, 1982:1397). The technical simplification is that the production functions' cross-product parameters for inputs and outputs be equal within and across firms. The authors point out that these restrictions are not onerous because "The translog distance function is capable of providing a second-order approximation to an arbitrary distance function. Thus the technologies in the two firms can be virtually arbi- trary (to the second-order) except for the restrictions" [Caves, Christensen, and Diewert, 1982:1398]. Nonetheless, research will be needed to assess whether or not these restrictions are reasonable in the higher education context. The resulting theorem, Caves Equation (15) reproduced below, shows how one can compute the geometric average of the two input indices using only ob- servable data. This is all that's needed to estimate productivity change. k w k w lj wn l (3)1 2 1 2 1 n ln Q k ( x l , x k ) + ln Q l ( x l , x k ) = 2 j =1 [wj w xk k n + w l x l ][ln x l j j ]. - ln x k The new symbols wk and wl are vectors of input prices and wk xk is total ex- penditure, the dot-product of price and quantity. This means the fractions repre- sent shares of input expenditure. Hence the right-hand side defines the log of a Trnqvist index, denoted by: namely, the geometric average of physical inputs using expenditure shares as weights.

OCR for page 61
84 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION The quantity Q(wl,wk,xl,xk) is shown for the transitions from Period 1 to Period 2 and Period 2 to Period 3, in the second-to-last line of Table 4.2 (the first line of step 4). Step 2 of Table 4.2 shows the inputs for the quantity-change (i.e., the last) term in the right-hand side of (3): i.e., in x lj . Step 3 shows the weights and the resulting geometric averages of the quantity changes. Notice that the weights are themselves an average of the weights in the two periods being considered. The Overall Productivity Measure Section 4 of Caves, Christensen, and Diewert (1982:1401), points out that there are "two natural approaches" to the measurement of productivity changes: differences in maximum output conditional on a given level of inputs ("output- based" indices) and those based on minimum input requirements conditional on a given level of output ("input-based" indices). Furthermore, the two approaches "differ from each other by a factor that reflects the returns to scale of the produc- tion structure." Without going into the details, it is intuitively reasonable that a geometric average of the output-based and input-based indices represents a good overall measure of productivity.13 Therefore, the desired productivity index is the ratio of the output-based and input-based Trnqvist indices. Changes in pro- ductivity are obtained by taking the ratios of the indices in successive periods. Step 4 of Table 4.2 performs these final calculations. The first line is the productivity index itself: the ratio of the output to the input indices. The second line shows productivity change: the ratio of the indices in the two successive periods, minus one. Applying the Index to Nonprofit Enterprises The proof of optimality for the Trnqvist index depends on the assumption of profit maximization. As noted above, this assumption is dubious when applied to traditional universities (it applies perfectly well to for-profit universities, how- ever). This leads to two questions that need to be addressed by further research: (1) Will application of the index to nonprofits produce misleading results? (2) What modifications to the Trnqvist (or perhaps an entirely different approach) will likely be better than the traditional index? Regarding the first question, we note that a lack of optimality is not equiva- lent to a lack of efficacy. Many algorithms and measures are used, in economics 13Adjustments for decreasing and increasing returns to scale are described in later sections of the Caves paper. They are interesting but, we believe, not of particular concern to productivity measure- ment in colleges and universities. While scale economies in higher education are intuitively plausible, it appears that significant size increases are likely to trigger institutional responses (e.g., scope or support service increases) that tend to drive up costs (cf., Brinkman, 1990:120). Our model does not adjust for scale effects.

OCR for page 61
ADVANCING THE CONCEPTUAL FRAMEWORK 85 and elsewhere, whose optimality cannot be proved or that have even been shown to be suboptimal. The question is an open one which calls for additional research. Regarding the second question, Massy (2012) has proposed a modification to Equation (3) to achieve optimality in the nonprofit case. Still, additional research will be needed to determine how the required new parameters can be estimated and to identify the conditions under which the new model produces results that differ materially from the traditional one. The question of material differences will, in turn, shed light on the answer to question (1).

OCR for page 61