Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 145
Appendix B
Methods for Measuring Comparative
Quality and Cost Developed by the National
Center for Academic Transformation
The National Center for Academic Transformation (NCAT) is an indepen-
dent nonprofit organization that describes itself as "dedicated to the effective use
of information technology to improve student learning outcomes and reduce the
cost of higher education."1 NCAT works with institutions to analyze their instruc-
tional models and identify ways to efficiently leverage technology, faculty, gradu-
ate students, peer interactions, and other learning resources to improve quality
and efficiency. In the NCAT model, students are transformed from "passive note
takers" in a standard lecture format to active participants in their own learning,
leveraging a set of tools and resources. Based on a review of the participating
institutions, NCAT has identified six redesign models that vary in format, student
experience, and use of technology.
Course redesigns focus primarily, but not exclusively, on large-enrollment,
introductory courses in multiple disciplines, including 16 in the humanities (de-
velopmental reading and writing, English composition, fine arts, history, music,
Spanish, literature, and women's studies), 60 in quantitative subjects (develop-
mental and college-level mathematics, statistics, and computing), 23 in the social
sciences (political science, economics, psychology, and sociology), 15 in the
natural sciences (anatomy and physiology, astronomy, biology, chemistry, and
geology) and 6 in professional studies (accounting, business, education, engineer-
ing, and nursing).
1This appendix draws heavily from the NCAT Web site: http://www.thencat.org/ [November 2011].
145
OCR for page 146
146 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
QUALITY
NCAT requires each participating institution to conduct a rigorous evalua-
tion of the impact of the redesign on learning outcomes as measured by student
performance and achievement. National experts have provided consultation and
oversight regarding the assessment of learning outcomes to ensure that the re-
sults are as reliable and valid as possible. To date, results show improved student
learning in 72 percent of the redesigns, with the remaining 28 percent showing
learning outcomes equivalent to traditional formats. Other qualitative outcomes
achieved in the course redesigns include increased course-completion rates, im-
proved retention, better student attitudes toward the subject matter, increased
student performance in downstream courses and increased student satisfaction
with the mode of instruction.
All NCAT redesign projects compare student learning outcomes in the tra-
ditional format with those achieved in the redesigned format. This is done by
either running parallel sections of the course in the two formats or comparing
baseline data from an offering of the traditional course to a later offering of the
redesigned course, looking at differences in outcomes from before and after. The
four measurement methods used to assess student learning are Comparisons of
Common Final Exams, Comparisons of Common Content Items Selected from
Exams, Comparisons of Pre- and Post-tests, and Comparisons of Student Work
Using Common Rubrics.
COST
NCAT requires each participating institution to establish a team of faculty
and staff to conduct the redesign. Each team develops a detailed cost analysis of
both the traditional and the redesigned course formats. All 120 course redesigns
have reduced costs--by 37 percent, on average, with a range of 9 percent to 77
percent. The 120 redesigned courses have affected more than 160,000 students
nationwide and produced a savings of about $9.5 million each year.
Each team analyzes and documents "before and after" course costs using
activity-based costing. NCAT developed a spreadsheet-based course planning tool
(CPT) that allows institutions to (1) determine all personnel costs; (2) identify
the tasks associated with preparing and offering the course in the traditional for-
mat and determine how much time each type of personnel spends on each of the
tasks; and (3) identify the tasks associated with preparing and offering the course
in the redesigned format, and determine how much time each type of personnel
spends on each of the tasks. The CPT then automatically converts the data to a
comparable cost-per-student measure. At the beginning of each project, baseline
cost data for the traditional course and projected redesigned course costs are col-
lected; actual redesigned course costs are collected at the end.
Completing the CPT allows faculty members to consider changes in specific
OCR for page 147
APPENDIX B 147
instructional tasks, make decisions about how to use technology (or not) for
specific tasks, visualize duplicative or unnecessary effort and complete a cost/
benefit analysis regarding the right type of personnel for each instructional task.
The CPT consists of four worksheets: Instructional Costs per Hour, used to deter-
mine all personnel costs involved in the course and to express each kind of cost
as an hourly rate; Traditional Course Activities and Cost, used to determine the
activities involved in and the costs of preparing and delivering the course in its
traditional format; Redesigned Course Activities and Cost, used to determine the
activities involved in preparing and delivering the course in its redesigned format
when it is fully operational; and Annual Cost Comparison, used to compare the
annual costs of the traditional course and the redesigned course.
Among the assumptions of the NCAT Model are the following:
Developmental costs are not included. NCAT's model compares the before
costs (current/historical/ traditional) and the after costs (what the course will cost
when it is fully operational)--i.e., it asks the team to plan what the redesigned
course will look like at the end of the developmental process. It does not include
the up-front developmental costs of either the traditional or the redesigned course.
Institution-wide support services, administrative overhead, infrastructure
and equipment costs are not included. The assumption is that these costs are
constant--are part of the campus environment--for both the traditional and
redesigned courses. Campus networking, site licenses for course management
systems and desktop PCs for faculty, for example, are part of the campus-wide IT
environment. Costs that are particular to the course are included in "other costs."
EXAMPLES (CASE STUDIES)
Research University: Louisiana State University
Louisiana State University (LSU) redesigned College Algebra, a three-credit
course enrolling 4,900 students annually. The university moved all structured
learning activity to a computer lab where students work with an instructional
software package--which includes interactive tutorials, computational exer-
cises, videos, practice exercises, and online quizzes--and receive just-in-time
assistance from instructors and undergraduate assistants. Instructors also meet
students in a once-a-week focus group which builds community among students
and instructors.
The redesign reduced costs by serving the same number of students with
half of the personnel used in the traditional model. Section size stayed at 40-44
students, but the number of class meetings each week was reduced from three to
one. The redesigned format allows one instructor to teach twice as many students
as in the traditional format without increasing class size and without increasing
workload. In the traditional format, each instructor taught one three-day-a-week
OCR for page 148
148 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
section with 44 students. In the redesigned format, that same instructor teaches
two sections of 44 students and spends four hours tutoring in the lab. This can be
accomplished because the class only meets once a week and because no hand-
grading is required. While the cost of adding tutors in the computer lab as well
as increased time for coordination and systems administration reduced the net
savings, the redesign reduced the cost-per-student from $121 to $78.
Learning outcomes were measured by comparing medians on a common final
exam. Final exam medians for traditional fall sections ranged from 70 percent
to 76 percent. After redesign, the final exam median in fall 2006 was 78 percent,
the highest ever achieved. In the traditional format, final exams were graded by
individual instructors and partial credit was allowed. In the redesigned format,
final exams were group-graded, which means that grading was more consistent
from one section to the next, and partial credit was not allowed, yet exam medi-
ans were higher. The success rate (grades of C or better) for College Algebra in
the five years prior to the redesign averaged 64 percent. In fall 2006, after full
implementation of the redesign, students had a success rate of 75 percent.
Comprehensive Four-Year Institution:
The University of Southern Mississippi
The University of Southern Mississippi (USM) redesigned World Literature,
a course enrolling more than 1,000 students each term, in order to eliminate
course drift and inconsistent student learning experiences. The traditional course
was offered in 16 sections of about 65 students each: 8 sections taught by full-
time faculty and 8 by adjuncts. The redesign placed all students in a coherent
single online section and replaced the passive lecture environment with media-
enriched presentations that required active student engagement. A course coordi-
nator directed the team teaching of four faculty members, each of whom taught
his or her area of expertise of four weeks, and four graduate teaching assistant
(GTA) graders. The faculty team offered course content through a combination
of live lectures with optional attendance and required Web-delivered, media- and
resource-enhanced presentations.
USM reduced the number of sections from 30 to 2 and increased the number
of students in each section from 65 to 1,000. These changes enabled the university
to reduce the number of faculty teaching the course from 16 to the equivalent of
2 full-time faculty and 4 GTAs. USM eliminated adjuncts completely. The course
is now taught 100 percent by full-time faculty supported by GTAs for writing
assignment grading. By making these changes, six full-time faculty were freed
to teach other courses, and the funds previously used to hire adjuncts were made
available for a variety of academic enhancements in the department. The cost-
per-student was reduced from $70 to $31.
Student performance improved on multiple dimensions in the redesign.
Grades on weekly quizzes with common content showed an increase in grades
OCR for page 149
APPENDIX B 149
of C or better from 68 percent in the traditional course to 88 percent in the re-
designed course. Writing scores of C or better increased from 61 percent to 77
percent. Essay scores increased in from 7.11 in the traditional mode to 8.10 in
the redesigned format. In the traditional version of the course, faculty-taught
sections typically retained about 75 percent of their students, while adjunct- and
GTA-taught sections retained 85 percent. In the fall 2003 semester of full imple-
mentation of the redesign, retention was 87 percent, with all students being taught
solely by full-time faculty. At the same time, the rate of D and F grades dropped
from 37 percent to 27 percent in the redesigned course.
Private, Liberal Arts Institution: Fairfield University
Fairfield University, a comprehensive Jesuit university with 3,100 full-time
undergraduates, redesigned its general biology course to improve the quality of
student learning and to reduce its cost. The traditional biology course was taught
in a multiple-section model, with 35-40 students per section, and met three times
per week with a three-hour lab. Four faculty members provided lectures; addi-
tional faculty and professionals were needed to staff the labs. The cost of offering
the traditional course to 260 students annually was $131,560, which translates to
a per-student cost of $506.
Significant cost savings were realized from reducing faculty time in three
major areas: (1) materials development for lectures; (2) out-of-class course meet-
ings; and (3) in-class lectures and labs. The number of faculty needed to teach the
course declined from seven to four. Faculty time was reconfigured to support a
division of teaching responsibilities so that the four faculty members now teach
from their areas of expertise. Faculty time devoted to this course decreased from
1,550 hours to 1,063 hours. Consolidation of the seven lecture sections into two
in the redesigned course and the introduction of computer-based modules in the
lecture and laboratory resulted in a cost-per-student reduction from $506 to $350.
Students in the redesigned course performed significantly better on bench-
mark exam questions. The questions on the new exams were designed to test
higher order thinking and allow students to synthesize material from the basic
concepts.
In addition, specific exam questions incorporated in the second year genet-
ics course (required of all biology majors) were used to measure the retention of
key concepts and compare the performance of traditional students and redesign
students. Students from the redesigned course performed significantly better on
this set of questions than did students from the traditional course (88 percent cor-
rect vs. 79 percent correct, respectively). The DFW rate dropped: only 3 percent
of the students in the redesign dropped the course compared to an average of 8
percent in previous years. The number of students who decided to enroll in the
second semester of the course approached 85 percent compared to less than 75
percent in previous years.
OCR for page 150
150 IMPROVING MEASUREMENT OF PRODUCTIVITY IN HIGHER EDUCATION
Community College: Tallahassee Community College
Tallahassee Community College (TCC) redesigned College Composition,
a required course serving approximately 3,000 students annually. The tradi-
tional format, which combined lecture and writing activities in sections of 30
students each, made it difficult to address individual needs. The redesign used
technology to provide diagnostic assessments resulting in individualized learn-
ing plans; interactive tutorials in grammar, mechanics, reading comprehension,
and basic research skills; online tutorials for feedback on written assignments;
follow-up assessments; and discussion boards to facilitate the development of
learning communities. The classroom was restructured to include a wide range
of learner- centered writing activities that fostered collaboration, proficiency, and
higher levels of thinking. By shifting many of the basic instructional activities
to technology, faculty could focus the classroom portion of the course on the
writing process.
TCC reduced the number of full-time faculty involved in teaching the course
from 32 to 8 and substituted less expensive adjunct faculty without sacrificing
quality and consistency. In the traditional course, full-time faculty taught 70
percent of the course, and adjuncts taught 30 percent. In the redesigned course,
full-time faculty teach 33 percent of the course, and adjuncts teach 67 percent.
Overall, the cost-per-student was reduced from $252 to $145, a savings of 43
percent. Full-time faculty were freed to teach second-level courses where finding
adjuncts was much more difficult.
Five final out-of-class essays were selected randomly from the traditional and
redesigned sections and were graded by an independent group of faculty using an
established holistic scoring rubric. Students in the redesigned composition course
scored significantly higher (p = 0.03) and had an average score of 8.35 compared
to 7.32 for the students in the traditional course. Students in redesigned sections
had a 62 percent success rate (grades of C or better) compared with 56 percent in
traditional sections, representing a 13.6 percent decrease in the DFW rate. Suc-
cess rates of students in the second-level English course increased (79.3 percent
success for redesigned compared to 76.1 percent for traditional).
