Click for next page ( 2


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary of Major Conclusions and Recommendations Since World War II, the financing of higher education has changed through the expansion of the role of need-based student aid. Scholarships for students who were both meritorious and needy have always been avail- able. The introduction of need-tested aid in the l950s, however, and the major federal support and regulation of student aid distribution that fol- lowed have resulted in what is today a complex aid system. Postsecondary students can receive financial aid from the federal gov- ernment, states, postsecondary institutions, other private organizations, or a combination of these sources. Direct aid to students is dominated, however, by federal aid programs (grants, loans, and work-study programs covered by Title IV of the Higher Education Act of 1965~. In the academic year 1989-90, total available student aid equaled $27.3 billion, of which gener- ally available federal aid (Title IV programs only) equaled $19.0 billion. Of the students enrolled full-time in postsecondary schools in 1989-90, 57 percent received aid from at least one source, and federal funding went to 44 percent of the students enrolled. The large expenditures and the wide- spread use of student aid create great uncertainty concerning the adequacy of the system for delivering federal student financial aid. Especially troublesome for the U.S. Department of Education are the recurring questions about the accuracy with which the share of educational expenses to be borne by the family is determined and the effectiveness of the systems of accountability mandated by federal regulations. To address these questions, the U.S. Department of Education requested that the Na- tional Academy of Sciences convene a panel to study quality control in the

OCR for page 1
2 QUALITY IN STUDENT FINANCIAL AID PROGRAMS Title IV student financial aid programs. In particular, the panel was asked to study (1) the quality control practices employed by the Department of Education to measure the accuracy of awards provided to students and (2) the methods used by program managers, based on this information, to re- duce errors. The panel's report goes beyond a simple assessment of quality control, however. It also considers possible use of this information to pre- vent errors through appropriate corrective actions. This summary presents the panel's major conclusions and recommenda- tions. Details of the analysis and other suggestions for improvement are provided in subsequent chapters. "ERROR" IN STUDENT FINANCIAL AID PROGRAMS Types of Error A study of the accuracy of awards made during the 1988-89 academic year found that, of the approximately $15.4 billion in aid distributed, almost 11 percent was awarded in error (Price Waterhouse, 1990~. Although inac- curacies are possible in this and other estimates of program error, there is little doubt that the errors, whatever their exact total, are considerable. At each step of the delivery process (applying, determining eligibility, calculat- ing awards, disbursing awards, and monitoring educational progress) errors can be made. Efforts have been focused on detecting errors in examined files and correcting them, but too little has been done to prevent error- changing the procedures that allowed the errors to occur. Although reducing payment error (defined below) is an important as- pect of quality particularly in programs that disburse public funds it is only one of many dimensions of quality. Its pervasiveness, however, makes an understanding of payment error prerequisite to an understanding of "quality control" as practiced by the Department of Education. In principle, two kinds of payment error occur in student financial aid programs: (1) errors of overpayment and (2) errors of underpayment. Over- payments can be subdivided into (1) excess payments to eligible recipients and (2) all payments to ineligibles. In parallel fashion, underpayments comprise (1) insufficient payments to eligibles and (2) the lack of payments to those mistakenly classified as ineligible. Errors can be further classified as substantive or technical. Errors in the information used to determine eligibility for student financial aid as well as errors made in the calculation of the award based on that informa- tion are called substantive errors. Technical errors occur when a legally required document is missing from a student's file. Depending on other factors, inclusion of the document may or may not have rendered the stu- dent eligible. The lack of such a document makes the student categorically

OCR for page 1
SUMMARY 3 - ineligible for financial aid. Examples of technical errors include failure to document satisfactory educational progress, failure to indicate registration for Selective Service, failure to include a statement of educational purpose, and failure to include a financial aid transcript. A third distinction deals with the originating source of the error whether it is the institution, data processing contractor, or the student. The three sources of error are the primary focus of quality control efforts. Student Error Student error may occur because of failure to use correct data, inaccu- racies in forecasting data, and/or complexities of the application and in- structions. For errors that are not made deliberately, several "student er- rors" could not be corrected even with additional care on the part of the applicant. For example, some problems arise because, to apply for aid from some states and institutions, the application must be completed before the applicant completes his or her federal income tax form. Thus, information provided on dependency status, taxes, and adjusted gross income is gener- ally an estimate of the data that will be reported in the tax return. Similarly, household size and the number of dependents in postsecondary school are necessarily projections for the following academic year; as circumstances change from current plans, this information will turn out to have been "in error." The panel recommends simplifying the application procedure to include fewer items subject to error. Recommendation 5-1: For applicants who have filed a tax return in either of the prior two tax years, the information used to complete the application and determine the award should be based on the most re- cently filed of these income tax returns. When the earlier tax year is used, updated information should be required as soon as a new tax return is filed. The updated information should not be used to change the award before the next term. Further, "household size" and "num- ber in postsecondary school" should be based on the situation as of the application date. Institutional Error Institutional errors occur when institutions inaccurately process appli- cations or do not adhere to established guidelines. The panel found that technical errors, such as a missing statement of educational purpose or financial aid transcript, have a large dollar impact on measured error, since their absence renders the entire award erroneous. Such cases may or may not otherwise be eligible for financial aid. The General Accounting Office

OCR for page 1
4 QUALITY IN STUDENT FINANCIAL AID PROGRAMS (GAO) requested from the Department of Education that some student files be analyzed for substantive errors only (technical errors due to missing documentation were to be ignored). The analysis found that the estimated dollar value of errors dropped by about a third, and the percentage of cases with "institutional error" dropped by half (General Accounting Office, 1985~. The panel believes that there is a need to refocus the effort used to detect technical errors. Recommendation 5-2: The Department of Education should determine which, if any, of the administrative requirements that can lead to "tech- nical error" are useful surrogates for variables that deal directly with program goals but are difficult to measure. Administrative require- ments that are useful in this sense should be retained, but others should be eliminated. Failure to Estimate Error Leading to Nonawards According to the Department of Education's data for the 1982-83 aca- demic year, 42 percent of Title IV aid recipients received overawards and 21 percent received underawards. Typically, overaward has been estimated as much larger than underaward whenever such a classification is made. There is, however, a serious defect in all the major studies of error in student financial aid programs. The various surveys analyze only cases of awards; there is no coverage of students to whom financial aid was totally denied. It is thus not surprising that the studies find more overpayment error than underpayment error. On a more basic level, a major concern of the panel is an underemphasis by the department on measuring progress toward the program goals of access and choice in postsecondary education. Potential recipients mistakenly denied financial aid despite actual eligibility surely have restricted access and choice. Recommendation 5-5: The Department of Education should improve estimates of "error" by including estimates of the coverage of student financial aid programs, that is, ascertaining the frequency with which eligible applicants are mistakenly deniedfinancial aid and the underaward amounts associated therewith. CURRENT QUALITY CONTROL PROCEDURES The Department of Education's current quality control process is de- scribed as focusing on prevention, inspection, and oversight. Prevention activities include training provided by the department to data entry contrac- tors, financial aid administrators, and other institutional officers, lenders, accrediting agencies, state scholarship and guaranty agencies, and others in

OCR for page 1
SUMMARY s the student financial aid community. Other activities that are carried out in an effort to prevent award error entail editing and verifying student data. Inspection consists of after-the-fact monitoring activities audits and program reviews. Such activities are often developed in the belief that they help to ensure compliance because the possibility of penalties and/or sanc- tions acts as a deterrent. Oversight consists of periodic studies of various program areas within the Department of Education. The studies are conducted by the department's Office of Inspector General or the GAO and focus on review of procedures for monitoring compliance with various requirements and the management of those activities. The panel found information in the various studies to be very useful in its deliberations. Edits and \7erification of Applicant Data Applicant-supplied data undergo several levels of editing and review. First, edits are performed by data entry contractors. During the edits, ques- tionable data are identified and "highlighted" on the Student Aid Report (SARI that is mailed to the student. The student is asked to review and correct all data and, if corrected, return the SAR for reprocessing. Verification of applicant-reported data items by the institution is the Department of Education's primary tool in its efforts to control applicant error. Verification activities place a major responsibility and burden upon the schools and, as a result, are very controversial. Each school is required to verify key elements of the applicant's record for all records flagged by the department's selection methodology (in recent years, up to 30 percent of an institution's student aid applications, but the 1992 reauthorization of the Higher Education Act allows the Secretary of Education to mandate up to 100 percent verification). The department describes the current verification system as an attempt to strike a balance between error reduction and the burden imposed on institutions. Efforts to move toward 100 percent verifi- cation of applicant data have been tempered by institutional lobbying to stress the excessive burden that institutions must endure to correct errors not of their making. Applicant records are selected for verification through a complex se- quence of statistical procedures intended to target the most error-prone ap- plicants. The verification system has been studied and modified over the years, but an extensive review by the panel yielded no evidence that major advances have been made in responding to the major criticisms of the sys- tem. The cost-benefit of the system remains a question. Although some cases of fraud or abuse undoubtedly will arise in a program of this size, previous studies have not uncovered any widespread problems of fraud by students. The panel recognizes that the Department of

OCR for page 1
6 QUALITY IN STUDENT FINANCIAL AID PROGRAMS Education believes verification is necessary to fulfill its congressional man- date, yet the panel concludes that the verification process as currently used is unfairly burdensome for the overwhelming majority of well-intentioned people working within the system. In using any other technique, the em- phasis should be to move from the current focus on catching mistakes to also providing information for future preventive actions. Recommendation 4-1: A research effort should be initiated to assist the Department of Education's contractor in developing better criteria for selecting records to be verified and, more important, identifying oppor- tunities for earlier removal of errors through instructions, form im- provements, and SAR highlighting strategies. Current andfuture changes to the verification selection methodology should be carefully documented. Audits and Program Reviews The Department of Education has the responsibility to ensure that the approximately 8,000 participating postsecondary institutions administer stu- dent financial aid programs in compliance with federal law and regulations. The department executes this responsibility through two major retrospective activities: (1) requiring schools to contract for compliance audits by an independent auditor and (2) conducting program reviews through its re gional offices. The Title IV student financial assistance audit and program review pro- cesses are designed to determine the accuracy with which an institution administers its student financial aid programs. Specifically, the Title IV audit objectives are to measure the reliability of an institution's financial data, the adequacy of its internal control systems, and its compliance with program regulations. The objectives of a program review of an institution are to determine compliance with program regulations and to evaluate fi- nancial and administrative capabilities. The panel examined the operations of these retrospective activities as they relate to the problem of error in the delivery of student financial aid. The rules and regulations governing student eligibility for financial aid are complex. The panel questioned whether and, if so, to what extent the complexities themselves are significant sources of error. The current moni- toring and compliance activities and data bases do not address this issue. They are designed to assess an institution's absolute performance with re- spect to the accurate administration of student financial aid programs, to count the occurrences of error, and to impose sanctions on the institution based on error. The audit and program review processes used by the Department of Education to enforce quality standards are marked by considerable duplica

OCR for page 1
SUMMARY lion, ineffectiveness, and wasted effort. The independent audit of schools checks for internal controls and compliance with program regulations in a way that essentially duplicates parts of the program review process. Such checks are more effectively performed by knowledgeable reviewers and should be fully incorporated within the program review process. Inspection is a deterrent to poor work only when it is possible to do a better job. To maximize an institution's potential, the audit and review processes must be better integrated and redesigned so as to have a sharper focus on addressing meaningful measures of quality and quality improve- ment rather than the current concept of measuring compliance with an unre- alistic "zero defect" standard. To expend resources where risk is greatest, for example, data from past audits and reviews should be used to improve the methods for selecting institutions and the methods for sampling records at those institutions. Audit and review activities are necessary if the Department of Educa- tion is to fulfill its responsibility to ensure that program participation is limited to those institutions that are willing and able to operate responsibly, in accordance with program goals and expectations. Since factors such as changing economic conditions and student financial aid personnel affect an institution's ability to maintain desired levels of quality, "problem" institu- tions will appear sporadically and must be dealt with promptly and effi- ciently. Audits provide reasonable promptness, especially now that they are an annual requirement. Program reviews are relatively infrequent, however, because they must be performed under departmental budgetary and person- nel ceilings. Yet, the review process has the greater potential to be proactive and to provide the useful instruction and technical assistance that promote quality and help to build a sense of partnership between the institutions and the department. Further, the match between the capabilities of the inspector and the inspection function warrants careful attention. More information is needed about the reliability of independent auditor findings. While the audit func- tion can provide timely information, program review is the only system of institutional quality checks that is entirely in the control of the Department of Education. The panel believes that program reviews have the greatest potential for reliability and objectivity. The panel also believes that the department needs to revamp the audit and program review systems in order to support the gate-keeping function, provide technical assistance to institu- tions, and provide useful data to policymakers. Recommendation 4-3: The Department of Education should redesign the current system of program reviews and independent audits. Pro gram reviews should focus on compliance as part of an overall quality improvement program. Checks on institutional compliance and internal

OCR for page 1
8 QUALITY IN STUDENT FINANCIAL AID PROGRAMS controls should be performed only in program reviews and audits should focus only on financial attestation. The Department of Education should also develop, test, and imple- ment methods to systematize and standardize the program review pro- cess. The department should not interpret the 1992 reauthorization of the Higher Education Act as a requirement to review every school ac- cording to a fixed schedule. Risk-based statistical methods should be used to identify problem schools for more frequent reviews, and other schools should be selected randomly at a nominal rate that would fulfill the necessary '~gate-keeping" functions. The department must improve the evaluative feedback and technical assistance provided to institu- tions during reviews. At the same time, the review should be used to accumulate data that provide the department with a continuous over- view of error rates, compliance levels, and other information of signifi- cance for management in making policy. THE APPLICANT'S VIEW As the ultimate customer of financial aid services and as the provider of the information that puts the overwhelming inspection processes into mo- tion, the applicant's or potential applicant's experience must weigh heavily in decisions concerning process improvement. The panel focused on two areas of importance related to the applicant's need for quality: early aware- ness and the application process. Early Awareness Very few ongoing data sets are available from which to obtain infoIma- tion about students who have not applied for financial aid. The Department of Education maintains administrative records on only those students apply- ing for financial aid, and studies of error have captured data on aid recipi- ents only. Better use of the National Postsecondary Student Aid Study and coordination with the National Center for Education Statistics could help. Recommendation 5-7: The panel commends the Department of Educa- tion for its recent efforts to improve early awareness of federal finan- cial aid programs, such as providing financial aid software to high schools. The panel recommends increasing those efforts. Studies of the effectiveness of the efforts and users' reactions to the timeliness, accu- racy, usefulness, and clarity of outreach and counseling services are needed.

OCR for page 1
SUMMARY 9 Recommendation 5-8: The Department of Education should gather data on the reasons for nonutilization of student financial aid by potential recipients. Consideration should be given to ways of estimating the number of potentially qualified applicants who are discouraged for one reason or another from applying for financial aid in the first place. Further, the department should consider ways of estimating the number of potential students who do not even attempt to enter a postsecondary school because of their ignorance of available financial aid. Also, data on the knowledge of aid should be collected from students and their families before the student finishes secondary school. Such data could be obtained in a variety of ways, including ongoing national surveys. The resulting information will be important in devising a program for reaching those who are eligible but do not apply. The Application Process Students who wish to apply for federal student financial aid can make application in one of three ways: complete the Application for Student Financial Aid (AFSA) produced and distributed by the Department of Edu- cation; complete an application form produced by one of the multiple data entry contractors who process applications; or enter application data through the Electronic Data Exchange application process. Panel members were concerned that the current design of these application "forms" could cause confusion among applicants and financial aid officers, lead to inadvertent errors in financial aid decisions, and deter some students from applying at all. Such up-front errors could then lead the department to perceive a need for more inspection. Testimony before the panel, as well as the panel's own deliberations, suggest that the application forms for student financial aid might be difficult to complete, especially for the typical lower income ap- plicant, and that the design of the for might complicate the work of stu- dent financial aid officers who advise students about financial aid, check the accuracy of applicants' responses on applications forms, and attempt to correct erroneous data. As a result, the panel undertook a study of the federally produced AFSA and the Electronic Data Exchange process. Based on that analysis, the panel makes two recommendations: Recommendation 5-9: The complexity of the forms, instructions, and information booklets leads to excessive burden for applicants and is a cause of error. Thus, the Department of Education should consult ex- perts in form and question development, such as those found at cogni- tive research laboratories, to aid in its efforts to improve the applica- tion materials.

OCR for page 1
10 QUALITY IN STUDENT FINANCIAL AID PROGRAMS Recommendation 5-10: The Department of Education should continue to improve and expand the availability of its electronic data exchanges, making sure to address the issue of appropriate balance between the potential for improved data and additional burdens that might be placed on the schools. CONCERNS OF POSTSECONDARY INSTITUTIONS Postsecondary institutions are increasingly burdened by the administra- tive complexities associated with federal student aid programs. From an original promise to administer the eligibility requirements centrally, the gov- ernment has moved to an increasingly complex financial statement form, which must be verified by school personnel. The addition of information unrelated to need, such as Selective Service status, is collected through this form, which has grown from two pages to four. The Department of Education requires institutions to collect signed statements regarding nonuse of illegal drugs and educational purpose, but the certifica- tion is not of direct use to the institution because it neither prevents drug use nor ensures that the student has a valid educational purpose. Under the current system, institutions bear a considerable amount of the burden in obtaining this type of certification. In addition, hard copies of applications and related "underwriting" materials must be physically retained by the institution for long periods of time, which creates the need for expensive warehousing in financial aid offices. There is, however, no reason why the depository cannot be changed so that the department can take on an equi- table amount of responsibility. Congress has made some progress in initiating changes along these lines. The 1992 reauthorization of the Higher Education Act calls for in- creased utilization of verification mechanisms in which students' reported information can be verified through record matching, by using either an automated or other system. These provisions include requiring that data base matches with the Selective Service be made and that the Social Secu- rity Number of all aid recipients be verified. With this advancement toward electronic data transfer of student financial aid information, the burden im- posed by the collection of student certifications could be greatly reduced. Recommendation 5-11: The Department of Education should increase its efforts to remove unnecessary burden from students and institutions by further development of automated data matches whenever possible. A VISION FOR A TOTAL QUALITY APPROACH Many industrial and service sector organizations have recognized that a system based on prevention of errors rather than inspection is necessary to

OCR for page 1
SUMMARY 1 1 achieve quality improvement and efficient operation. Such systems utilize a total quality approach. A total quality approach requires a commitment by the highest levels of management to lead quality improvement efforts rather than merely assigning responsibility for quality assurance to an orga- nizational unit. It also requires a focus on the "customer" and having suppliers and customers work together to define and solve problems. It requires process simplification to reduce waste, remove redundant efforts, and eliminate time-consuming and non-value-adding steps. It requires data collection and analysis that are designed for fact-based decision making and problem solving. These strategies are as valid in the Department of Education and other governmental agencies as they are in private industry. A total quality ap- proach to administering student financial aid would seek not only to im- prove action taken at each step of the process, but also seek to continuously improve important outcomes expected of the entire process, such as accu- racy of awards and accessibility of aid for eligible students. Additionally, a total quality approach seeks improvements that may eliminate mistakes be- fore they are made and that strives to lower costs, provide faster processing, and reduce red tape. Such an approach brings greater value to all the customers and stakeholders the Department of Education, Congress, postsecondary institutions, students, parents, and taxpayers. A NEW STRATEGY FOR THE DEPARTMENT OF EDUCATION The Department of Education's old strategies for ensuring quality are characterized by a centralization of authority, reliance on after-the-fact in- spections to bring about quality, and burdensome sets of frequently changed regulations that focus on compliance with process requirements. Under the old strategies, the student financial aid delivery system is burdened with a multiplicity of process checkpoints that add to the cost of running the sys- tem for schools and for the government. They also contribute to a tendency for the relationship between customers and suppliers in the system to be perceived as confrontational. The deficiencies of these strategies have led some within the department to seek new strategies. One of the difficulties in developing new strategies to improve quality is conceptual. In the absence of a statutory definition of quality, regulatory efforts fall into the trap of having no margin for error. As reported to the panel, the Department of Education has been operating with a "zero error" standard that is unattainable in practice and therefore not effective in pro- moting quality improvements. Departmental staff reported to the panel on efforts to develop new strategies that are characterized by delegation of authority and responsibility, the "ownership" of performance measures by

OCR for page 1
2 QUALITY IN STUDENT FINANCIAL AID PROGRAMS those who are performing the functions of the system, and a deregulation of the aid delivery process that empowers an institution to determine how best to accomplish program objectives. The new strategies, as found in the Institutional Quality Control (IQC) Pilot Project, focus accountability on results rather than process and emphasize at participating institutions the continuous collection of data leading to quality improvement. Foregoing the usual imposition of external quotas, these strategies are intended to form proactive partnerships rather than confrontational relationships and to focus on service improvements for the end customers. At the same time, they are intended to meet the need to assess overall system performance by measuring quality on an ongoing basis. With this experiment, the Depart- ment of Education hopes to reduce the burden on educational institutions, encourage the development of innovative management approaches, improve service to students, and reduce error in the delivery of Title IV financial assistance. Support of the IQC Project Participation in the IQC Pilot Project is voluntary and assumes that the commitment to ensure data quality already exists in participating schools. The Department of Education has many positive activities associated with efforts to support IQC institutions, including training programs for new and continuing institutions, providing quality control software, recognition and awards processes, publishing a newsletter, state and regional meetings, and providing technical assistance from departmental and contracted personnel. The IQC staff within the department is continually updating its support mechanisms to be able to offer more help to the institutions. The panel examined the IQC project at great length and gave much consideration to its future. The panel had many questions about the effec- tiveness of the project, but too little objective data to answer most of them. The IQC project offers a chance for the Department of Education to be more proactive, but it may not be making the best use of the chance. Possibly, the IQC process is overwhelming. Perhaps the institutions need more advice on the quantitative aspects of the process. Many institu- tions will have someone outside the institution's aid office who is fairly knowledgeable about such quantitative methods and could learn more about the aid process as he or she goes along. Other institutions may have to ask for outside help. Statistical expertise also needs to be developed at the Department of Education to generate the appropriate statistical information from the IQC institutions and share the aggregate results with everyone. Recommendation 8-3: Institutions in the IQC pilot program should have more at-hand statistical support, and institutions volunteering to

OCR for page 1
SUMMARY 13 participate in IQC activities should be required to show access to sta- tistical expertise in sampling and data analysis as a condition of being in the program. To some extent this can be accomplished by improving the training of internal staff. Adding training material on basic statisti- cal skills to the training program could help, but more effective strate- gies might include subsidizing locally obtained coursework for finan- cial aid office staff or developing cooperative agreements with local quantitative departments to provide ongoing assistance. The Depart- ment of Education should assist the IQC institutions in these efforts by helping develop the cooperative ties and providing monetary support. Recommendation 8-4: The IQC program should be retained and given the support of top management. However, when the common-cause regulatory burdens imposed by audits and reviews are eliminated (for details see Chapters 4 and 5), the incentives to join the pilot project will change. Thus, the following improvements should be considered: Incentives for institutions to join the IQC should be reexamined with the aim of encouraging schools with known administrative difficul- ties to seek help from Department of Education staff. The department's measure of success in the IQC program should be reexamined with the aim of going beyond measuring retention in the IQC program to identifying ways to improve processes. More use should be made of IQC data. For example, the Depart- ment of Education should inform all institutions about the progress of the IQC program and suggest use of measurements and practices found to be successful in addressing specific concerns about quality. FURTHER INITIATIVES FOR IMPROVEMENT IN THE SYSTEM In its review of the issues related to quality in the delivery of student financial aid, the panel has endorsed a customer-oriented focus that is con- sistent with a mission of supporting the educational needs of the student applicant while making the most efficient use possible of taxpayer dollars. A customer service philosophy must embrace all of the internal customers of the system: the students, the postsecondary educational institutions, the lenders, guaranty agencies, the various relevant units within the Department of Education, and the Congress. The 1992 reauthorization of the Higher Education Act, which occurred as the panel was engaged in its delibera- tions, is a step in the right direction. One of its main goals is to simplify student aid programs. As a result of the passage of the bill, some of the concerns already expressed in this report concerning the complexity of the

OCR for page 1
4 QUALITY IN STUDENT FINANCIAL AID PROGRAMS application and its effect on student error have been reduced or even elimi- nated. Part H of the reauthorization bill focuses on the strengthening of the process by which institutions of higher education are allowed to participate in federal student aid programs through state licensure, accreditation, and federal eligibility, certification, and program review processes. The panel hopes that enactment of this section will lead to the development of perfor- mance-based statutes that avoid affixing common-cause solutions to spe- cial-case events and that aim at achieving measurable and meaningful out- comes rather than additional layers of bureaucracy. Examples of meaningful outcomes would include lower default rates and higher graduation rates, employment rates, and customer satisfaction. Toward a New Applicant Processing System The panel strongly believes that there is a need to change the current institutional verification process, which is fundamentally redundant and in- efficient. The panel suggests that the Department of Education use the knowledge it has at hand in much of the contract research that has ad- dressed verification. Those reports have indicated that (1) large errors re- main even after verification, (2) the cause of much of "student error" lies in the complicated application process, and (3) data items that must be fore- cast (e.g., estimated income, household size, and number in college) are the main contributors to student error. The reports have produced little evi- dence that the institutions themselves can develop procedures that will fur- ther reduce student error very much. True corrective actions lie mainly in the hands of the Department of Education and the Congress. The Department of Education must move from a system that focuses on detecting errors after awards have been made to a system that prevents error to the extent possible. Some error could be removed by making the applica- tion materials more user friendly, but additional action is needed. A combi- nation of changes to some data requirements and changes in the activities associated with the SAR and verification could result in large reductions in errors. For example, the panel believes that the department should develop a front-end match of applicant data with Internal Revenue Service (IRS) data tapes. Such matches are being performed at several agencies and are reportedly very successful. (The panel recognizes that the department will likely need the help of Congress to pass legislation directing IRS to allow these matches and to make some changes in data requirements.) The de- partment must take actions that affect the system in order to reduce "student error" appreciably and not impose further requirements on the institution in the verification and compliance-review procedures.

OCR for page 1
SUMMARY 15 Recommendation 9-1: The Department of Education and the Congress should work together to effectuate changes in data requirements that should simplify award determination and increase accuracy. Develop- ment of a system that requires applicant data to be matched with IRS data should receive top priority. Centralized Verification The panel believes that a system based on centralized verification wills in the long run, simplify the verification process and increase accuracy. Institutional burden will be removed if the signed statements of educational purpose, Selective Service compliance, and the like, are required as part of the initial application. This change would also remove considerable burden from the institution and from the Department of Education's audit and re- view activities. Similarly, data base matching with other agencies as di- rected by Congress in Title IV legislation should be used to the fullest extent possible. Federal concerns internal to the department, such as loan repayment problems and prior aid received, as measured by the financial aid transcript, should be identified from the department's own records. Here too, Congress has instructed the department through Title IV legislation to improve its internal data bases. The remaining major sources of institutional error (cost of attendance computation, enrollment monitoring, special situation exception determina- tions, bachelor's degree determination, and aid packaging) are associated with information and actions most easily found at the school. With the reduction of other unnecessary activities, the Department of Education and the schools could concentrate on developing effective methods to reduce errors in these activities. Freed of trivial inspection activities, the institu- tions and the department should have more time to focus on important issues of quality, such as effectiveness of educational programs, institu- tional integrity, and the value of the service purchased by the student and taxpayer. With data base matches and up-front certifications, many of the under lying causes of student error would be removed. Thus, the verification strategy should change. Rather than flagging the record for institutional action during verification, the SAR could request corrections or clarifying information when there are conflicts with data base information. These changes could be processed centrally, which would result in an initial award determination that would be considerably more accurate than is produced under the current procedures. It is possible that a small group of applica- tions might require assignment to institutions for further examination to resolve special situations that might otherwise delay the determination of award.

OCR for page 1
16 QUALITY IN STUDENT FINANCIAL AID PROGRAMS While up-front data processing costs might increase under this pro- posal, the Department of Education should consider total cost and service improvement in making decisions on implementation. Repeated processing of SARs would likely decrease. Financial aid administrators would be freed of burden, allowing them to spend more time on cases requiring special attention. Awards might be determined sooner, a service improvement for applicants. Consistency of verification and documentation would also be better controlled. Finally, award error would be controlled on all applica- tions, not just those selected for verification, as is now the case. Recommendation 9-2: The Department of Education should begin the development of a front-end and centralized verification system in which the schools' verification burden is drastically reduced. Under a new system, the department would identify exceptional cases, far fewer than the current 30 percent, for institutional review activities. Organizing Information About the System The Department of Education also asked the panel to comment on the organization of available information about award error and the usefulness of such data bases for making program decisions. In a similar investigation, the General Accounting Office (1985) found the department's approach to issues regarding the quality of awards to be unsystematic. That study found that decisions that might affect improvement were hindered by a lack of goals and analyses that were hurried due to operational demands. The study cited problems in coordinating offices within the department that needed to work together and labeled the resultant reactive decision making "remedial" and "not preventive." Recommendation 5-6: The Department of Education should not rou- tinely embark on surveys of the type that have been used in the past to estimate total error levels in student financial aid programs. The re- sources are likely to be better spent on continuous monitorin~using data from audit and review activities, for example~nd other approaches to quality improvement. Special studies should be done, but only when the study objectives are clearly linked to a policy evaluation or process improvement plan and steps can be taken to eliminate the methodologi- cal defects of past surveys. The panel, after discussions with Department of Education personnel and review of earlier studies, also found no particularly well-organized use of the data that focused on improving the quality of award decisions. It is evident that operational demands continue to limit proactive thinking. At- tempts to bring the parts of the department together to work on cross

OCR for page 1
SUMMARY 17 organizational issues are being made, but comments to the panel indicate that there is no strong management direction in these areas. Recommendation 6-1: The panel believes that a totally integrated ap- plicant data base is essential for problem identification, the develop- ment of solutions, and monitoring progress important activities if true quality improvements are to take place. The data base would compile information from all the Title IV aid programs and across years to provide, for each applicant, a complete history of participation in the financial aid system, thus facilitating a broad range of cross-sectional and longitudinal statistical analyses. The Department of Education should allocate sufficient resources and personnel to accomplish this task without further delay. Top Management Leadership Despite the genuine movement toward quality improvement that the 1992 reauthorization will bring about, there is much more in that direction to be accomplished. For example, a major detriment to sustaining the de- sired focus on quality is a lack of stability in leadership positions that influence student financial aid programs. Changing leadership may cause inconsistency in long-term vision. Frequent turnover in the position of Assistant Secretary for Postsecondary Education, a major reorganization of the Office of Postsecondary Education, and a number of shifts in line man- agement assignments are examples of the shifting leadership at the Depart- ment of Education. While, in themselves, these changes may not be bad, and some might be necessary, the panel sensed that the long-term vision for quality in student financial aid programs often suffered as a result of fre- quent turnover. The panel found that while the Department of Education required schools to invest heavily in quality inspection activities, it allocated limited re- sources for its own quality improvement activities. Further, the panel noted either inadequacies in the information technology available to the Depart- ment of Education or the lack of use of available information by the depart- ment in almost all the areas studied in this review. Recommendation 9-3: To signal top management's commitment to a Total Quality approach at the Department of Education, the department should initiate a Total Quality training program that starts with the leadership group and rapidly includes everyone in the department. In this way the fundamentals, vocabulary, and principles of Total Quality will be integrated into the department. Corporations have found this to be a critical first step in implementing quality improvement. Such a training effort should include leadership's role in the commitment to

OCR for page 1
8 QUALITY IN STUDENT FINANCIAL AID PROGRAMS change, process-improvement and problem-solving methods, Total Quality tools, focus on customer satisfaction and its measurement, teamwork, and quality planningldeployment. Recommendation 9-4: In its review of survey activities, verification procedures, and management information systems, the panel was disap- pointed to find a paucity of statistically trained personnel available to the student financial aid programs to analyze data and to interact with contractors. The Department of Education should develop a much greater in-house statistical capability to manage contracts that demand high levels of statistical expertise, and the data developed by contractors should be thoroughly documented and made available for in-house analysis. The department should also strengthen the analytic capabilities of its entire work force, including those who will not be expected to attain the level of statistical "expert" but yet should be skilled in handling and interpreting data. Specifically, the department should perform a needs analysis of statistical and computer literacy. Then it should develop training programs to improve abilities and purchase the hardware re- quired to carry out the necessary analytic tasks. The panel recognizes that technical improvement cannot begin without some initial funding targeted toward efforts such as pilot projects, data base improvement, and hiring staff with appropriate skills to help in these ef- forts. Thus, the panel requests that Congress take action to help make this possible. Recommendation 9-5: Congress should ensure that there is adequate funding and staffing to develop the quantitative information needed to manage and review the student financial aid programs effectively. The panel found an absence of performance measures linked to long- term planning. Setting long-term plans that transcend leadership changes is important. To make sound decisions on necessary changes to the programs, a new top management team must gain experience and familiarity with the system and its processes. A thorough understanding of a system as complex as the student financial aid programs takes time. Recommendation 9-6: To ensure the development of and commitment to long-term planning, the politically appointed position in charge of the student financial aid programs (the assistant secretary for postsecondary education) should have a fixed (commonly five-year) term.