Click for next page ( 186


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 185
APPENDIX A A Review of the Methodology Used in the Integrated Quality Control Measurement Project Mark Reiser Arizona State University This paper is a review of the methodology used in the Integrated Qual- ity Control Measurement Project (IQCMP), a project conducted to evaluate the quality of financial awards made in the 1988-89 award year under three Title IV financial aid programs: the Pell Grant Program, Campus-Based Programs, and the Stafford Loan Program (see Price Waterhouse, 1990~. In addition to comments on the methodology that was used in the IQCMP, possible alternative methodologies are suggested. ERROR The IQCMP used the following definition of error: "Error is the differ- ence between the award actually distributed and the award that would be calculated based on the best available data." Data that were considered to be the best available were obtained from the source of highest reliability (e.g., income tax returns and interviewers). One source was defined to be the most reliable, and it was then used as though it contained no error. When one source of information is treated as though it contains no error, it is sometimes referred to as the gold standard. The difference in awards as defined above had to exceed $50.00 before it was considered to be an error. In the IQCMP, the student and the academic institution were examined as sources of error in each of the Title IV programs studied. Overall and composite error were also examined. Overall error was error resulting from either student or institutional error. Student error and institutional 185

OCR for page 185
86 QUALITY IN STUDENT FINANCIAL AID PROGRAMS error do not necessarily add to overall error, however, because they do not necessarily have an additive effect. Composite error is error in the financial award from all programs com- bined. For this type of error, two main types of analysis were performed. First, the magnitude and percentage of errors were examined. Then, in error-profile analyses, the presence or absence of error was modeled using several independent variables simultaneously. As noted above, error calculations were performed using one or another source of information as an error-free gold standard. Since these sources used as standards almost certainly contain error themselves, the calculation of errors itself contains error. This feature constitutes a source of nonsampling error in the IQCMP. When no single source of information can be com- pletely free of error, adopting a statistical model for measurement error is a better approach, because the true value is considered to be a latent (unob- servable) variable and the different sources of measurement, called indica- tor or manifest variables, are all acknowledged as being subject to error. In many areas of social science, the reliability (or reliability ratio) is used as an indicator of the degree to which a variable is free from measurement error. In a technical definition, the reliability is expressed as the ratio of true variance to total variance, where total variance is equal to true variance plus measurement-error variance. Since no single source of information for determining financial aid eligibility is absolutely error free, it would have been very useful if the IQCMP had analyzed and reported the reliability of the different data sources. If more than one indicator is available, the reliabilities may be calculated in a straightforward way. The true value can also be estimated from a measurement-error model for multiple indicators. In future studies of quality control in student financial aid programs, the estimated true value for variables, such as wealth, could be calculated and used to determine the error in the amount of the financial award. Generally, variables with low reliability are not very useful for calculat- ing eligibility for financial awards. In the past, it appears that variables with large measurement error (low reliability) were either dropped from use on the student financial aid application or were the focus of regulations intended to decrease error during data collection. To the extent that error can be reduced with reasonable efforts, it is a good strategy to try to do so. But for some variables, such as wealth, it is very difficult to measure the true value directly, and an alternative to dropping such variables would be to estimate the latent true value from multiple sources of information, each of which may not be very reliable by itself. Measurement-error methodology should be investigated in any future studies of quality control in student financial aid programs. It is an ap- proach that is more efficient and justifiable than declaring one of the indica- tors to be the gold standard.

OCR for page 185
APPENDIX A 187 SAMPLING The IQCMP was based on a sample of 3,310 students from 350 postsecondary institutions. Because data were not available for some students, 2,653 stu- dents were in the final data base. Details of the sampling procedure are not given in the IQCMP report, but they are provided in a separate sampling plan document (Price Waterhouse, 1989~. A two-stage cluster sample was used in which postsecondary institutions were sampled in clusters of size 1, 2, 3, or 4. Because observations within each cluster are correlated to an unknown extent, methods of analysis that assume simple random samples are not appropriate. For proper analysis, each element of the sample must be assigned a weight based on the probability of selection for that element. In the sampling plan, expressions for the probability of selection are given on pages IV-8 and IV-9. The expressions appear to be incorrect, however, because they give the probability of selection as a sum of sampling rates across programs. The probability of selection calculated from this expres- sion could be greater than 1.O, although the sampling rates actually used in the study are probably not high enough for that to happen. The correct expression for the probability of selection for an element, given the sam- pling plan, should be 1.0 - (probability not selected for Pell sample) x (probability not selected for Campus-Based sample) x (probability not selected for Stafford sample). The probability calculated from this expression applies to the selection for the sample of all financial aid recipients. It is not the probability of selec- tion for an element in any combined sample (i.e., a direct sample plus elements obtained as a by-product of sampling other programs). Moreover, it is not clear how the probability of selection for an element in a combined sample should be calculated, because it would require knowing the prob- ability that a recipient is in one program given that he or she is in another program. If probabilities of selection were calculated by the expressions given on pages IV-8 and IV-9 of the sampling plan, the weights used for calculating errors would have been incorrect, and estimates of error would be biased. It is not possible to say how large the bias would be: It could be trivial or it could be substantial. In addition to sampling clusters of schools, the efficiency of the sam- pling design could have been increased through stratification by a number of variables, such as financial dependence versus independence. Since strati- fication was not used, post-stratification could be used to increase the preci- sion of some estimates.

OCR for page 185
88 QUALITY IN STUDENT FINANCIAL AID PROGRAMS As stated in the IQCMP report (p. I-2), the sample of students was designed to be large enough to permit statistical inferences about the per- centage of awards in error for each Title IV program with + 5 percent accuracy at a 90 percent confidence level. This statement was apparently made with complex sampling in mind, since a simple random sample of 2,653 would allow statistical inference with + 1.6 percent accuracy at a 90 percent confidence level. However, as discussed next, standard errors are not used consistently in the IQCMP report. The exhibits in Section II ("Error in the Title IV Financial Aid Pro- grams") of the IQCMP report present standard errors for many estimates. The basis for the standard errors is not clear, however. The standard errors given in the Section II exhibits have magnitudes (0.4 to 2.5) that appear to be based on an assumption of a simple random sample rather than the complex sample that was actually used. Although discussion in the sam- pling plan indicates that some standard errors were calculated by a boot- strap method, some may have been calculated by assuming a stratified ran- dom sample and ignoring clustering. It is not clear why standard errors would be calculated that way if bootstrap methods were available. More- over, estimators for standard errors under two-stage cluster sampling are known (Cochran, 1977), and computer programs are widely available to perform the calculations (e.g., Super Carp). The description of the boot- strap method proposed in the sampling plan is very brief, and it is not clear how the bootstrap was implemented, particularly with regard to the selec ~ . . lion of primary sampling units. Only Section II of the IQCMP report gives standard errors. Other sections of the report give a large number of statistics, but standard errors are not given. A footnote to Exhibit II-1 states that the percentage of recipients with error is based on 6.0 million students awarded Title IV aid during the 1988- 89 award year. Similar footnotes can be found in other Section II exhibits. The purpose of the footnotes appears to be to emphasize that the errors were examined only among the recipients of Title IV financial aid, and not among students who applied for but were denied financial aid. Nevertheless, the footnotes are confusing and should be reworded, because they can be inter- preted to mean that the percentages are based on the entire population of 6 million recipients rather than a sample. Many sections of the IQCMP report compare percentages across vari- ables and across domains. Financially dependent versus independent stu- dents and proprietary versus nonproprietary schools are frequently men- tioned domains. The statement in Section II regarding + 5 percent accuracy has very limited applicability for comparisons across variables or domains, and generally the appropriate interval would be larger than + 5 percent. For comparisons across domains, one would probably be willing to assume in

OCR for page 185
APPENDIX A 189 dependent samples for the different domains, although that would not nec- essarily be true, since both financially dependent and independent students would be selected from each school, and the clustering effect would induce a correlation among the students from the same school. However, assuming samples are independent across domains and that the domains split the sample into roughly equal-sized groups, the variance for the difference be- tween two percentages would be roughly double the variance for a single percentage based on the whole sample. Therefore, confidence intervals would be about 40 percent larger and a difference between two domains is probably not significant unless it is at least as large as 7 percentage points. Interpretations of differences that are less than 7 percentage points are un- warranted, and the IQCMP report contains many such instances. When percentages are compared across variables, the standard error for the difference may also be larger than the standard error for a single per- centage based on the whole sample. However, since the statistics on each variable are calculated from the same observations, the variables have a nonzero covariance, and the standard error for the difference requires the inclusion of that covariance. There is not enough information in the IQCMP report to obtain the covariances between the variables, and so it is not possible to say, based on the report, how large the standard error for differ- ences across variables might be. A statement such as the following from page II-1, "absolute student error was higher than absolute institutional error (6.6 percent of dollars and 5.1 percent of dollars, respectively)," is probably not warranted because the difference is probably not significant. Similar statements appear on pages II-6 and II-15. MARGINAL ANALYSIS The marginal analysis was conducted by comparing an award as actu- ally distributed with the award calculated by substituting the value from the most reliable source for the variable under study. As discussed above, reliability ratios would be a very useful statistic for comparing variables and sources of information. Also, it would be much more preferable to use an estimated true value based on a measurement-error model than to declare one source of information to be the most reliable. The marginal analysis found that the variables for students' (and spouses') cash, savings, and checking, as well as various measures of income, household size, and num- ber in college, were sources of a large amount of error. ERROR PROFILE ANALYSIS In addition to the marginal analyses discussed above, an error-profile analysis was conducted to examine the joint effects of predictor variables,

OCR for page 185
190 QUALITY IN STUDENT FINANCIAL AID PROGRAMS such as income, assets, and year in college. Since the dependent variable in this analysis was presence or absence of error (either student error or insti- tutional error), a preliminary analysis was performed using chi-square tests of association in contingency tables. All continuous variables were catego- rized for this analysis. Note that an analysis performed with contingency tables is still a marginal approach. The joint analysis was performed essen- tially by multiple regression, but an unusual series of steps were taken for the joint approach. The preliminary analysis using chi-square tests is very close in purpose to the marginal analyses described in an earlier section of the IQCMP re- port, except now the dependent variable is the presence or absence of an error rather than the magnitude of the error. Hence, one would expect to find essentially the same results here as in the marginal analyses. It is not clear from the IQCMP report why continuous variables were categorized. With a categorical dependent variable and continuous independent variables, a logistic regression, which does not require the independent variables to be categorical, can be used for the same purpose as the contingency tables described above. Since the logistic regression does not throw away infor- mation by categorizing the variables, it would have been a more efficient approach for the continuous variables than the contingency table approach that was used. The loss of information due to categorizing the variables would generally be no more than 5 percent, so it should not be expected that there would be extensive differences if logistic regression had been used instead. (Logistic regression can be performed with PROC LOGISTIC in SAS.) Since the categorization of continuous variables was based on percen- tile ranking, one possible benefit of the procedure would be to reduce skew- ness in the independent variables. If skewness was present (it is not pos- sible to tell if it is from the IQCMP report) it could be reduced to an accept- able level by a more straightforward transformation. With variables such as income, the use of the log transformation effectively reduces skewness. Although loss of information due to categorizing continuous variables is probably not large, future studies of quality control in student financial aid programs should use logistic regression when appropriate. It is also not clear from the IQCMP report whether the statistical meth- ods used for the contingency table analyses were appropriate for a complex sample involving clustering. Standard computer packages (e.g., SAS and SPSS) operate on the assumption of a simple random sample; packages that are based on methods for cluster samples are widely available (e.g., Super Carp). The erroneous use of simple random sample methods will tend to produce p values that are too small; hence, some variables that are not related to error in financial awards will appear to be related. Logistic regression

OCR for page 185
APPENDIX A 191 methods for cluster samples are also available (e.g., PC Carp) and would be the more appropriate tool. Results from the contingency table analyses showed that various mea- sures of income, various measures of assets, household size, and marital status were all strongly associated with student error. Among other vari- ables, institution type and the method used by an institution to recheck data and calculations were strongly associated with institutional error. To assess the effects of independent variables jointly on error, the IQCMP used a multiple regression approach. The multiple regression was based on some unusual methodology, few details of which are given in the IQCMP report. From the description, it appears that the dependent variable in these regressions was the presence or absence of error, that is, still a discrete variable. If the dependent variable is discrete, logistic regression, not ordi- nary multiple regression, should be used. The IQCMP report should clarify the method of estimation used here. In preparation for the multiple regressions, continuous independent variables were created by a transformation of dubious merit. The transformation consisted of assigning a value to the categorical variables based on the proportion of students with error for the relevant category from the contin- gency table analysis. It is not clear why this transformation was used, since the transformation does not make the variables continuous. It changes the distance between the categories, but it does not produce a continuous vari- able. Moreover, it is not clear why continuous variables were thought to be necessary. For a discrete dependent variable and for discrete independent variables, a logit analysis can be performed to accomplish the same purpose that a multiple regression accomplishes for continuous variables. (PROC CATMOD in SAS can be used to perform a logit analysis.) It is also not clear whether the method of assigning a value to categori- cal variables based on the proportion of students with error for the relevant category from the contingency table analysis was used on only the variables that were originally categorical or also on the variables that were originally continuous. For variables that were originally continuous, it would be pointless to categorize them by percentile rankings (as done for the contingency table analysis) and then transform them back to continuous variables by using the contingency table proportions for the multiple regression analysis. For the multiple regressions, the variables that were originally continuous should be used in that form, and not categorized first. The continuous variables could be log-transformed to stabilize the variance, and/or normalized, but trans- forming them from continuous to categorical and then back to continuous would be difficult to justify. The effects of cluster sampling should also be recognized in the regres- sion analyses used for the error-profile analyses. Ordinary least squares (OLS) regression is based on the assumption that observations are indepen

OCR for page 185
92 QUALITY IN STUDENT FINANCIAL AID PROGRAMS dent. Since cluster sampling was used for the IQCMP, observations within schools are correlated, and OLS should not be used to estimate parameters. In this situation, a nested-error regression model could be used to estimate model parameters by generalized least squares. The nested-error model has been well developed, and the effect of using it instead of an OLS model would be to widen confidence intervals around estimated values. That is, a variable is less likely to appear to be significantly related to error, because recognizing the correlation among observations within a school has the effect of reducing the sample size. (The nested-error regression model has been used with cluster sample data by Battese et al., 1988.) Estimation for the nested-error regression model can be carried out with the Super Carp computer program. Linear regression using weights to reflect the complex sampling could also be used, but there are advantages to the variance com- ponents approach used in the nested-error model. Another useful approach to the joint analyses would have been to esti- mate a covariance matrix for the (marginal) errors calculated in dollars. This covariance matrix could have been used with exploratory multivariate techniques, such as a principal component analysis and a canonical correla- tion analysis. The latter analysis would have been particularly useful for finding clusters of independent variables that relate to key dependent vari- ables. (The Super Carp computer package can be used to calculate the covariance matrix based on a two-stage cluster sample.) The results of the error-profile analyses based on multiple regressions showed that errors in reported income accounted for a large portion of the student-based errors in financial aid awards. Whether or not a tax form was filed, use of estimated income, and indicator variables for type of award sought were also significant variables. Institutional variables strongly asso- ciated with error included type of institution control, institution type, method used to recheck files, and indicators for type of award. SUMMARY Some of the statistical methods used for the IQCMP were inefficient, especially in the joint analyses for deriving error profiles. Some alternative methodologies, such as including the reliability of variables and using esti- mated true values, are recommended strongly for any future studies of qual- ity control in student financial aid programs. Despite the inefficiencies of some of the methods, there is a consis- tency to the findings given in the IQCMP report. The marginal analyses, the contingency table analyses, and the joint error-profile analyses all found that reported income was an important source of error in the awards made under the Title IV programs. Reported household size was also found to be a source of error in many analyses.

OCR for page 185
APPENDIX A 193 The IQCMP report also addresses simplification of the financial aid formula and potential corrective actions. It is natural to consider whether the simplifications and corrective actions reflect the findings from the sta- tistical analysis of errors. Several of the suggested corrective actions do reflect findings from the statistical analysis. In particular, the suggestion that awards not be based on tax information estimated by the applicant and the suggestions to require specific information on household size, number in college, and home value and debt are actions that address some of the consistently found sources of error. However, error in reported income, the variable most strongly associated with error over the different analyses, may not be reduced significantly by these suggested corrective actions. Perhaps other corrective actions should be considered for reported income. REFERENCES Battese, G. E., R. M. Carter, W. A. Fuller 1988 An error components model for prediction of county crop areas using survey and satellite data. Journal of the American Statistical Association 83(401):28-36. Cochran, W. G. 1977 Sampling Techniques. New York: John Wiley & Sons. Price Waterhouse 1989 Sampling Plan for the Integrated Quality Control Measurement Project. Prepared in association with Pelavin Associates, Inc. and The Gallup Organization. Wash ington, D.C.: Price Waterhouse. 1990 Integrated Quality Control Measurement Project, Findings and Corrective Actions. Prepared in association with Pelavin Associates, Inc. and The Gallup Organiza tion. Washington, D.C.: Price Waterhouse.