Click for next page ( 89


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 88
8 Meta-Analysis: A Quantitative Approach to Research Integration* STEPHEN B. THACKER The goal of an integrative literature review is to summarize the accumulated knowledge concerning a field of interest and to highlight important issues that researchers have left unresolved (1~. Traditionally, the medical literature has been integrated in the narrative form. An expert in a field will review studies, decide which are relevant, and highlight his or her findings, both in terms of results and, to a lesser degree, methodology. Topics for further research may also be proposed. Such narrative reviews have two basic weaknesses (2,3~. First, no systematic approach is prescribed to obtain primary data or to integrate findings; rather, the subjective judgment of the reviewer is used. As a result, no explicit standards exist to assess the quality of a review. Second, the narrative reviewer does not synthesize data quantitatively across literature. Consequently, as the number of studies in any discipline increases, so does the probability that erroneous conclusions will be reached in a narrative review (4~. Scientific research is founded on integration and replication of results; with the possible exception of a new discovery, a single study rarely makes a dramat- ic contribution to the advancement of knowledge (5~. In this article I summa- rize the constraints on reviewers of the medical literature and review alternative methods for synthesizing scientific studies. In particular, I examine meta-analy- sis, a quantitative method to combine data, and illustrate with a clinical example its application to the medical literature. Then, I describe the strengths and weakness of meta-analysis and approaches to its evaluation. Finally, I discuss * This paper was previously published in The Journal of the American Medical Association 1988;259:1685-1689. 88

OCR for page 88
META-ANALYSIS: A QUANTITATIVE APPROACH 89 current research issues related to meta-analysis and highlight future research directions. CONSTRAINTS ON LITERATURE REVIEW The limitations of any approach to literature review can be summarized as follows (6~: (a) sampling bias due to reporting and publication policies; (b) the absence in published studies of specific data desired for review; (c) biased exclusion of studies by the investigator; (d) the uneven quality of the primary data; and (e) biased outcome interpretation. These concerns are applicable to any form of literature review. Two types of bias in the published literature must concern a reviewer. First, because authors and journal editors tend to report statistically significant find- ings, a review limited to published studies will tend to overestimate the effect size. In a survey, for example, 58 investigators indicated that they had conduct- ed 921 randomized controlled trials, and that 96 (21.3 percent) were unpub- lished. Positive randomized controlled trials were significantly more likely to be published than negative trials (77 percent versus 42 percent, P < .001) (7~. At the same time, one should not uncritically assume that methods are better in published studies, as the quality of published papers varies dramatically (8~. Second, another form of publication bias, the confirmatory bias, tends to emphasize and believe experiences that support one's views and to ignore or discredit those that do not. Results of a study of 75 journal reviewers asked to referee identical experimental procedures showed poor interrater agreement and a bias against results contrary to their theoretical perspective (9~. Consequently, new or unpopular data tend also to be underreported in the published literature. Data available from primary research studies may be inadequate for the liter- ature reviewer. The reviewer is often confronted with selective reporting of pri- mary findings, incorrect primary data analysis, and inadequate descriptions of original studies (10~. In a study of psychotherapy outcomes, for example, an effect could not be calculated in 26 percent of studies because of missing data, a number comparable with previous reports (11~. In addition to identifying studies, the investigator must decide which reports to include in a review (3~. One option is to use all available data and thereby maximize the representativeness of the conclusions. Using this approach, how- ever, one will decrease the statistical validity of the data synthesis by including less rigorous studies. Exclusion of studies for methodological reasons, on the other hand, will increase the statistical validity but will decrease the size of the overall pool of data and may sacrifice the ability to generalize from the results. Variable data quality is probably the most critical limitation for the reviewer. The effect of data quality was seen in a study of quality of life outcomes follow- ing coronary bypass graft surgery, when investigators found the estimates of benefit to be 15 percent less in randomized controlled trials than in trials using matching (12~. Similarly, results of studies in medical care tend to show

OCR for page 88
go STEPHEN B. THACKER decreasing odds ratios with increased rigor of studies (8), although in one large study of psychotherapy, the effect was found to increase with increasing rigor (111. In quantitative reviews, statistical methods, including stratified analyses and multivariate methods, can be used to measure the impact on the results of varying quality in studies (8,13,141. Although these constraints have been recognized previously, the more recent efforts to address concerns about research integration have stimulated new efforts to deal with them. QUANTITATIVE APPROACHES TO SUMMARIZING ACROSS STUDIES During the past several years, there have been several different approaches developed to summarize quantitatively data found in different studies of the same or similar research problems. The simplest approach to the quantitative integration of research is vote counting. With this approach, results of studies under consideration are classified into three categories: (a) statistically signifi- cant in one direction, (b) statistically significant in the opposite direction, or (c) no statistically significant difference. Then, the category receiving the most votes is judged to approximate truth (151. Although simple to use, voting meth- ods do not take into account the magnitude of effect or sample size. In addition, this approach does not address the aforementioned problems inherent in tradi- tional reviews, such as inadequate study methodology and uneven data quality. In 1971, Light and Smith (15) proposed an alternative to voting methods that takes advantage of natural aggregations, or clusters, in the population. In this approach, one studies a problem in various clusters, such as neighborhoods or classrooms, and searches for explanations for differences among clusters. If these differences are explainable, the data can be combined and statistical vari- ability can be described. A third method for combining literature is pooling, a method by which data from multiple studies of a single topic, such as p-blockade after myocardial infarction, are combined in a single analysis (161. This method is limited by the availability of raw data; the variation in study methods, populations, and out- comes under study; and statistical considerations (17,18~. In a 1976 study of the efficacy of psychotherapy, Glass (19) coined the term meta-analysis, "the statistical analysis of a large collection of results from indi- vidual literature, for the purpose of integrating the findings." Alternatively, meta-analysis can be defined as any systematic method that uses statistical anal- yses for combining data from independent studies to obtain a numerical esti- mate of the overall effect of a particular procedure or variable on a defined out- come (20~. While there have been several approaches to meta-analysis, the steps can be defined generally as (a) defining the problem and criteria for admission of stud- ies, (b) locating research studies, (c) classifying and coding study characteris-

OCR for page 88
META-ANALYSIS: A QUANTITATIVE APPROACH 9 tics, (d) quantitatively measuring study characteristics on a common scale, (e) aggregating study findings and relating findings to study characteristics (analysis and interpretation), and (f) reporting the results (21,22~. Problem formulation includes the explicit definition of both outcomes and potentially confounding variables. Carefully done, this step enables the investi- gator to focus on the relevant measures in the studies under consideration and to specify relevant methods to classify and code study characteristics. The literature search includes a systematic approach to locating studies (11 First, one obtains information from the so-called invisible college, i.e., the informal exchange of information among colleagues in a particular discipline Second, one searches indexes (e.g., Index Medicus and the Social Science Citation Index), abstracting services (e.g., International Pharmaceutical Abstracts), and computerized searches (e.g., MEDLINE and TOXLINE) to obtain research articles and sources of both published and unpublished data. Third, references in available studies identify further sources. The retrieval from academic, private, and government researchers of unreferenced reports, the so-called fugitive literature, as well as unpublished data, further minimizes selective reporting and publication biases. Several methods are used to measure the results across studies (3,231. The most commonly used measure in the social sciences is the effect size, an index of both the direction and magnitude of the effect of a procedure under study (191. Glass and his colleagues (24) developed this method when assessing the efficacy of psychotherapy on the basis of data from controlled studies. One estimate of effect size for quantitative data is the difference between two group means divided by the control group SD: (X~- Xc)/Sc, where X~ is the mean of the experimental or exposed group, Xc is the mean of the control or unexposed group, and Sc is the SD of the control group. Effect size expresses differences in SD units so that, for example, if a study has an effect size of 0.2 SD units, the overall effect size is half that of another study that has an effect size of 0.4 SD units. The appropriate measure of effect across literature will vary according both to the nature of the problem being assessed and to the availability of pub- lished data (7,251. Pooling of data from controlled clinical trials, for example, has been more widely used in the medical literature (16,261. Effect size for proportions has been calculated in cohort literature as either a difference, P' - Pc, or as a ratio, Pi /Pc (31. The latter has the advantage of con- sidering the change relative to the control percentage and, in epidemiologic studies, is equivalent analytically to the concept of the risk ratio. Whatever combination statistic is used, a systematic quantitative procedure to accumulate results across studies should include the following (271: (a) sum- mary descriptive statistics across studies and the averaging of those statistics; (b) calculation of the variance of a statistic across studies (i.e., tests for hetero- geneity); (c) correction of the variance by subtracting sampling error; (d) cor- rection in the mean and variance for study artifacts other than sampling, such as measurement error; and (e) comparison of the corrected SD to the mean to

OCR for page 88
92 STEPHEN B. THACKER assess the size of the potential variation across studies. A growing literature on statistical methods deals with problems in calculating effect size or significance testing as it relates to meta-analysis (28,293. BENEFITS OF META-ANALYSIS Meta-analysis forces systematic thought about methods, outcomes, catego- rizations, populations, and interventions as one accumulates evidence. In addi- tion, it offers a mechanism for estimating the magnitude of effect in terms of a statistically significant effect size or pooled odds ratio. Furthermore, the combi- nation of data from several studies increases generalizability and potentially increases statistical power, thus enabling one to assess more completely the impact of a procedure or variable (30~. Quantitative measures across studies can also give insight into the nature of relationships among variables and pro- vide a mechanism for detecting and exploring apparent contradictions in results. Finally, users of meta-analysis have expressed the hope that this systematic approach would be less subjective and would decrease investigator bias. APPLICATIONS OF META-ANALYSIS IN HEALTH Interest in clinical applications of meta-analysis has risen dramatically in recent years (31,32~. An increasing number of attempts have been made to use meta-analysis outside of mental health or educational settings, including such other settings as chemotherapy in breast cancer (33), patient education interven- tions in clinical medicine (34), spinal manipulation (35), the effects of exercise on serum lipid levels (36), and duodenal ulcer therapy (37~. There has also been discussion of the potential applications of meta-analysis to public health (38~. An interesting application of meta-analysis was an effort to quantify the impact on survival and safety of a wide range of surgical and anesthetic innova- tions (39~. More typical are efforts to draw conclusions from data pooled from a limited number of studies, usually controlled clinical trials (26,40-47~. Pooling techniques have also been applied to data from non-randomized studies in attempts to address incompletely studied problems and to increase represen- tativeness (25,48,49~. A CASE STUDY: ELECTRONIC FETAL MONITORING In a 1979 review of the efficacy and safety of intrapartum electronic fetal monitoring, Banta and Thacker (50) set out to assess the evidence for the efD~ca- cy and safety of the routine use of electronic fetal monitoring. The independent variable was defined as the clinical application of all forms of electronic fetal monitoring to both high- and low-risk pregnant women; the outcomes measured were various measures of maternal and fetal morbidity and mortality, as well as the occurrence of cesarean delivery. Cost issues were also addressed.

OCR for page 88
META-ANALYSIS: A QUANTITATIVE APPROACH 93 A literature search began with the exchange of information with colleagues in obstetrics, pediatrics, epidemiology, technology assessment, and economics. References to published research articles were obtained from MEDLINE and Index Medicus and supplemented with references in articles under review. Efforts were also made to obtain unpublished reports and professional meeting abstracts. Although this review was systematic and extensive and comparable evidence from studies was sought, a quantitative analysis across studies was limited to descriptive statistics. A 1987 meta-analysis of this same issue focused on evidence from random- ized controlled trials and the previous literature search supplemented with infor- mation from the Oxford Data Base of Perinatal Trials and from direct corre- spondence with individual investigators (51~. Variables were codified and, where possible, made comparable. For example, published measures of the Apgar score varied in timing (at 1, 2, and 5 minutes) and classification (abnor- mal was defined variably to include or exclude a score of 7~; authors were asked to provide one-minute Apgar scores where a normal score included 7. The primary data were then organized into descriptive tables that listed study results for specific outcomes, such as low Apgar score, perinatal mortality, and cesarean delivery, as well as for measures of diagnostic precision, such as sensi- tivity, specificity, and predictive value (see Table 8.1) (50~. The findings of the randomized controlled trials were evaluated for comparability and then pooled (see Table 8.2), and the pooled analyses were stratified by data quality (51~. The results of the pooled analyses were then reported, conclusions were drawn, and recommendations were made. TABLE 8.1 Accuracy of electronic fetal monitoring using Apgar score as mea- sure of outcomea Number of Investigator, year Patients PPV NPV Sensitivity, % Specificity,% Bissonnette (69), 714 80 94 57 82 1975 Gabert and Stenchever (70), 749 66 91 84 80 1973 Schifrin and Dame (71), 307 43 93 54 90 1972 Saldanaetal.(72), 620 23 86 71 44 1976 Tipton and Shelley (73), 100 81 93 82 93 1971 aAbnormality defined as one-minute Apgar score < 7 (except Gabertone-minute Apgar score < 6). PPV, positive predictive value; NPV, negative predictive value.

OCR for page 88
94 STEPHEN B. THACKER TABLE 8.2 Pooled data from six controlled trials assessing efficacy of routine electronic fetal monitoring in labor Pooled 95% Confidence Test for Odds Ratio Interval Heterogeneitya Apgar score < 7 1.07 0.88 - 1.30 X26 = 1.28 Apgar score < 4 0.87 0.56 - 1.30 x25 = 1.84 Neonatal seizures 0.39 0.14- 1.08 X26 = 3~75 NICU admissions 1.01 0.84 - 1.22 X26 = 12.15 Pennatal deadens 1.73 0.53 - 5.64 X26 = 5.51 aAll tests were not significant. The subscript numbers refer to degrees of freedom. NICU, neonatal intensive care unit. The 1979 study concluded that the data did not support the routine use of electronic fetal monitoring and recommended additional randomized controlled trials and limitation of electronic fetal monitoring to high-risk pregnancies (501. The 1987 report included randomized controlled trials already cited in the origi- nal study and three additional randomized controlled trials (seven randomized controlled trials from five countries). No known clinical trials were excluded from this report although the largest trial (52), which included more subjects than the other six in combination, was analyzed separately and compared with the pooled results of the others. Analyses of different subsets of these studies based on differences in design (e.g., use of fetal scalp blood in sampling) and study quality found minor varia- tions in results, but no changes in the basic findings. In both reports the pooled cesarean delivery rate was twofold higher in the group with electronic fetal monitoring. Data from the randomized controlled trial that scored highest in an assessment of the quality of study design and implementation, however, indicat- ed that electronic fetal monitoring combined with fetal scalp blood sampling could be used to identify infants at risk of neonatal seizures (521. That study had been suggested by pooled analyses of earlier randomized controlled trials (53~. While both of these reports illustrate the advantages of the systematic and comprehensive approach to a literature review, the meta-analytic methods used in Me 1987 report illustrate both increased statistical power derived from data pooling and increased information found from stratification of studies. Subsequently available trials reported results consistent with that meta-analysis (54,551. CRITICISMS OF META-ANALYSIS When meta-analysis was introduced in the psychology literature, it did not meet with universal acceptance. It was variously described as "an exercise in

OCR for page 88
META-ANALYSIS: A QUANTITATIVE APPROACH 95 mega-silliness" and "an abuse of research integration" (56,57~. In addition to the constraints listed above related to literature review, the meta-analyst is con- fronted with additional challenges in an effort to synthesize data quantitatively across studies. Statistical significance testing that is familiar to most clinicians is based on an assumption that data are selected randomly from a well-specified population. Non-random selection of studies and multiple tests of the same data, either through repeated publication of partial or entire data sets or through use of more than one outcome for each person, are two ways that this assumption is violat- ed. Nevertheless, standard parametric statistics have been considered to be suf- ficiently robust to be usable in meta-analyses (581. The current use of parametric statistical methods for meta-analysis requires additional theoretical study (29~. Other methodological issues of concern to meta-analysts include bias (59), variability between studies (60), and the devel- opment of models to measure variability across studies (611. Additional statis- tical research should include study of the impact of outliers on the meta-analy- sis and the potential insight that they could provide into a research question (281. Statistically valid methods to combine data across studies of varying quality and design, including data from case-control studies, will enable meta- analysts to maximize the value of their data syntheses (48~. One serious concern about quantitative reviews of the literature is that although meta-analysis is more explicit, it may be no more objective than a nar- rative review (621. Both critics and advocates of meta-analysis are concerned that an unwarranted sense of scientific validity, rather than true understanding, may result from quantification (63,641. In other words, sophisticated statistics will not improve poor data but could lead to an unwarranted comfort with one's conclusions (651. EVALUATION OF META-ANALYSIS The evaluation of a literature review, like its conduct, should be systematic and quantitative. Evaluation criteria for meta-analysis include the need for the following: (a) clear identification of the problems under study; (b) active effort to include all available studies; (c) assessment of publication bias; (d) identifi- cation of data used; (e) selection and coding based on theoretical framework, not convenience; (f) detailed documentation of coding; (g) use of multiple raters to assess coding, including assessment of interrater reliability; (h) assess- ment of comparability of the cases, controls, and circumstances in the studies analyzed; (i) consideration of alternative explanations in the discussion; (J.) rela- tion of study characteristics to problems under review; (k) careful limitation of generalization to the domain of the literature review; (l) reporting in enough detail to enable replication by a reviewer; and (m) guidelines for future research (3,661.

OCR for page 88
96 STEPHEN B. THACKER COMMENT Meta-analysis is an attempt to improve traditional methods of narrative review by systematically aggregating information and quantifying its impact. Meta-analysis was introduced to address the problem of synthesizing the large quantity of information on a particular subject, a problem that has been exacer- bated by the large volume of published research in the past 20 years. It is viewed, however, only as a step in the process of developing better tools to quantify information across studies. It should neither be considered the final word in quantitative reviewing nor be dropped in haste because of the problems and criticisms discussed above. Certainly, benefits are to be obtained from sys- tematic and rigorous review of available information, including increases in power and generalizability, better understanding of complex issues, identifica- tion of correlations among variables, and identification of gaps to be addressed by appropriate research. When criticizing meta-analysis, one must distinguish between those prob- lems that are inherent in any literature review and those that are specifically a problem with meta-analysis. For example, data quality, sampling bias, and data retrieval are limitations inherent in any literature review. Similarly, while out- come interpretation may be affected by the various styles of summarizing research findings, biases are not limited to the meta-analyst. On the other hand, one must be wary of inappropriate weight being given to a procedure just because it is quantitative, particularly when used by those who do not under- stand the limitations of the statistical methods utilized. Finally, critics should empirically test the impact of their criticisms so as to take meta-analysis or its alternative methods of quantitative summarization of research to the next level of usefulness. It has been suggested that investigators should combine quantitative and qualitative review data to enable practitioners to apply results to individual patients or program problems (67~. In this way, researchers can investigate issues that are important but difficult to quantify. Nonquantitative information, such as expert opinion and anecdotal evidence, does have a significant impact on policy. Finally, one must be concerned that although even the best meta- analysis may represent all available trials and relevant studies, it may not repre- sent clinical practice because of the nature of how and where research is con- ducted (631. Several things can be done to assess meta-analysis and to improve methods of quantitative review. First, one can compare the results of meta-analysis with those of narrative reviews to identify differences in interpretation and conclu- sions. In one study where a statistical procedure for summarizing research find- ings was compared with narrative reviews, it was found that the statistical reviewer was more likely to support the hypothesis both in direction and magni- tude, although the basic recommendations did not differ between groups (681. A second important area of research is in statistical methodology. Both theoret-

OCR for page 88
META-ANALYSIS: A QUANTITATIVE APPROACH 97 ical research into the assumptions of alternative methods and empirical research testing of the accuracy and efficiency of these methods need to be undertaken. Third, methods to assess the quality of meta-analysis need to be tested and refined (661. Finally, in assessing meta-analysis, one must be careful to limit the extrapolation of conclusions to the field of study covered by the literature review. Although this is true of any cumulative review, the boundaries of the review must be carefully delineated and interpretation confined to those boundaries. In summary, the systematic, quantitative review and organization of the cumulative experience in a subject matter is fundamental to good scientific practice. Meta-analysis is a methodology that warrants testing and empirical evaluation. This is similarly true of alternative approaches to synthesizing information. The need to use available information optimally cannot be avoid- ed by We rational scientist. The particular framework of review be it meta- analysis or some over approachshould be addressed as an important scientif- ic endeavor. The importance of addressing this issue must be underscored in an era where scientific information is increasing exponentially and Me potential for application of these findings is unprecedented. REFERENCES 1. Cooper HM. Scientific guidelines for conducting integrative research reviews. Review of Educational Research 1982;52:291-302. 2. Jackson GB. Methods for integrative reviews. Review of Educational Research 1980;50:438-460. 3. Light RJ, Pillemer DB. Summing Up the Science of Reviewing Research. Cambridge, Mass.: Harvard University Press, 1984. 4. Levine RL, Hunter JE. Regression methodology: Correlation, meta-analysis, confi- dence intervals, and reliability. Journal of Leisure Research 1983;15:323-343. 5. Fiske DW. The meta-analytic revolution in outcome research. Journal of Consulting and Clinical Psychology 1983;51:65-70. 6. Hattie JA, Hansford BC. Meta-analysis: A reflection on problems. Australian Journal of Psychology 1984;36:239-254. 7. Chan SS, Sacks HS, Chalmers TC. The epidemiology of unpublished randomized control trials, abstracted. Clinical Research 1982;30:234A. 8. Chalmers TC, Smith H Jr, Blackburn B et al. A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 1981;2:31-49. 9. Mahoney MJ. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1977;1:161-175. 10. Searles IS. A methodological and empirical critique of psychotherapy outcome meta-analysis. Behaviour Research and Therapy 1985;23:453~63. 11. Landman JT, Dawes RM. Psychotherapy outcome: Smith and Glass' conclusions stand up under scrutiny. American Psychologist 1982;37:50~516. 12. Wortman PM, Yeaton WH. Cumulating quality of life results in controlled trials of coronary artery bypass graft surgery. Controlled Clinical Trials 1985;6:289-305. 13. Chalmers TC, Celano P. Sacks HS et al. Bias in treatment assignment in controlled clinical trials. New England Journal of Medicine 1983;309:1358-1361.

OCR for page 88
98 STEPHEN B. THACKER 14. Yeaton WH, Wortman PM. Evaluation issues in medical research synthesis. In Yeaton WH, Wortman PM (eds). Issues in Data Synthesis. San Francisco: Jossey- Bass Publications, 1984:43-56. 15. Light RJ, Smith PV. Accumulating evidence: Procedures for resolving contradic- tions among different research studies. Harvard Educational Review 1971 ;41 :429-471. 16. Yusef S. Peto R. Lewis J et al. Beta blockade during and after myocardial infarc- tion: An overview of the randomized trials. Progress in Cardiovascular Diseases 1985;27:335-371. 17. Einarson TR, McGhain WF, Boatman JL et al. Meta-analysis: Quantitative integra- tion of independent research results. American Journal of Hospital Pharmacy 1985;42:1957-1964. 18. Bryant FB, Wortman PM. Methodological issues in the Meta-analysis of quasi- experiments. In Yeaton WH, Wortman PM (eds). Issues in Data Synthesis. San Francisco: Jossey-Bass Publications, 1984:15-24. 19. Glass GV. Primary, secondary, and Meta-analysis of research. Educational Researcher 1976;5:3-8. 20. Leviton LC, Cook TD. What differentiates Meta-analysis from other forms of review? Journal of Personality 1981;49:231-236. 21. Ottenbacher KJ, Petersen P. Quantitative reviewing of medical literature: An approach to synthesizing research results in clinical procedures. Clinical Pediatrics 1983;28:423-427. 22. Curlette WL, Cannella KS. Going beyond the narrative summarization of research findings: The Meta-analysis approach. Research in Nursing and Health 1985;8:293-301. 23. Rosenthal R. Combining results of independent studies. Psychological Bulletin 1978;85: 185-193. 24. Glass GV, McGraw B. Smith ML. Meta-analysis in Social Research. Beverly Hills, Calif.: Sage Publications, 1981. 25. Goldsmith JR, Beeser S. Strategies for pooling data in occupational epidemiologi- cal studies. Annals: Academy of Medicine (Singapore) 1984;13(suppl 2~:297-307. 26. Sacks HS, Chalmers TC, Berk AA et al. Should mild hypertension be treated? An attempted Meta-analysis of the clinical trials. Mount Sinai Journal of Medicine 1985;52:265-270. 27. Hunter JE, Schmidt FL, Jackson GB. Meta-analysis: Cumulating Research Findings Across Studies. Beverly Hills, Calif.: Sage Publications, 1982. 28. Klitgaard RF. Identifying exceptional performers. Policy Analysis 1978;4:529-547. 29. Hedges LV, Olkin I. Statistical Methods for Meta-analysis. Orlando, Fla.: Academic Press, 1985. 30. Pillemer DB, Light RJ. Synthesizing extremes: How to use research evidence from many studies. Harvard Educational Review 1980;50:176-195. 31. Proceedings of the Workshop on Methodologic Issues in Overviews of Randomized Clinical Trials. Statistics in Medicine 1987;6:217~09. 32. L' Abbe KA, Detsky AS, O'Rourke K. Meta-analysis in clinical research. Annals of Internal Medicine 1987;107:22~233. Himel HN, Liberati A, Gelber RD et al. Adjuvant chemotherapy for breast cancer: A pooled estimate based on published randomized control trials. Journal of the

OCR for page 88
META-ANALYSIS: A QUANTITATIVE APPROACH 99 American Medical Association 1986;256: 1148-1159. 34. Mumford E, Schlesinger HI, Glass GV. The effects of psychological intervention on recovery from surgery and heart attacks: An analysis of the literature. American Journal of Public Health 1982;72:141-151. 35. Ottenbacher K, Difabio RP. Efficacy of spinal manipulation/mobilization therapy: A meta-analysis. Spine 1985; 10:833-837. 36. Tran ZV, Weltman A. Differential effects of exercise on serum lipid and lipoprotein levels seen with changes in body weight: A meta-analysis. Journal of the American Medical Association 1985;254:919-924. 37. Christensen E, Juhl E, Tygstrup N. Treatment of duodenal ulcer: Randomized clini- cal trials of a decade (1964 to 1974~. Gastroenterology 1977;73: 1170-1178. 38. Louis TA, Fineberg HV, Mosteller F. Findings for public health from meta-analy- sis. Annual Review of Public Health 1985;6:1-20. 39. Gilbert JP, McPeek B. Mosteller F. Progress in surgery and anesthesia: Benefits and risks of innovative therapy. In Bunker JP, Barnes BA, Mosteller F (eds). Costs, Risks, and Benefits of Surgery. New York: Oxford University Press, 1977: 124-169. 40. Chalmers TC, Matta RJ, Smith H Jr et al. Evidence favoring the use of anticoagu- lants in the hospital phase of acute myocardial infarction. New England Journal of Medicine 19;297:1091-1096. 41. Stewart AL, Reynolds EOR, Lipscomb AP. Outcome for infants of very low birth- weight: Survey of world literature. Lancet 1981;1:1038-1041. 42. Baum ML, Anish DS, Chalmers TC et al. A survey of clinical trials of antibiotic prophlaxis in colon surgery: Evidence against further use of no-treatment controls. New England Journal of Medicine 1981;305:795-799. 43. Blackburn BA, Smith H. Chalmers TC. The inadequate evidence for short hospital stay after hernia for varicose vein stripping surgery. Mount Sinai Journal of Medicine 1982;49:383-390. 44. Stamfer MJ, Gol&aber SZ, Yusuf S et al. Effect of intravenous streptokinase on acute myocardial infarction. New England Journal of Medicine 1982;307:1180-1182. 45. Toth PJ, Horwitz RI. Conflicting clinical trials and the uncertainty of treating mild hypertension. American Journal of Medicine 1983;75:482-488. 46. Thacker SB. Quality of controlled clinical trials: The case of imaging ultrasound in obstetrics: A review. British Journal of Obstetrics and Gynaecology 1985;92:437 4~14. 47. Colditz GA, Tuden RL, Oster G. Rates of venous thrombosis after general surgery: Combined results of randomized clinical trials. Lancet 1986;2:143-146. 48. Eddy DM. The use of conf~dence profiles to assess tissue-type plasminogen activa- tor. In Califf RM, Wagner GS (eds). Acute Coronary Care 1987. Boston: Martinus Nijhoff, 1986, chap 7. 49. Schneider AP II. Breast milk jaundice in the newborn: A real entity. Journal of the American Medical Association 1986;255:3270-3274. 50. Banta HD, Thacker SB. Assessing the costs and benefits of electronic fetal moni- toring. Obstetrical and Gynecological Survey 1979;34:627-642. 51. Thacker SB. The eff~cacy of intrapartum electronic fetal monitoring. American Journal of Obstetrics and Gynecology 1987;156:2~30. 52. MacDonald D, Grant A, Sheridan-Pereira M et al. The Dublin randomized con-

OCR for page 88
100 STEPHEN B. THACKER trolled trial of intrapartum fetal heart rate monitoring. American Journal of Obstetrics and Gynecology 1985; 152:524-539. 53. Chalmers I. Randomized controlled trials of intrapartum monitoring, in Thalhammer O. Baumgarten KV, Pollack A (eds). Perinatal Medicine. Stuttgart, West Germany: Georg Thieme Verlag, 1979:260-265. 54. Leveno KJ, Cunningham FG, Nelson S. et al. A prospective comparison of selec- tive and universal electronic fetal monitoring in 34,995 pregnancies. New England Journal of Medicine 1986;315:615-619. Luthy DA, Shy KK, Van Bell G et al. A randomized trial of electronic fetal moni- toring in premature labor. Obstetrics and Gynecology 1987;69:687-695. 56. Eysenck HI. An exercise in mega-silliness. American Psychologist 1978;33:517. 57. Eysenck HI. Meta-analysis: An abuse of research integration. Journal of Special Education 1984;18:41-59. 58. Taveggia TC. Resolving research controversy through empirical cumulation. Sociological Methods and Research 1974;2:395-407. 59. Chalmers TC, Levin H. Sacks HS et al. Meta-analysis of clinical trials as a scientif- ic discipline: I. Control of bias and comparison with large cooperative trials. Statistics in Medicine 1987;6:315-325. 60. Chalmers TC, Berrier J. Sacks HS et al. Meta-analysis of clinical trials as a scien- tific discipline: II. Replicate variability and comparison of studies that agree and disagree. Statistics in Medicine 1987;6:733-744. 61. DerSimonian R. Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials 1986;7:177-188. 62. Mintz J. Integrating research evidence: A commentary on meta-analysis. 1983;51 :71-75. 63. Cook TD, Leviton LC. Reviewing the literature: A comparison of traditional meth- ods with meta-analysis. Journal of Personality 1980;48:449-472. 64. Cooper HM, Arkin RM. On quantitative reviewing. Journal of Personality 1981;49:225-230. 65. Wilson GT, Rachman SJ. Meta-analysis and the evaluation of psychotherapy out- come: Limitations and liabilities. Journal of Consulting and Clinical Psychology 1983;51:54-64. 66. Sacks HS, Berrier J. Reitman D et al. Meta-analysis of randomized controlled tr' als. New England Journal of Medicine 1987;316:450-455. 67. Light RJ, Pillemer DB. Numbers and narrative: Combining their strengths in research reviews. Harvard Educational Review 1982;52:1-26. 68. Cooper HM, Rosenthal R. Statistical versus traditional procedures for summarizing research f~ndings. Psychological Bulletin 1980;87:442 449. Bissonnette JM. Relationship between continuous fetal heart rate patterns and Apgar score in the newborn. British Journal of Obstetrics and Gynaecology 1975;82:2~28. Gabert HA, Stenchever MA. Continuous electronic monitoring of fetal heart rate during labor. American Journal of Obstetrics and Gynecology 1973;115:919-923. Schifrin BS, Dame L. Fetal heart rate patterns, prediction of Apgar score. Journal of the American Medical Association 1972;219:1322-1325. 72. Saldana LR, Schulman H. Yang W. Electronic fetal monitoring during labor. Obstetrics and Gynecology 1976;47:706-710. 73. Tipton R. Shelley T. An index of fetal welfare in labour. Journal of Obstetrics and Gynaecology of the British Commonwealth 1971;78:702-706. 69. 70