Appendix A
Literature Review Tables

Evidence on the effectiveness of continuing education (CE) and CE methods was identified through a literature review. Although nonexhaustive, the review included a comprehensive search of the Research and Development Resource Base (RDRB), a bibliographic database of more than 18,000 articles from fields including CE, knowledge translation, interprofessional literature, and faculty development. Articles in the RDRB are culled from Medline, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Excerpta Medica Database (EMBASE), Education Resources Information Center (ERIC), Sociological Abstracts, PsychoInfo, Library Information and Science Abstracts (LISA), and business databases, as well as automatic retrieval of articles from journals dedicated to medical education (e.g., Journal of Continuing Education in the Health Professions, Medical Education, Studies in Continuing Education).

The RDRB was searched using keywords,1 and the results of the searches were culled by two independent reviewers using an iterative approach. Studies collected were from 1989 to April 2009.

1

Keywords used to search the RDRB included “patient participation,” “patient initiated,” “patient mediated,” “physician prompt,” “audit,” “feedback,” “checklist,” “checklists,” “protocol,” “protocols,” “reminder,” “reminders,” “academic detailing,” “simulation,” “simulations,” “lifelong learning,” “experiential,” “self-directed,” “reflection,” “problem based,” “model,” and “modeling.” These keywords were used alone or in combination.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 147
Appendix A Literature Review Tables E vidence on the effectiveness of continuing education (CE) and CE methods was identified through a literature review. Although nonexhaustive, the review included a compre- hensive search of the Research and Development Resource Base (RDRB), a bibliographic database of more than 18,000 articles from fields including CE, knowledge translation, interprofessional lit - erature, and faculty development. Articles in the RDRB are culled from Medline, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Excerpta Medica Database (EMBASE), Edu- cation Resources Information Center (ERIC), Sociological Abstracts, PsychoInfo, Library Information and Science Abstracts (LISA), and business databases, as well as automatic retrieval of articles from journals dedicated to medical education (e.g., Journal of Continuing Education in the Health Professions, Medical Education, Studies in Con- tinuing Education). The RDRB was searched using keywords,1 and the results of the searches were culled by two independent reviewers using an iterative approach. Studies collected were from 1989 to April 2009. 1 Keywords used to search the RDRB included “patient participation,” “patient initiated,” “patient mediated,” “physician prompt,” “audit,” “feedback,” “checklist,” “checklists,” “protocol,” “protocols,” “reminder,” “reminders,” “academic detailing,” “simulation,” “simulations,” “lifelong learning,” “experiential,” “self-directed,” “re - flection,” “problem based,” “model,” and “modeling.” These keywords were used alone or in combination. 147

OCR for page 147
148 REDESIGNING CE IN THE HEALTH PROFESSIONS Abstracts of search results were reviewed to eliminate articles that clearly did not pertain to CE methods, cost-effectiveness, or educa - tional theory and to categorize the studies as informative, equivocal, or not informative of CE effectiveness. A wide range of designs were classified as informative, including randomized controlled trials, prospective cohort studies, observational studies, and studies with pre- and post-intervention assessment methodologies. Quantitative and qualitative approaches were included, and inclusion was not limited to studies with positive results. The most common reasons articles were classified as not informative were absence of a trial design, small sample size, and high likelihood of confounding fac - tors in the design that could affect outcomes. The two reviewers independently classified abstracts and full texts of the articles and then compared their classification results. Interreviewer reliability was greater than 80 percent, and discrepancies were resolved by a consensus process. A third reviewer verified the results classified as informative or equivocal in a final round of detailed assessment of the study design, populations, intervention, type of outcome, and conclusions for each article. Systematic reviews and meta- analyses are included in Table A-1; studies and articles are included in Table A-2.

OCR for page 147
149 APPENDIX A Table A-1 begins on the next page.

OCR for page 147
150 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-1 Summary of Systematic Reviews on Effectiveness of CE Methods Number of Studies, Inclusion Criteria, and Reference Purpose Databases Searched Reflection Ruth-Sahd, L. A. 2003. * Identify common Sample: 20 articles, 12 Reflective practice: themes that emerge doctoral dissertations, and A critical analysis of from data-based 6 books data-based studies studies Inclusion criteria: and implications for * Identify implications Delineated methodology nursing education. for reflective practice section; emphasis on in the field of nursing reflective practice in Journal of Nursing Education 42(11):488- education an education setting; 497. publication between 1992 and 2002; English language Databases: CINAHL, Dissertation Abstracts International, ERIC, PsychInfo Simulation Issenberg, S. B., Determine the Sample: 109 articles W. C. McGaghie, E. R. features and uses of Inclusion criteria: Petrusa, D. L. Gordon, high-fidelity medical Empirical study; use and R. J. Scalese. 2005. simulators that lead of a simulator as an Features and uses of to the most effective education assessment high-fidelity medical learning (high-fidelity or intervention; learner simulations that lead simulators are models, outcomes measured to effective learning: mannequins, or virtual quantitatively; A BEME systematic packages that utilize experimental or quasi- review. Medical Teacher realistic materials experimental design 27:10-28. and equipment and Databases: ERIC, incorporate feedback, Medline, PsychInfo, Web computerized control, of Science, Timelit or other advanced technology)

OCR for page 147
151 APPENDIX A Main Results Limitations * Conditions necessary for * No research on how unconscious reflection to be successful: knowledge is affected by reflective • ctive motivation A practice • afe learning environments S * Lack of hypothesis testing in • ime availability T reviewed studies * Students require guidance about how to practice reflection High fidelity simulators facilitate Heterogeneity of research designs, learning under certain conditions: educational interventions, outcome • epetitive practice R measures, and time frame precluded • sed in conjunction with U data synthesis using meta-analysis multiple learning strategies • ariety of clinical conditions V captured • ontrolled environment C where errors can be made and corrected • ndividualized learning where I participants are actively involved continued

OCR for page 147
152 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-1 Continued Number of Studies, Inclusion Criteria, and Reference Purpose Databases Searched Sutherland, L. M., Evaluate the Sample: 30 trials with 760 P. F. Middleton, A. effectiveness of participants Anthony, J. Hamdorf, surgical simulation Inclusion criteria: P. Cregan, D. Scott, and compared with other Randomized controlled G. J. Maddern. 2006. methods of surgical trial; assessing surgical Surgical simulation: training simulation; measures of A systematic review. surgical task performance Databases: Medline, Annals of Surgery 243(3):291-300. EMBASE, Cochrane Library, PsycINFO, CINAHL, Science Citation Index Reminders Balas, E. A., S. M. Determine the clinical Sample: 98 articles Austin, J. A. Mitchell, settings, types of reporting on 100 trials B. G. Ewigman, interventions, and Inclusion criteria: K. D. Bopp, and effects of studies in Randomized controlled G. D. Brown. 1996. randomized clinical trial (RCT); computerized The clinical value trials addressing the information intervention of computerized efficacy of clinical in the experimental information services. information systems group; effect measured on A review of 98 the process or outcome randomized clinical of care trials. Archies of Databases: Medline Family Medicine 5(5):271-278. Shea, S., W. Assess the overall Sample: 16 trials DuMouchel, and L. effectiveness of Inclusion criteria: Bahamonde. 1996. computer-based Randomized controlled A meta-analysis reminder systems in trial; computer-based of 16 randomized ambulatory settings reminder; control group controlled trials to directed at preventive received no intervention evaluate computer- care Databases: Medline, based clinical reminder Nursing and Allied systems for preventive Health database, care in the ambulatory Health Planning and setting. Journal of Administration database the American Medical Informatics Association 3(6):399-409.

OCR for page 147
153 APPENDIX A Main Results Limitations Computer simulation generally Insufficient evidence to evaluate types showed better results than of simulation because outcomes were no training at all but was not often not comparable across studies superior to standard training (e.g., surgical drills) or video simulation Patient and physician reminders, Many trials evaluate the effect of computerized treatment planners, information services on care processes and interactive patient education as opposed to patient outcomes can make a significant difference in managing care (P < 0.05) * Computer reminders Heterogeneity in study designs and the improved preventive practices ways in which results were presented for vaccinations, breast cancer screening, colorectal cancer screening, and cardiovascular screening * Computerized reminders did not improve preventive practices for cervical cancer screening continued

OCR for page 147
154 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-1 Continued Number of Studies, Inclusion Criteria, and Reference Purpose Databases Searched Audit and Feedback Jamtvedt, G., Review the effects of Sample: 118 trials J. M. Young, D. T. audit and feedback on Inclusion criteria: Kristoffersen, M. A. improving professional Randomized controlled O’Brien, and A. D. practice trials; utilized audit Oxman. 2006. Does and feedback; objective telling people what measures of provider they have been doing performance change what they do? Databases: Cochrane A systematic review Library of the effects of audit and feedback. Quality & Safety in Health Care 15(6):433-436. Multifaceted Interventions and Reviews of Multiple Methods Cheraghi-Sohi, S., * Review the efficacy Sample: 9 studies and P. Bower. 2008. of patient feedback on Inclusion criteria: Can the feedback of the interpersonal care Randomized controlled patient assessments, skills of primary care trials; published in brief training, or their physicians English; based on primary combination, improve * Review the efficacy care practitioners and the interpersonal of brief training (up their patients; utilized skills of primary to one working week patient feedback or brief care physicians? A in length) focused on training or a combination systematic review. the improvement of of these methods; interpersonal care outcome measure was a BMC Health Serices Research 8. patient-based assessment in change Databases: CENTRAL, Medline, EMBASE

OCR for page 147
155 APPENDIX A Main Results Limitations * Effects of audit and feedback on * Lack of a process evaluation improving professional practice embedded in trials are generally small to moderate * Few studies compare audit and * Effects of audit and feedback are feedback to other interventions likely to be larger when baseline adherence to recommended practice is low and audit and feedback are delivered more frequently and over longer periods of time Brief training as currently * Limited evidence on the effects of delivered is not effective patient-based feedback for changes in primary care physician behavior * Evidence is not definitive due to the small number of trials * Variation in training methods and goals * Lack of theory linking feedback to behavior change continued

OCR for page 147
156 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-1 Continued Number of Studies, Inclusion Criteria, and Reference Purpose Databases Searched Davis, D., M. Review, collate, and Sample: 14 studies A. O’Brien, N. interpret the effect Inclusion criteria: Freemantle, F. M. of formal continuing Randomized controlled Wolf, P. Mazmanian, medical education trial of formal didactic and A. Taylor-Vaisey. (CME) interventions on and/or interactive CME; 1999. Impact of physician performance >50% physicians formal continuing and health care Databases: RDRB, medical education: outcomes Cochrane Library, Medline Do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA 282(9):867-874. Forsetlund, L., A. To assess the effects of Sample: 81 trials Bjørndal, A. Rashidian, educational meetings involving more than G. Jamtvedt, on professional 11,000 health professionals M. A. O’Brien, F. practice and health Inclusion criteria: Wolf, D. Davis, J. care outcomes Randomized controlled Odgaard-Jensen, and trial of educational A. D. Oxman. 2009. meetings that reported Continuing education an objective measure of meetings and professional practice or workshops: Effects on health care outcomes professional practice Databases: Cochrane and health care Library outcomes. Cochrane Database Systematic Reiews (2):CD003030. Grimshaw, J., L. Identify, appraise, and Sample: 41 reviews Shirran, R. Thomas, synthesize systematic Inclusion criteria: G. Mowatt, C. Fraser, reviews of professional Interventions targeted L. Bero, R. Grilli, E. education or quality at health professionals; Harvey, A. Oxman, and assurance interventions reported measures of M. A. O’Brien. 2001. to improve quality of professional performance Changing provider care and/or patient outcomes; behavior: An overview study design included of systematic reviews explicit selection criteria of interventions. Databases: Medline, Medical Care 39(8 Suppl Healthstar, Cochrane 2):II2-II45. Library

OCR for page 147
157 APPENDIX A Main Results Limitations * Interactive CME sessions that * Limited number of randomized enhance participant activity controlled trials and settings limits and provide the opportunity to generalizability of findings practice skills can effect change * The comparability of CME in professional practice and, on interventions is debatable due to the occasion, health outcomes lack of comparability of reviewed * Didactic sessions did not interventions appear to be effective in changing physician performance * Educational meetings alone * Heterogeneity in study designs are not likely to be effective for and the ways in which results were changing behaviors presented * The effect of educational * Observed differences in changing meetings combined with other behaviors cannot be explained with interventions is most likely to be confidence small and similar to other types of CE, such as audit and feedback, and educational outreach visits * Passive approaches generally Lack of agreement within the research ineffective community on a theoretical or * Active approaches effective empirical framework for classifying under some circumstances interventions * Multifaceted interventions more likely to be effective than interventions with one method continued

OCR for page 147
216 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-2 Continued Sample, Method, Outcome Reference Study Purpose Measures, and Duration Martin, C. M., G. S. Test the hypothesis Sample: 499 patients in Doig, D. K. Heyland, that evidence-based 14 ICUs over an 11-month T. Morrison, and algorithms to improve period W. J. Sibbald. 2004. nutritional support in Method: Cluster Multicentre, cluster- the intensive care unit randomized controlled randomized clinical (ICU) would improve trial • xperimental group: trial of algorithms for patient outcomes E critical-care enteral introduction of and parenteral therapy evidence-based (ACCEPT). Canadian recommendations • ontrol group: no C Medical Association Journal 170(2):197-204. intervention Outcome measures: Days of enteral nutrition, length of stay in hospital, mortality rates, length of stay in ICU Duration: 11 months Monaghan, M. S., P. D. Determine whether a Sample: 25 pharmacists Turner, M. Z. Skrabal, CE approach to disease participating in a training and R. M. Jones. 2000. management training program Evaluating the format in diabetes mellitus Method: Cohort study and effectiveness is an effective means with pre- and post- of a disease state of improving both intervention design management training cognitive knowledge Outcome measures: program for diabetes. and confidence levels Scores on a pre- and post- of participants test examination; scores American Journal on a 15-item attitudinal of Pharmaceutical Education 64(2):181-184. questionnaire Duration: 14 months

OCR for page 147
217 APPENDIX A Description of Educational Method Findings Evidence-based recommendations * Patients in intervention ICUs received were introduced via in-service significantly more days of enteral education sessions, reminders nutrition (6.7 vs. 5.4 per 10 patient- by a local dietitian, posters, and days; p = 0.042), had a significantly academic detailing shorter mean stay in hospital (25 vs. 35 days; p = 0.003), and showed a trend toward reduced mortality (27% vs. 37%; p = 0.058) than patients in control ICUs * Mean stay in the ICU did not differ between control and experimental groups Traditional lectures and small- * Cognitive post-test scores (68.8%) group exercises in which improved significantly (p < 0.001) over participants obtained “hands- the pre-test scores (49.6%) on” information related to the * Post-test scores on all 15 attitudinal pharmacist’s role items significantly improved over pre- test scores (p < 0.012) continued

OCR for page 147
218 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-2 Continued Sample, Method, Outcome Reference Study Purpose Measures, and Duration Naunton, M., G. M. Assess a Sample: All patients Peterson, G. Jones, comprehensive admitted to the Royal G. M. Griffin, and educational program Hobart Hospital, M. D. Bleasel. aimed at increasing Australia; all physicians 2004. Multifaceted the use of osteoporosis and pharmacists in 2 educational program preventive therapy in regions in Australia increases prescribing of patients prescribed Method: Controlled trial • xperimental group: preventive medication long-term oral E for corticosteroid corticosteroids geographic region induced osteoporosis. received multifaceted educational program Journal of Rheumatology • ontrol group: 31(3):550-556. C geographic region received no intervention Outcome measures: Evaluation feedback from GPs and pharmacists; drug utilization data Duration: 17 months Pronovost, P. J., Describe the design Sample: 99 ICUs across S. M. Berenholtz, C. and lessons learned the state of Michigan over Goeschel, I. Thom, from implementing 24 months S. R. Watson, C. G. a large-scale patient Method: Cohort study of Holzmueller, J. S. safety collaborative ICU teams Lyon, L. H. Lubomski, and the impact of Outcome measures: D. A. Thompson, D. an intervention on Improvements in safety Needham, R. Hyzy, teamwork climate in culture scores using a R. Welsh, G. Roth, J. intensive care units teamwork questionnaire; Bander, L. Morlock, adherence to evidence- and J. B. Sexton. 2008. based interventions for Improving patient ventilated patients safety in intensive care Duration: 17 months units in Michigan. Journal of Critical Care 23(2):207-221.

OCR for page 147
219 APPENDIX A Description of Educational Method Findings All GPs and pharmacies in the * Use of preventive therapy increased study area were sent educational from 31% of admitted hospital patients materials and guidelines; received taking corticosteroids to 57% post- academic detailing visits and intervention (p < 0.0001) reminders; and were provided * Significant increase in the use of educational magnets for their preventive therapy in the intervention patients region over the control region (p < 0.01) * Collaborative project included * Teamwork climate improved from group meetings and conference baseline to post-intervention calls to share best practices and (t(71) = –2.921, p < 0.005) evaluate performance * Post-intervention: 46% had >60% * Partnership between hospital consensus of good teamwork; pre- leadership, ICU improvement intervention: 17% of ICUs had >60% teams, and ICU staff to identify consensus of good teamwork and resolve barriers * Daily goals communication toolkits for staff education, redesign of work processes, and support of local opinion leaders continued

OCR for page 147
220 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-2 Continued Sample, Method, Outcome Reference Study Purpose Measures, and Duration Rashotte, J., M. Examine the impact Sample: 23 pediatric Thomas, D. Grégoire, and sustained change critical care nurses in a and S. Ledoux. 2008. of a 2-part, unit-based Canadian pediatric ICU Implementation of a multiple intervention Method: Cohort study two-part unit-based on the use by Outcome measures: multiple intervention: pediatric critical care Before-and-after Moving evidence- nurses of guidelines measures of frequency based practice into for pressure-ulcer of use of interventions action. Canadian Journal prevention as documented in patient records and by of Nursing Research 40(2):94-114. observation Duration: 6 months Richards, D., L. Toop, Determine whether a Sample: 230 GPs in urban and P. Graham. 2003. peer-led small-group New Zealand Do clinical practice educational program Method: Retrospective education groups is an effective tool analysis of a controlled result in sustained in changing practice trial • xperimental group: change in GP when added to E prescribing? Family audit and feedback, audit and feedback, Practice 20(2):199-206. academic detailing, individual academic and educational detailing, educational bulletins bulletins, and peer- led group academic detailing sessions • ontrol group: C audit and feedback, academic detailing, and educational bulletins Outcome measure: Targeted prescribing for 12 months before and 24 months after education sessions Duration: 36 months

OCR for page 147
221 APPENDIX A Description of Educational Method Findings * Part I targeted individuals with Significant change in implementation independent and group learning of 2 of 11 recommended practices activities: laminated pocket following both interventions (p < 0.001) guides, bedside decision-making algorithm * Part II incorporated local and organizational strategies: unit champions, bedside coaching, development of standards * Clinical practice education * Peer-led small-group discussions groups met monthly had a sustained, positive effect on * GP-led discussion of evidence- prescribing behavior that was in based topics addition to any effect of the other * Individual prescribing data educational methods (mean effect size provided to each GP = 1.20) * Mean duration of significant effect was 14.5 months (CI: 95%) continued

OCR for page 147
222 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-2 Continued Sample, Method, Outcome Reference Study Purpose Measures, and Duration Saini, B., L. Smith, C. Test the effect of Sample: 27 pharmacists Armour, and I. Krass. an educational providing asthma care to 2006. An educational intervention on 102 patients in Australia intervention to pharmacist satisfaction Method: Controlled trial • xperimental train community and practice behavior E pharmacists in as well as patient group: educational providing specialized outcomes intervention • ontrol group: no asthma care. C intervention American Journal Outcome measures: of Pharmaceutical Education 70(5):118-126. Participant reactions gauged using a questionnaire; asthma severity; peak flow indices; medication costs per patient Duration: 6 months Schneeweiss, S., Evaluate the Sample: 48 emergency and S. Ratnapalan. effectiveness of a staff physicians, fellows, 2007. Impact of a sedation course in and residents in a multifaceted pediatric improving physicians’ pediatric emergency sedation course: Self- knowledge of pediatric department directed learning procedural sedation Method: Randomized versus a formal guidelines, relative to controlled trial • xperimental group: continuing medical self-directed learning E education course to self-directed learning • ontrol group: formal, improve knowledge of C sedation guidelines. 4-hour course Outcome measures: Canadian Journal of Scores on multiple choice Emergency Medical Care 9(2):93-100. pre- and post-intervention exam Duration: 2 weeks

OCR for page 147
223 APPENDIX A Description of Educational Method Findings Self-directed learning, small-group * Significant reduction in asthma learning, and workshops with case severity in the experimental group studies in addition to asthma care (p < 0.001) vs. the control group training provided in a lecture * In the experimental group, peak flow indices improved from 82.7% at baseline to 87.4% (p < 0.0010) at the final visit * Significant reduction in defined daily dose of albuterol used by patients (p < 0.015) * The 4-hour course consisted Control group’s median exam score of small-group and didactic (83.3%; range: 75.8-96.5%) was instruction with case studies significantly higher (p < 0.0001) than * The self-directed group received median exam score of the experimental a package with learning objectives, group (73.3%; range: 43.5-86.6%) guidelines, a pocket card, and reading materials continued

OCR for page 147
224 REDESIGNING CE IN THE HEALTH PROFESSIONS TABLE A-2 Continued Sample, Method, Outcome Reference Study Purpose Measures, and Duration Scholes, D., L. Evaluate an Sample: 23 primary care Grothaus, J. McClure, intervention to clinics; 3,509 sexually R. Reid, P. Fishman, C. increase guideline- active females ages 14-25 Sisk, J. E. Lindenbaum, recommended Method: Randomized B. Green, J. Grafton, Chlamydia screening controlled trial • xperimental group: and R. S. Thompson. E 2006. A randomized enhanced guideline trial of strategies to intervention • ontrol group: increase Chlamydia C screening in young standard guideline women. Preentie implementation Medicine 43(4):343-350. instructions Outcome measures: Post- intervention Chlamydia testing rates Duration: 27 months Young, J. M., C. Evaluate a Sample: 60 family D’Este, and J. E. Ward. multifaceted, practice- physicians in Australia 2002. Improving based intervention Method: Cluster family physicians’ involving audit, randomized controlled use of evidence-based feedback, and trial • xperimental smoking cessation academic detailing E strategies: A cluster to improve family group: multifaceted randomization trial. physician smoking intervention • ontrol group: no cessation advice C Preentie Medicine 35(6):572-583. intervention Outcome measures: Delivery of smoking cessation advice determined by patient recall, physician report, and medical record audit; utilization of nicotine replacement therapies Duration: 6 months NOTE: NA = Not applicable.

OCR for page 147
225 APPENDIX A Description of Educational Method Findings The enhanced guideline group * Enhanced intervention did not used clinic-based opinion leaders, significantly affect Chlamydia testing individual measurement and (OR = 1.08; 95% CI: 0.92-1.26; p = 0.31) feedback, exam room reminders, * Testing rates increased among women and chart prompts making preventive care visits in intervention vs. control clinics * Audit and feedback conducted * Significant increase in the by a medical peer experimental group over the * Medical record prompt in the control group in the use of nicotine form of Post-it notes on medical replacement gum (p = 0.0002) and records patches (p = 0.0056) * Provision of additional resources * No significant differences between for physicians and patients groups in smokers’ recall or documentation in medical record of specific cessation advice

OCR for page 147