Measuring the Impact of Interprofessional Education on Collaborative Practice and Patient Outcomes (2015)

Appendix A

Review: Measuring the Impact of Interprofessional Education (IPE) on Collaborative Practice and Patient Outcomes

Valentina Brashers, M.D.; Elayne Phillips, M.P.H., Ph.D., R.N.; Jessica Malpass, Ph.D., R.N.; John Owen, Ed.D., M.Sc.

BACKGROUND

Although the complexity of patient care demands that health care teams collaborate effectively, there remains a paucity of high-quality research that measures the impact of interprofessional education (IPE) on practice processes and patient outcomes. A recent Cochrane review found a total of 15 articles published between 1999 and 2011 whose methodology met their stringent criteria for inclusion (Reeves et al., 2013). While those studies did provide evidence that IPE interventions can produce positive outcomes, there remains a need to identify best practices for research that effectively link IPE interventions with measurable changes in practice processes and patient outcomes.

OBJECTIVES

The two objectives of this review are to

• examine the currently best-available methods used for measuring the impact of IPE on collaborative practice, patient outcomes, or both; and
• describe the challenges to conducting high-quality research that seeks to link IPE interventions with measurable changes in practice and patient outcomes.

METHODS

This review focuses on studies reviewed in the Reeves and colleagues (2013) Cochrane review, and on any national and international studies published from January 2011 to July 2014.

Criteria for Considering Studies for This Review

Types of Studies

This review includes randomized controlled trials (RCTs), controlled before-and-after (CBA) studies, interrupted time series (ITS) studies, and uncontrolled before-and-after (BA) studies.

Types of Participants

This review includes various types of health care professionals (physicians, dentists, chiropractors, midwives, nurses, nurse practitioners, physical therapists, occupational therapists, respiratory therapists, speech and language therapists, pharmacists, technicians, psychotherapists, and social workers).

Types of Interventions

As defined by Reeves and colleagues (2013, p. 5), “An IPE intervention occurs when members of more than one health or social care (or both) profession learn interactively together, for the explicit purpose of improving interprofessional collaboration or the health/well-being (or both) of patients/clients. Interactive learning requires active learner participation, and active exchange between learners from different professions.”

Types of Outcome Measures

Outcome measures include

• objectively measured patient/client outcomes (disease incidence; morbidity, mortality, readmission, and complication rates; length of stay; patient/family satisfaction);
• objectively measured health care process measurements (changes in efficiency [resources, time, cost]; teamwork; approach to patient care or follow-up); and
• subjective self-reported outcomes, included only when objective measures were also reported.

Search Methods

For this review, the following search methods were used:

• A search was conducted of Ovid, PubMed, and CINAHL (Cumulative Index to Nursing and Allied Health Literature) via MeSH (Medical Subject Headings) terms “Interprofessional education AND (Cochrane terms OR Quality OR Clinical Outcomes OR Patient Outcomes OR Cost Benefit OR Quality OR Patient Safety OR Patient Satisfaction OR Provider Satisfaction OR Morbidity)” from January 2011 to the present.
• A keyword search from PubMed using “interprofessional education” or “team training” in the title/abstract (limit 2008-July 2014) was also conducted.
• Articles were hand-pulled from the Reeves et al. (2013) Cochrane review.

Data Collection and Analysis

Two of the review authors (EKP and JKM) jointly reviewed 2,347 abstracts retrieved by the searches to identify all those that indicated

• an IPE intervention was implemented;
• health care clinicians of various backgrounds were trained; and
• patient outcomes (patient safety, patient satisfaction, quality of care, cost, clinical outcomes, community health outcomes, etc.) and/or provider outcomes (provider satisfaction, measures of collaborative practice, communication) were reported.

Abstracts were excluded if

• the interprofessional intervention lacked a concrete educational component;
• interprofessional activities involved only students;
• learning outcomes were the only outcomes measured; or
• reported outcomes included only feelings, beliefs, attitudes, or perceptions.

Forty-seven studies were identified from the abstract search as potentially meeting these inclusion criteria. The full text of each of these articles as well as each of the 15 articles pulled from the Cochrane review was independently reviewed by three of the review authors (EKP, JKM, VLB). An appraisal form was developed specifically for this review that evaluated the studies for

• type of study (RCT, CBA, ITS, or BA study with historical control, contemporaneous control, or no control);
• outcome measures;
• outcome tool;
• sample size and composition;
• setting;
• type of IPE intervention; and
• findings (a brief overview of findings is included in a detailed table in the annex at the end of this appendix, but findings are not discussed as part of this review, which is focused on methodology).

These data were entered into a spreadsheet, and any disagreements and uncertainties were resolved by discussion. These studies were then given an overall rating based on the following definitions:

The following descriptions were used as general guidelines for rating:

A - HIGH

• Consistent, generalizable results
• Sufficient sample size
• Adequate control
• Definitive conclusions
• Consistent recommendations based on a comprehensive literature review that includes thorough reference to scientific evidence

B - GOOD

• Reasonably consistent results
• Sufficient sample size for the study design
• Some control
• Fairly definitive conclusions

• Reasonably consistent recommendations based on a fairly comprehensive literature review that includes some reference to scientific evidence

C - LOW

• Little evidence with inconsistent results
• Insufficient sample size for study design
• Conclusions cannot be drawn

MAIN RESULTS

In addition to the 15 studies from the Cochrane review, 24 additional studies met all criteria and were included in this review. Table A-1 presents an overview of the results of the review.

Study Types

Randomized Controlled Trials: Three new RCTs (Hoffmann et al., 2014; Nurok et al., 2011; Riley et al., 2011) were added to the seven RCTs described in the 2013 Cochrane review (Reeves et al., 2013). These three studies suffered from many of the same methodologic limitations noted for the studies discussed in the Cochrane review, such as the lack of concealed allocation, inadequate blinding in the assessment of outcomes, and evidence of selective outcome reporting. These studies were also characterized by additional sources of error that are common in evaluating educational programs (Sullivan, 2011), including differences in the quality of the education intervention (e.g., type of learners trained, variation in learner and instructor experience and training) and difficult-to-measure endpoints.

Controlled Before-and-After Studies: No new CBAs were added during this review. As described in the 2013 Cochrane review, the CBAs were characterized by many of the same limitations described for RCTs, except that there was often a more well-documented effort to ensure that baseline characteristics of the intervention and control groups were similar.

Interrupted Time Series Studies: One additional ITS (Pettker et al., 2009) was added to those listed in the 2013 Cochrane Review. The primary strength of this study was the documentation of long-term changes in outcomes. There was also a sequential introduction of interventions in an effort to isolate the effect of the IPE intervention from numerous other practice changes introduced during the study period. However, while the trend in outcomes was calculated on a monthly basis, it is not clear from the analysis whether the team training alone significantly affected outcome trends.

Before-and-After Studies: The 20 BA studies that were included in this review were carefully chosen for having used credible research methods

TABLE A-1 Overview of Results

NOTE: Detailed results are presented in Annex A-1 at the end of this appendix.

based on our rating scale (i.e., IIB or IIC, as defined earlier). These studies were highly diverse in their outcome measures, measurement tools, setting, number and composition of participants, presence of historical controls, and type and quality of IPE interventions. Two BA studies that were rated IIC were included because of the quality of their design, but their interpretation of the results went beyond what the data could support (Capella et al., 2010; Pingleton et al., 2013). One study rated IIC was included because it was conducted in an unusual but important care setting (Lang et al., 2010).

Outcome Measures

Studies chosen for inclusion in this review reported objective and measurable outcomes. Patient outcome measures addressed many important issues in care quality, such as number of adverse events, specific indices of disease progression, length of stay, improvement in symptoms, morbidity, and mortality as derived from review of the clinical database for BA IPE interventions. Two studies assessed provider-with-patient communication skills (Brown et al., 1999; Helitzer et al., 2011). Only four studies measured patient satisfaction (Banki et al., 2013; Brown et al., 1999; Campbell et al., 2001; Morey et al., 2002), and one measured family satisfaction (Shaw et al., 2014).

Practice outcome measures most often addressed clinical decision making, behaviors related to patient safety, care efficiency, error reporting, adherence to guidelines, use of checklists, organization of care, and specific care competencies. Nine studies included objective observation of teamwork skills in the actual delivery of care (Bliss et al., 2012; Capella et al., 2010; Halverson et al., 2009; Mayer et al., 2011; Morey et al., 2002; Nurok et al., 2011; Patterson et al., 2013; Steinemann et al., 2011; Weaver et al., 2010), and two studies reported observed team behaviors in the simulated setting in addition to the care delivery site (Knight et al., 2014;

Patterson et al., 2013). Only one study directly measured changes in practice costs (Banki et al., 2013).

Several studies measured outcomes over many months and even years to assess for sustained changes in patient or provider outcomes (Armour Forse et al., 2011; Helitzer et al., 2011; Mayer et al., 2011; Morey et al., 2002; Pettker et al., 2009; Phipps et al., 2012; Pingleton et al., 2013; Rask et al., 2007; Sax et al., 2009; Thompson et al., 2000b; Wolf et al., 2010). For these studies, improvements were sustained over the study period, although some reported partial decay over time. Another complication is that while these studies included graphics that listed outcomes at multiple time points before and after the IPE intervention, only two were actual ITS studies (Hanbury et al., 2009; Pettker et al., 2009). One based its conclusions on the single lowest and highest pre- and postintervention values (Pingleton et al., 2013), and the rest based their conclusions on the average of before and after outcomes.

Patient and Practice Outcome Tools

The most commonly used measurement tool for both provider and patient outcomes involved chart review/clinical database access for retrieving specific patient data, error/adverse event/incident reporting, and OR reports. Most observational studies used validated tools such as the Trauma Oxford Non-Technical Skills scale (Steinemann et al., 2011), Teamwork Evaluation of Non-Technical Skills tool (Mayer et al., 2011), American College of Surgeons National Surgical Quality Improvement Program tool (Bliss et al., 2012), Behavioral Markers for Neonatal Resuscitation Scale (Patterson et al., 2013), Medical Performance Assessment Tool for Communication and Teamwork (Weaver et al., 2010), and Trauma Team Performance Observation Tool (Capella et al., 2010). One study used the validated Roter Interaction Analysis System provider–patient communication tool (Helitzer et al., 2011). Shaw and colleagues (2014) used a validated Family Satisfaction in the ICU tool to link teamwork with family-perceived provider communication. Patient satisfaction was measured using the Press Ganey Patient Satisfaction Tool in one study (Banki et al., 2013), and the Patient Safety Satisfaction Survey in another (Campbell et al., 2001). A blended tool taken from several sources was used in one study (Morey et al., 2002), and a tool designed by the researchers was used in another (Brown et al., 1999).

Sample Size and Composition of Providers Trained

All but 3 of the 10 RCTs (Brown et al., 1999; Helitzer et al., 2011; Nurok et al., 2011) and 1 CBA (Weaver et al., 2010) described in this

updated review had large sample sizes involving multiple practice sites. For example, one cluster RCT trained more than 1,300 providers whose outcomes were measured in 15 military and civilian hospitals across multiple states (Nielsen et al., 2007). Sample size in the ITS and BA studies varied widely, and several studies failed to report a specific number of participants trained (Armour Forse et al., 2011; Knight et al., 2014; Nurok et al., 2011; Theilen et al., 2013). The composition of providers trained varied significantly. All studies included nurses (either registered nurse [RN] or advanced practice registered nurses [APRN]), and only two did not include physicians (Lang et al., 2010; Rask et al., 2007); however, the specific number of participating physicians often was not reported. Four studies specifically listed doctorate of pharmacy (PharmD) participation, eight reported therapist participation, nine reported technician participation, and four reported social worker participation. Other reported participants included nutritionist, housekeeping, scheduler, physician assistant, unit secretary, chaplain, psychologist, and security officer. The accuracy of these counts is limited because some of these participants may have been included in a broad description such as “ancillary personnel,” “support personnel,” “OR team,” “health care team,” and “health care assistants.” The number of patient and provider outcomes measured in each study also varied widely. For example, one study reported patient outcomes for only 21 patients (Helitzer et al., 2011), whereas another reported outcomes for 21,409 patients (Thompson et al., 2000a).

Setting

This review included studies reflecting a broad range of locales, including inpatient and outpatient settings. Interestingly, there were similar numbers of U.S. studies conducted in community hospitals and practices (14) and in academic health centers (15). The OR was the most commonly studied academic setting, accounting for six studies (Armour Forse et al., 2011; Bliss et al., 2012; Halverson et al., 2009; Nurok et al., 2011; Sax et al., 2009; Wolf et al., 2010). Acute care settings accounted for 10 of the 13 U.S. studies conducted in the community, while primary care clinics (including mental health) accounted for only 3 studies (Taylor et al., 2007; Thompson et al., 2000b; Young et al., 2005). The Veterans Health Administration hosted three large studies (Neily et al., 2010; Strasser et al., 2008; Young-Xu et al., 2011). Five international studies were included (Britain = three, Germany = one, Mexico = one). An unusual setting for reporting team training was U.S. combat operations in Iraq. Finally, one nursing home and one free-standing MRI facility were included.

Type of IPE Intervention

The type of IPE intervention varied widely. The two most cited interventions were Crew Resource Management (n = 9) and TeamSTEPPS (n = 6) (see Table A-2 in Annex A-1); however, these were almost always implemented in a modified format. Several other standardized programs were used (see Annex A-1), but in-house-designed programs were the most common type of IPE intervention. The descriptions of these programs varied from general and nonspecific to highly detailed. Several studies combined teamwork training with training focused on selected care outcomes, such as prevention of venous thromboembolism (Pingleton et al., 2013; Tapson et al., 2011) or best practices in diabetes management (Barceló et al., 2010; Janson et al., 2009; Taylor et al., 2007).

Overview of Findings

Learner teamwork competencies and communication skills were improved in most of the observational studies. Morbidity and mortality were directly measured in some of the larger studies, especially those focused on the OR (Armour Forse et al., 2011; Bliss et al., 2012; Neily et al., 2010; Young-Xu et al., 2011) and labor and delivery (Riley et al., 2011). One study looked at teamwork during resuscitations in the ICU and found significant improvements in survival (Knight et al., 2014). Care quality was improved in the majority of studies included in this review, most often reported as changes in practice processes, such as adherence to best practices, use of checklists, and participation in briefings. For most of these studies, team training was implemented as one part of a more comprehensive approach to practice changes (e.g., procedure manuals, mandatory OR briefings, checklists, new reporting systems). Improvements in specific patient care quality outcomes, such as HgbA1C, cholesterol, blood pressure, and mobility after stroke, were reported in four studies (Barceló et al., 2010; Janson et al., 2009; Strasser et al., 2008; Taylor et al., 2007). Patient safety outcomes were also improved in most studies as measured by decreases in adverse outcomes (Bliss et al., 2012; Mayer et al., 2011; Patterson et al., 2013; Pettker et al., 2009; Phipps et al., 2012; Pingleton et al., 2013; Riley et al., 2011) and error reporting (Hoffmann et al., 2014). A reduction in error rates was reported in two studies (Deering et al., 2011; Morey et al., 2002). Patient satisfaction was improved in two studies (Banki et al., 2013; Campbell et al., 2001) and unchanged in two others (Brown et al., 1999; Morey et al., 2002). Care efficiency improvements were measured in several studies (Banki et al., 2013; Capella et al., 2010; Wolf et al., 2010), and direct improvements in costs were reported in one study (Banki et al., 2013).

Overview of Methodologic Limitations

The following methodologic limitations were noted:

• for controlled studies, inability to control for differences between control and intervention study groups, lack of concealed allocation, inadequate blinding in the assessment of outcomes, evidence of selective outcome reporting, differences in the type and quality of the educational intervention, and difficult-to-measure endpoints;
• inadequate control for multiple other simultaneous practice changes that affect patient outcomes;
• lack of adequate timeline to document sustained changes in practice or patient outcomes;
• paucity of evidence for patient-centered changes in care;
• lack of studies addressing cost outcomes (business case);
• poor description of participants (how many, which disciplines);
• lack of clarity as to whether those trained together actually worked as a team in the practice setting;
• lack of evidence that teamwork training resulted in improved teamwork behaviors prior to assessment of clinical outcomes; and
• lack of adequate description of the type and quality of the IPE intervention as significant variables influencing outcomes.

DISCUSSION

The number of studies that link IPE with changes in practice and patient outcomes is growing. However, methodologic limitations continue to confound interpretation and generalization of the results.

While the RCT is considered the “gold standard” methodology for clinical studies, for educational research, they (like CBAs) suffer from less well-matched controls resulting from differences both within and among care delivery settings. Smaller studies are particularly vulnerable to the impact of differences among study groups. These barriers can be minimized to some degree by large-scale studies in which many clinician learners and practice settings can be randomized; however, differences among study sites likely remain, limiting meaningful comparisons in measured outcomes. Other methodological challenges related to participant allocation, investigator blinding, and variations in the quality of the IPE intervention cannot be completely avoided (Sullivan, 2011). As was stated in an Institute of Medicine (IOM) report on continuing medical education, “While controlled trial methods produce quantifiable end points, they do not fully explain whether outcomes occur as a result of participation in CE [continuing education], thus, a variety of research methods may be necessary” (IOM, 2010, p. 39).

Regardless of the study type, the implementation of other practice changes during the course of the study makes it difficult to ascribe documented changes in outcomes directly to the IPE intervention alone. One can argue that a combination of teamwork training and other practice changes would likely be even more effective in improving care (Weaver et al., 2014). Nevertheless, it is still important to better understand the independent and relative impact of teamwork training given the challenges inherent in scheduling and appropriately implementing effective IPE interventions.

The choice of outcome measures and measurement tools is a complex decision. Most of the studies in this review used retrieval of data from medical records to identify patient and practice outcome measures. While broad justifications are included in the background or introduction portions of these articles, few of the investigators make clear why particular outcome measures were chosen. At least three limitations should be considered when interpreting these data. First, studies using aggregate data collected from medical records pre- and postintervention are less likely to account for other changes in care unrelated to the IPE intervention than are studies in which specific cohorts of patients are carefully monitored and compared over time. Second, as described in the 2013 Cochrane review (Reeves et al., 2013), careful reading suggests that at least some studies engaged in selective reporting of outcomes, which limits complex interpretation of the effectiveness of the intervention. Finally, it is of concern that only four studies in this review focused on patient and family satisfaction. While objective measurement of practice and patient outcomes is essential, a patient-centered approach requires a more focused and nuanced tool for linking teamwork-based changes in care with the patient and family experience. Patients should not only be safe and well cared for, but should also feel safe and well cared for, and it is important to identify those teamwork factors that best promote that perception. Future research should focus on developing IPE interventions that teach patient-centered skills along with those skills needed to affect objective outcomes.

As with any education intervention, there is concern that the impact on knowledge, skills, and behavior will decay over time. All 11 of the long-term studies included in this review document a sustained impact on provider or patient outcomes, although the effects tended to decay over time. This is consistent with a 2007 comprehensive analysis of the effectiveness of continuing medical education (CME) in imparting knowledge and skills, changing attitudes and practice behavior, and improving clinical outcomes (Marinopoulos et al., 2007). While fewer than half of the studies in that analysis measured outcomes beyond 30 days postintervention, those that did found sustained changes in practice behaviors. Additional studies are needed to explore the best timing, content, format, and length of IPE interventions to provide the most sustained impact.

One challenge is that care-delivery in most institutions does not occur in the context of stable teams composed of professionals who train and work together in an intact group. Teams are most often ad hoc and may change on a weekly, daily, or even hourly basis for any given patient. With the exception of some of the operating room studies in this review, it is not clear whether the teams that trained together actually functioned as a team at the bedside. Although one meta-analysis suggests that improvements in team performance with team training are similar for intact and ad hoc teams (Salas et al., 2008), it may be that a team needs a “critical mass” of trained members in order to function effectively. Furthermore, while many of the studies provide the overall number of trainees and a list of participating professions, few document whether the teams that participated in any specific training session actually represented an appropriate number of trainees from each profession. These limitations suggest that the demonstration of improved teamwork skills in the actual clinical setting is an essential step before measuring practice or patient outcomes. While the Hawthorne effect is a consideration, there is evidence that observation in the clinical setting does not result in prolonged contamination of the data (Hohenhaus et al., 2008; Schnelle et al., 2006). Observation of actual changes in team behaviors provides stronger evidence for the link between team training and measureable changes in practice and patient outcomes (Morey and Salisbury, 2002).

It is interesting to note that few of the studies in this review gave in-depth consideration to the influence of IPE intervention implementation factors (timing, content, format, length, instructor and learner preparation) on outcomes. Even when researchers used well-respected programs such as Crew Resource Management and TeamSTEPPS, the programs were frequently modified for logistical reasons. It is impossible to know how the modifications affected the outcomes; for that reason, the studies cannot be compared as if the same intervention were tested. The majority of investigators created IPE interventions of their own design. Many of the most effective IPE interventions in this review combined team training with “taskwork” training related to best practices for a specific patient population (e.g., diabetes patients). Salas and colleagues (2008) report that both teamwork and taskwork are effective in improving outcomes; however, the relative emphasis of each in the interventions in this study is not well described. IPE interventions that are created by local stakeholders to address institutional priorities have the advantage of eliciting increased participation by providers, integrating faculty development, and allowing for assessment of specific teamwork behaviors and competencies (Owen et al., 2012), but they often vary widely in scope, content, format, and duration. There is a great deal of information available to inform the design and implementation of continuing IPE programs. Core principles that should

be applied include ensuring adequate incorporation of effective theoretical foundations, adult learning principles, interprofessional learning objectives, and strategies for increased knowledge transfer and retention (IOM, 2013; Merriam and Leahy, 2005; Owen et al., 2014; Reeves and Hean, 2013). Yet for many of the studies in this review, it is not clear whether evidence-based principles were applied to the design and implementation of the IPE interventions. More guidance may be needed to help investigators choose the best approach.

Given the many methodologic limitations of these studies, outcome data must be interpreted carefully. Yet it is important to note that the majority of studies in this review found improvements in care processes, patient outcomes, or both. While the diversity of approaches and methodologic limitations make it difficult to draw clear conclusions with respect to best practices for linking IPE with patient and practice outcomes, this limited review suggests that the characteristics of those studies with the most significant improvements in outcomes include

• high learner participation rates or self-selection to intervention group,
• combination of IPE and goal-specific education (teamwork + taskwork),
• combination of IPE and other changes in practice processes,
• use of simulation and videotaping,
• repetition of IPE interventions with regular feedback to learners, and
• correlation of IPE intervention with observed and measurable changes in teamwork behaviors/skills.

While this review has attempted to describe the limitations of current research methodologies so that recommendations for future research can be made, it is important to recognize that many of the studies in this review represent high-quality groundbreaking research in a highly complex area of investigation. As stated in the 2010 IOM report, “In health care settings, it may remain difficult to measure dependent variables because linking participation in CE to changes in the practice setting is a complex process that cannot easily be tracked using current methods” (IOM, 2010, p. 35). In a recent synthesis of the team-training research literature, Weaver and colleagues (2014) note that research in this area is still plagued with limitations, including “small sample sizes, weak study design and limited detail regarding the team training curriculum or implementation strategy.” When research limitations are compounded by the complexities of bringing together professionals from diverse backgrounds and perspectives, it is unsurprising that much work remains to be done.

AUTHORS’ CONCLUSIONS

Based on this extensive review, it is the authors’ opinion that key recommendations necessary for meaningful research linking IPE interventions with sustained changes in practice and patient outcomes include the following:

• Conduct large-scale controlled studies that minimize confounding variables; when this is not possible, consideration should be given to conducting well-designed ITS studies with careful monitoring of the study cohort to account for other variables that may impact outcomes.
• Use objective, relevant provider and patient outcome measures chosen prospectively, and report all results.
• Implement the IPE intervention at a defined time and adequately isolated from other practice changes.
• Collect pre- and postintervention data at multiple time points over several years.
• Include in patient outcome data an assessment of patient-centered team-based care.
• Observe and measure team behaviors in the actual practice setting before collecting practice or patient outcome data.
• Ensure that the IPE intervention is evidence- and competency-based, builds on sound theoretical underpinnings, is conducted by well-trained instructors, and is provided to the proper mix of learners.

REFERENCES

Armour Forse, R., J. D. Bramble, and R. McQuillan. 2011. Team training can improve operating room performance. Surgery 150(4):771-778.

Banki, F., K. Ochoa, M. E. Carrillo, S. S. Leake, A. L. Estrera, K. Khalil, and H. J. Safi. 2013. A surgical team with focus on staff education in a community hospital improves outcomes, costs and patient satisfaction. American Journal of Surgery 206(6):1007-1014; discussion 1014-1015.

Barceló, A., E. Cafiero, M. de Boer, A. E. Mesa, M. G. Lopez, R. A. Jimenez, A. L. Esqueda, J. A. Martinez, E. M. Holguin, M. Meiners, G. M. Bonfil, S. N. Ramirez, E. P. Flores, and S. Robles. 2010. Using collaborative learning to improve diabetes care and outcomes: The VIDA project. Primary Care Diabetes 4(3):145-153.

Bliss, L. A., C. B. Ross-Richardson, L. J. Sanzari, D. S. Shapiro, A. E. Lukianoff, B. A. Bernstein, and S. J. Ellner. 2012. Thirty-day outcomes support implementation of a surgical safety checklist. Journal of the American College of Surgeons 215(6):766-776.

Brown, J. B., M. Boles, J. P. Mullooly, and W. Levinson. 1999. Effect of clinician communication skills training on patient satisfaction. A randomized, controlled trial. Annals of Internal Medicine 131(11):822-829.

Campbell, J. C., J. H. Coben, E. McLoughlin, S. Dearwater, G. Nah, N. Glass, D. Lee, and N. Durborow. 2001. An evaluation of a system-change training model to improve emergency department response to battered women. Academic Emergency Medicine 8(2):131-138.

Capella, J., S. Smith, A. Philp, T. Putnam, C. Gilbert, W. Fry, E. Harvey, A. Wright, K. Henderson, and D. Baker. 2010. Teamwork training improves the clinical care of trauma patients. Journal of Surgical Education 67(6):439-443.

Deering, S., M. A. Rosen, V. Ludi, M. Munroe, A. Pocrnich, C. Laky, and P. G. Napolitano. 2011. On the front lines of patient safety: Implementation and evaluation of team training in Iraq. Joint Commission Journal on Quality & Patient Safety 37(8):350-356.

DeVita, M. A., J. Schaefer, J. Lutz, H. Wang, and T. Dongilli. 2005. Improving medical emergency team (MET) performance using a novel curriculum and a computerized human patient simulator. Quality & Safety in Health Care 14(5):326-331.

Halverson, A. L., J. L. Andersson, K. Anderson, J. Lombardo, C. S. Park, A. W. Rademaker, and D. W. Moorman. 2009. Surgical team training: The Northwestern Memorial Hospital experience. Archives of Surgery 144(2):107-112.

Hanbury, A., L. Wallace, and M. Clark. 2009. Use of a time series design to test effectiveness of a theory-based intervention targeting adherence of health professionals to a clinical guideline. British Journal of Health Psychology 14(Pt. 3):505-518.

Helitzer, D. L., M. Lanoue, B. Wilson, B. U. de Hernandez, T. Warner, and D. Roter. 2011. A randomized controlled trial of communication training with primary care providers to improve patient-centeredness and health risk communication. Patient Education and Counseling 82(1):21-29.

Hoffmann, B., V. Muller, J. Rochon, M. Gondan, B. Muller, Z. Albay, K. Weppler, M. Leifermann, C. Miessner, C. Guthlin, D. Parker, G. Hofinger, and F. M. Gerlach. 2014. Effects of a team-based assessment and intervention on patient safety culture in general practice: An open randomised controlled trial. BMJ Quality and Safety 23(1):35-46.

Hohenhaus, S. M., S. Powell, and R. Haskins. 2008. A practical approach to observation of the emergency care setting. Journal of Emergency Nursing 34(2):142-144.

IOM (Institute of Medicine). 2010. Redesigning continuing education in the health professions. Washington, DC: The National Academies Press.

IOM. 2013. Interprofessional education for collaboration: Learning how to improve health from interprofessional models across the continuum of education to practice: Workshop summary. Washington, DC: The National Academies Press.

Janson, S. L., M. Cooke, K. W. McGrath, L. A. Kroon, S. Robinson, and R. B. Baron. 2009. Improving chronic care of type 2 diabetes using teams of interprofessional learners. Academic Medicine 84(11):1540-1548.

Knight, L. J., J. M. Gabhart, K. S. Earnest, K. M. Leong, A. Anglemyer, and D. Franzon. 2014. Improving code team performance and survival outcomes: Implementation of pediatric resuscitation team training. Critical Care Medicine 42(2):243-251.

Lang, E. V., C. Ward, and E. Laser. 2010. Effect of team training on patients’ ability to complete MRI examinations. Academic Radiology 17(1):18-23.

Marinopoulos, S. S., T. Dorman, N. Ratanawongsa, L. M. Wilson, B. H. Ashar, J. L. Magaziner, R. G. Miller, P. A. Thomas, G. P. Prokopowicz, R. Qayyum, and E. B. Bass. 2007. Effectiveness of continuing medical education. Evidence Report/Technology Assessment No. 149. Rockville, MD: Agency for Healthcare Research and Quality.

Mayer, C. M., L. Cluff, W. T. Lin, T. S. Willis, R. E. Stafford, C. Williams, R. Saunders, K. A. Short, N. Lenfestey, H. L. Kane, and J. B. Amoozegar. 2011. Evaluating efforts to optimize TeamSTEPPS implementation in surgical and pediatric intensive care units. Joint Commission Journal on Quality and Patient Safety 37(8):365-374.

Merriam, S. B., and B. Leahy. 2005. Learning transfer: A review of the research in adult education and training. PAACE Journal of Lifelong Learning 14:1-24.

Morey, J. C., and M. Salisbury. 2002. Introducing teamwork training into healthcare organizations: Implementation issues and solutions. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Baltimore, MD. Santa Monica, CA: Human Factors and Ergonomics Society. Pp. 2069-2073.

Morey, J. C., R. Simon, G. D. Jay, R. L. Wears, M. Salisbury, K. A. Dukes, and S. D. Berns. 2002. Error reduction and performance improvement in the emergency department through formal teamwork training: Evaluation results of the MedTeams project. Health Services Research Journal 37(6):1553-1581.

Neily, J., P. D. Mills, Y. Young-Xu, B. T. Carney, P. West, D. H. Berger, L. M. Mazzia, D. E. Paull, and J. P. Bagian. 2010. Association between implementation of a medical team training program and surgical mortality. Journal of the American Medical Association 304(15):1693-1700.

Nielsen, P. E., M. B. Goldman, S. Mann, D. E. Shapiro, R. G. Marcus, S. D. Pratt, P. Greenberg, P. McNamee, M. Salisbury, D. J. Birnbach, P. A. Gluck, M. D. Pearlman, H. King, D. N. Tornberg, and B. P. Sachs. 2007. Effects of teamwork training on adverse outcomes and process of care in labor and delivery: A randomized controlled trial. Obstetrics & Gynecology 109(1):48-55.

Nurok, M., L. A. Evans, S. Lipsitz, P. Satwicz, A. Kelly, and A. Frankel. 2011. The relationship of the emotional climate of work and threat to patient outcome in a high-volume thoracic surgery operating room team. BMJ Quality and Safety 20(3):237-242.

Owen, J. A., V. L. Brashers, C. Peterson, L. Blackhall, and J. Erickson. 2012. Collaborative care best practice models: A new educational paradigm for developing interprofessional educational (IPE) experiences. Journal of Interprofessional Care 26(2):153-155.

Owen, J. A., V. L. Brashers, K. E. Littlewood, E. Wright, R. M. Childress, and S. Thomas. 2014. Designing and evaluating an effective theory-based continuing interprofessional education program to improve sepsis care by enhancing healthcare team collaboration. Journal of Interprofessional Care 28(3):212-217.

Patterson, M. D., G. L. Geis, T. LeMaster, and R. L. Wears. 2013. Impact of multidisciplinary simulation-based training on patient safety in a paediatric emergency department. BMJ Quality and Safety 22(5):383-393.

Pettker, C. M., S. F. Thung, E. R. Norwitz, C. S. Buhimschi, C. A. Raab, J. A. Copel, E. Kuczynski, C. J. Lockwood, and E. F. Funai. 2009. Impact of a comprehensive patient safety strategy on obstetric adverse events. American Journal of Obstetrics and Gynecology 200(5):492.e1-492.e8.

Phipps, M. G., D. G. Lindquist, E. McConaughey, J. A. O’Brien, C. A. Raker, and M. J. Paglia. 2012. Outcomes from a labor and delivery team training program with simulation component. American Journal of Obstetrics and Gynecology 206(1):3-9.

Pingleton, S. K., E. Carlton, S. Wilkinson, J. Beasley, T. King, C. Wittkopp, M. Moncure, and T. Williamson. 2013. Reduction of venous thromboembolism (VTE) in hospitalized patients: Aligning continuing education with interprofessional team-based quality improvement in an academic medical center. Academic Medicine 88(10):1454-1459.

Rask, K., P. A. Parmelee, J. A. Taylor, D. Green, H. Brown, J. Hawley, L. Schild, H. S. Strothers III, and J. G. Ouslander. 2007. Implementation and evaluation of a nursing home fall management program. Journal of the American Geriatric Society 55(3):342-349.

Reeves, S., and S. Hean. 2013. Why we need theory to help us better understand the nature of interprofessional education, practice and care. Journal of Interprofessional Care 27(1):1-3.

Reeves, S., L. Perrier, J. Goldman, D. Freeth, and M. Zwarenstein. 2013. Interprofessional education: Effects on professional practice and healthcare outcomes (update). Cochrane Database of Systematic Reviews 3:CD002213.

Riley, W., S. Davis, K. Miller, H. Hansen, F. Sainfort, and R. Sweet. 2011. Didactic and simulation nontechnical skills team training to improve perinatal patient outcomes in a community hospital. Joint Commission Journal on Quality and Patient Safety 37(8):357-364.

Salas, E., D. DiazGranados, C. Klein, C. S. Burke, K. C. Stagl, G. F. Goodwin, and S. M. Halpin. 2008. Does team training improve team performance? A meta-analysis. Human Factors: The Journal of the Human Factors and Ergonomics Society 50(6):903-933.

Sax, H. C., P. Browne, R. J. Mayewski, R. J. Panzer, K. C. Hittner, R. L. Burke, and S. Coletta. 2009. Can aviation-based team training elicit sustainable behavioral change? Archives of Surgery 144(12):1133-1137.

Schnelle, J. F., J. G. Ouslander, and S. F. Simmons. 2006. Direct observations of nursing home care quality: Does care change when observed? Journal of the American Medical Directors Association 7(9):541-544.

Shaw, D. J., J. E. Davidson, R. I. Smilde, T. Sondoozi, and D. Agan. 2014. Multidisciplinary team training to enhance family communication in the ICU. Critical Care Medicine 42(2):265-271.

Steinemann, S., B. Berg, A. Skinner, A. DiTulio, K. Anzelon, K. Terada, C. Oliver, H. C. Ho, and C. Speck. 2011. In situ, multidisciplinary, simulation-based teamwork training improves early trauma care. Journal of Surgical Education 68(6):472-477.

Strasser, D. C., J. A. Falconer, A. B. Stevens, J. M. Uomoto, J. Herrin, S. E. Bowen, and A. B. Burridge. 2008. Team training and stroke rehabilitation outcomes: A cluster randomized trial. Archives of Physical Medicine & Rehabilitation 89(1):10-15.

Sullivan, G. M. 2011. Getting off the “gold standard”: Randomized controlled trials and education research. Journal of Graduate Medical Education 3(3):285-289.

Tapson, V. F., R. B. Karcher, and R. Weeks. 2011. Crew resource management and VTE prophylaxis in surgery: A quality improvement initiative. American Journal of Medical Quality 26(6):423-432.

Taylor, C. R., J. T. Hepworth, P. I. Buerhaus, R. Dittus, and T. Speroff. 2007. Effect of crew resource management on diabetes care and patient outcomes in an inner-city primary care clinic. Quality & Safety in Health Care 16(4):244-247.

Theilen, U., P. Leonard, P. Jones, R. Ardill, J. Weitz, D. Agrawal, and D. Simpson. 2013. Regular in situ simulation training of paediatric medical emergency team improves hospital response to deteriorating patients. Resuscitation 84(2):218-222.

Thompson, C., A. L. Kinmonth, L. Stevens, R. C. Peveler, A. Stevens, K. J. Ostler, R. M. Pickering, N. G. Baker, A. Henson, J. Preece, D. Cooper, and M. J. Campbell. 2000a. Effects of a clinical-practice guideline and practice-based education on detection and outcome of depression in primary care: Hampshire depression project randomised controlled trial. Lancet 355(9199):185-191.

Thompson, R. S., F. P. Rivara, D. C. Thompson, W. E. Barlow, N. K. Sugg, R. D. Maiuro, and D. M. Rubanowice. 2000b. Identification and management of domestic violence: A randomized trial. American Journal of Preventive Medicine 19(4):253-263.

Weaver, S. J., M. A. Rosen, D. DiazGranados, E. H. Lazzara, R. Lyons, E. Salas, S. A. Knych, M. McKeever, L. Adler, M. Barker, and H. B. King. 2010. Does teamwork improve performance in the operating room? A multilevel evaluation. Joint Commission Journal on Quality and Patient Safety 36(3):133-142.

Weaver, S. J., S. M. Dy, and M. A. Rosen. 2014. Team-training in healthcare: A narrative synthesis of the literature. BMJ Quality and Safety 23(5):359-372.

Wolf, F. A., L. W. Way, and L. Stewart. 2010. The efficacy of medical team training: Improved team performance and decreased operating room delays: A detailed analysis of 4863 cases. Annals of Surgery 252(3):477-483.

Young, A. S., M. Chinman, S. L. Forquer, E. L. Knight, H. Vogel, A. Miller, M. Rowe, and J. Mintz. 2005. Use of a consumer-led intervention to improve provider competencies. Psychiatric Services 56(8):967-975.

Young-Xu, Y., J. Neily, P. D. Mills, B. T. Carney, P. West, D. H. Berger, L. M. Mazzia, D. E. Paull, and J. P. Bagian. 2011. Association between implementation of a medical team training program and surgical morbidity. Archives of Surgery 146(12):1368-1373.

ANNEX A-1

TABLE A-2 Measuring the Impact of IPE on Collaborative Practice and Patient Outcomes: Detailed Data Table

This page intentionally left blank.