Future Directions of Credentialing Research in Nursing: Workshop Summary (2015)

4

Challenges and Opportunities in Credentialing Research Methodologies

CHALLENGES AND OPPORTUNITIES IN NURSING CREDENTIALING RESEARCH DESIGN

Matthew McHugh, University of Pennsylvania School of Nursing¹

McHugh presented some important methodological challenges in nursing credentialing and suggested possible research questions to focus discussion on how to develop solid research strategies to answer those questions. What does a credential represent and how does that, in turn, shape research design selection, asked McHugh.

In the theoretical domain, McHugh and colleagues (2014) examined how opinions about what voluntary certification actually represent might affect research design. The human capital theory suggests that, during the credentialing journey, a nurse is fundamentally changed in terms of knowledge or skills—and, by extension, competence (McHugh et al., 2014). Conversely, the signaling theory suggests that a voluntary certification is a marker for underlying traits of the person seeking the credential. Such traits may be additional years of education or experience or other characteristics when designing research. The theory selected is important because it assumes different facts, which affects potential research questions. For example, if the goal is to increase the number of credentialed individuals, the signaling theory implies that only certain classes of employee (i.e., those with the necessary underlying traits) will benefit from credentialing programs. Box 4-1 includes examples of

__________________

¹This presentation drew largely from the IOM Perspective paper Challenges and Opportunities in Nursing Credentialing Research Design (McHugh et al., 2014).

research questions stemming from each theory. Moreover, the research question affects the relevant stakeholders, which has implications for research dissemination and funding. The natural tension between those who subscribe to the signaling theory versus the human capital theory “is at the forefront of the credentialing research challenges,” McHugh said.

As Needleman suggested earlier, McHugh also believes that voluntary certification reflects assumptions of both signaling and human capital theories, and the balance varies across individuals, institutions, and certification programs. Some baseline characteristics may be present, but the certification process (as either an individual or an organization) fundamentally alters an individual’s knowledge and skills or organizational structure, practices, and management styles, which can lead to observed quality differences.

Research needs to focus on the intermediate pathways between the decision to credential and outcomes to determine whether an observed effect can be attributed to the credential itself or other factors within the health care context. Additional work is needed to identify intervening variables and how those intervening variables interact and for what units

of analysis (e.g., institutions, teams, or individuals) and across what settings. To date, the dominant unit of analysis in nursing credentialing research is the institution, rather than the individual nurse. But as efforts to link individual health outcomes to an individual nurse increase, more evidence at this level of analysis will be available. The expansion of the Magnet recognition program internationally will increases opportunities for comparison studies, as well as alternative care settings (e.g., hospitals, nursing homes, ambulatory care). For example, McHugh cited two recent papers which found that the nurse work environment could account for observed differences between Magnet Recognition Hospitals and non-Magnet Hospitals (McHugh et al., 2013; Stimpfel et al., 2014).

Successful research designs in nursing credentialing will not necessarily require novel research designs and methods; rather, we need to rethink how to apply existing research tools to establish causal links, said McHugh. He suggested regression analyses, propensity matching techniques, use of instrumental variables, and longitudinal research that includes counterfactual testing. As noted by other speakers, research designs will also have to account for additional challenges, such as selection bias and a lack of data and standardized data across multiple certification and credentialing programs to improve interoperability.

Finally, good causal research, including longitudinal studies, will require additional funding. In turn, this funding will generate more evidence to attract more diverse funders. Alternative funding strategies are needed that can support a program of research and ongoing studies that build on one another. For example, credentialing organizations and other stakeholders allocate a proportion of their revenues to research by independent investigators.

INVESTIGATING CAUSAL PATHWAYS AND LINKAGES

Patrick S. Romano, University of California, Davis

Romano began by exploring the question of whether causality matters in nursing credentialing research. Researchers in this field are seeking answers to a “counterfactual” question: What would the current outcomes of this nurse or nursing organization be if it were not credentialed (when, in fact, it is)? Given the difficulty of answering this counterfactual question, Romano asserted that establishing causality may not be the critical question if signaling theory can be applied. Good signals may increase the efficiency of health care markets and nursing labor

markets. In the context of organizational credentialing, Magnet status may indicate better work environments to prospective nurse employees and better health outcomes, deserving higher payments from insurers. Similarly, employers may view nurses with voluntary credentials as employees who are more motivated and committed than their counterparts, leading to better hiring opportunities and compensation.

However, there are problems with ignoring causality and relying on signaling theory to support credentialing. First, more must be known about the signal itself and what it represents. To establish whether certification and credentialing programs are valid signals of improved outcomes, each program must be assessed independently. Researchers do not currently know which characteristics of certification programs—breadth, depth, requirements, difficulty—are associated with valid signals. Second, certification programs may not be feasible in all settings or specialties for various reasons. Third, if nursing credentialing is assumed to be associated with better health outcomes via signaling, then credentialed nurses and organizations are rewarded by the market through higher income or greater market share. Credentialing then becomes a “private good” unworthy of public investment. In this situation, credentialing may be “captured” by one stakeholder group (e.g., unions) for its own benefit, and public policies to encourage credentialing may be misguided.

If researchers could identify the pathways or specific nursing practices by which credentialing leads to better outcomes, then credentialing organizations would know how to design their programs to optimize outcomes through these evidence-based pathways. It may no longer be necessary to establish that a credentialing program itself causes better outcomes. Instead, certifying bodies could design their programs to promote adherence to evidence-based processes, leading to better outcomes. Markets would then reward only the credentialing programs that are able to demonstrate these effects. Moreover, in areas where credentialing is not possible, employers could achieve (and patients could receive) better outcomes by adhering to evidence-based guidelines, regardless of the credentialing status of a health care facility or an individual practitioner. This, in turn, would “weaken the power of credentialing monopolies and reduce the risk of [skewing market signals],” said Romano. It would also clarify whether public policy should encourage individuals and organizations to pursue credentialing.

Referencing the American Nurses Credentialing Center (ANCC) Model (see Figure 2-1), Romano noted that the nonrecursive relationships imply that some “Intervening Variables” may influence credential-

seeking behavior. Similarly, improved patient outcomes may lead to changes in pathways (e.g., processes of nursing care). The Expanded Conceptual Model (see Figure 2-2) identified three categories of intervening variables (i.e., individual nurse performance, organization of nursing work tasks, and the “invisible architecture” of the nursing work environment, including organizational climate, leadership, and culture). Combinations across these categories may lead to earlier recognition of problems and appropriate interventions that produce better patient outcomes, Romano proposed. For example, failure-to-rescue research suggests that high-quality organizations and high-quality nursing care improve patient outcomes by quickly identifying patients with complications rather than by preventing complications. Nurses observe and recognize warning signs and symptoms, interpret these signs and symptoms as problems that threaten a patient’s recovery, collect the additional information necessary to engage team members and facilitate decision making, quickly communicate this information to other team members, suggest specific treatments or additional tests, implement these interventions quickly and effectively, evaluate their impact in a timely and accurate manner, and finally communicate these findings back to the team. High-quality care “is very much a complex process of coordinating services, integrating care, and communicating effectively,” he said.

Figure 4-1 provides a simplified conceptual model of the causal pathway leading from individual nurse credentialing to better patient outcomes. This conceptual model can be applied to any process—education, experience, credentialing—that leads to enhanced and testable knowledge and skills, leading to improvements within the three categories of intervening variables, leading in turn to earlier problem recognition, intervention, and improved patient outcomes.

A fundamental problem in analyzing causal pathways is that they are so complex, in that different knowledge and skills contribute to different behaviors, which in turn contribute to different outcomes or to several outcomes. In addition, several types of behavior can lead to a single outcome. In some cases, feedback and balancing loops come into play—for example, in the adoption of a new practice. Romano used diagrams to illustrate these causal pathways and loops, which are complex and easily oversimplified. Other methodological challenges include the fact that knowledge and attitudes may translate poorly to skills and abilities “at the front line” of care; poor documentation of most nursing activities in medical records and coded data; difficulty linking patients to individual

FIGURE 4-1 Simplified conceptual model of credentialing pathway.
SOURCE: Romano, 2014.

nurses for attribution of outcomes; and partitioning variance in outcomes across unit-level, nurse-level, and patient-level factors, which requires large data sets.

A combination of enhanced data collection and better analytic methods will enable more rigorous exploration of causal pathways in future research. Some of the opportunities for new or improved research methods will arise from the availability of electronic health records (EHRs), which can link patient care to individual clinicians. Audio, video, and direct observation methods are now available to permit assessment of nursing processes in real-world situations. In addition, clearer definitions of nurse-specific work processes in the International Classification of Nursing Practice now allow for more reliable coding and, therefore, analysis.²

Sophisticated analytic methods also are increasingly available to analyze new forms of data. Hierarchical models may allow researchers to adjust for organization, unit, nurse, and patient-level factors using data

__________________

²The International Classification of Nursing Practice has been adopted by the World Health Organization’s Family of International Classifications and harmonized with the Systematized Nomenclature of Medicine–Clinical Terms (SNOMED CT).

sets in which all of the nurses who cared for an individual patient can be identified. Latent trait and class analysis techniques relate observed variables (e.g., credentials, experience) to underlying latent variables (e.g., quality of nursing care) without implying causality, which can be used to validate patient outcomes. Structural equation modeling incorporates latent variables and causal dependencies between exogenous and endogenous variables (e.g., path analysis), which supports estimation of both direct and indirect effects.

KNOWLEDGE DISCOVERY DATA ANALYTICS METHODS

Karen Monsen, University of Minnesota

Knowledge discovery or “Big Data” analytics is a new research paradigm that is emerging as massive data sets become available. Big Data shifts the focus of a research paradigm from scarcity, exactness, and imposition of controls to a “more, messy, good enough” approach. In this context, “more” data means less sampling error, “messy” means trying to account for inherent biases within observational data, and “good enough” means researchers no longer focus on causation but rather on learning what happens in an entire population—describing it, analyzing patterns, and generating new hypotheses.

Data infrastructures required for large data set analysis include the Nursing Minimum Data Set (NMDS) (Werley, 1991), which includes “a minimum set of elements of information with uniform definitions and categories concerning the specific dimensions of nursing, which meets the information needs of multiple data users in the health care system.” The Nursing Management Minimum Data Set (NMMDS) (Huber et al., 1997) includes “core essential data needed to support the administrative and management information needs for the provision of nursing care” (Monsen, 2014). The NMMDS can look at contextual factors and the cost of providing a credentialed nurse. The American Nurses Association has recognized several nursing terminologies used in EHRs that generate NMDS variables (Sewell and Thede, 2012).

Describing some of her work, Monsen demonstrated how knowledge discovery data analytics methods can be used to identify hidden patterns in nursing data (Monsen et al., 2010). Monsen and colleagues have used inductive, deductive, visualization, and mapping approaches to create intervention groups that reveal the complexity of nursing’s work and have shown how interventions relate to variability in outcomes.

She suggested that data visualization can be used to demonstrate a comprehensive, holistic nursing assessment, which depicts patient risk patterns for different problems related to knowledge, behavior, and status (Kim et al., 2013). Figure 4-2 provides examples of the types of “sunbursts,” which illustrates four unique patients’ risk patterns, created using public health nursing assessment data. Different colors correspond to different problems, with shading indicating degree of risk. Each problem includes three dimensions (knowledge, behavior, and status), which are represented by the three inner rings. The tabs around the outside edge mark related signs/symptoms.

Researchers have analyzed these types of clusters to identify co-occurring interventions associated with diverse problems. For example, researchers have used these types of sunbursts to explore whether public health nurses might change intervention approaches as patients’ conditions improve. Similarly, data-driven clusters were useful in predicting the likelihood that frail and nonfrail elders receiving home care services would be hospitalized (Monsen et al., 2011).

In the future, inclusion of additional variables in existing data sets may allow researchers to answers questions about nursing credentialing, to explore the association between intervention patterns and patient outcomes, and to evaluate patterns across agencies and programs. Monsen suggested that inclusion of a nurse-identifier and nursing credentials in nurse-generated data sets would enable a Big Data approach to answering questions about credentialing, exploring patterns in credentialed versus noncredentialed nurse practice, and evaluating associations between credentialing and patient outcomes.

FIGURE 4-2 Sunburst images used to depict patient risk patterns.
SOURCE: Kim et al., 2013.

NEW RESEARCH OPPORTUNITIES: BUILDING ON THE MOMENTUM OF THE AFFORDABLE CARE ACT

Ellen-Marie Whelan, Centers for Medicare & Medicaid Services

The Patient Protection and Affordable Care Act of 2010 (ACA) has created new opportunities for nurses and “expanded the role that research is playing in helping us better understand the role that nurses are playing,” as well as the roles nurses might play in the future, said Whelan. One goal of the ACA is to move from a producer-centered, volume-driven health care delivery system to a people-centered, outcomes-driven system. Fragmented systems of care are morphing into more coordinated systems, using a variety of new payment models in place of fee-for-service payments.

Transformation of the U.S. health care system requires three types of research:

1. Basic research using clinical efficacy studies to test what care works;
2. Outcomes, comparative effectiveness, and health services research to test who benefits from new care models; and
3. Quality measurement and improvement research to test how best to deliver care in different settings (Dougherty and Conway, 2008; Whelan, 2014).

Research about credentialing fits into the third category, as does the work of part of the Center for Medicare & Medicaid Innovation (Innovation Center), said Whelan. The Innovation Center tests new payment and service delivery and payment models to improve quality and control health care expenditures for a wide variety of patient and provider populations. Relevant research involves aligning payment incentives, health information technology, quality improvement collaboratives and learning networks, and efforts to improve training of clinicians and multidisciplinary teams.

Existing activities within the Innovation Center are generating new data sources that could be used for credentialing research, said Whelan. Two of the Centers for Medicare & Medicaid Services’ (CMS’s) bundled payment models, which provide a single payment for all services a patient receives during an episodes of care, involve post-acute care, in which nurses play a critical role. Nurses providing post-acute care hold

many different certifications, especially in gerontological specialties, creating an opportunity to investigate the impact of varying credentials. Similarly, CMS’s Strong Start for Mothers and Newborns initiative tests the effectiveness of different prenatal care approaches in reducing preterm births for at-risk women covered by Medicaid or the Children’s Health Insurance Program. All three models include variations in nurse staffing, which again may allow for comparison studies.

Newly tested models within the Innovation Center may also help target appropriate nursing services, which may lead to new credentials or modified requirements for existing credentials. In addition to promoting value-based purchasing and other programs that include incentives for adoption of EHRs and quality reporting requirements, CMS’s innovation portfolio also includes programs that provide research opportunities related to nursing and nurse credentialing. Some of the Innovation Center’s initial work involved accountable care organizations, which are designed to reduce costs and provide better quality care through shared savings and improved care coordination. Research on the nurse’s role within a care coordination team could better clarify how savings should be distributed through the system.

Initiatives on primary care delivery are examining how to define the role of different health care providers within cost-effective, team-based care. New CMS data about the care management component of comprehensive primary care have suggested that registered nurses (RNs) and nurse practitioners provide one-third of care management services, with physicians and medical assistants providing the remaining two-thirds within participating primary care practices. If lower-cost employees can perform a particular service with equal quality outcomes, the role of those employees may change accordingly.³ Other examples of new staff roles within the shifting health care landscape include “care coordinator,” “health systems engineer,” and “clinical nurse leader,” the latter of which the American Association of Colleges of Nursing (AACN) is certifying.

In the future, Whelan believes quality measures will move away from narrow, setting-specific snapshots of care delivery and toward a reorientation and alignment of measures around patient-centered

__________________

³Whelan also briefly mentioned providing money directly to governors to encourage blended funding streams, with a specific emphasis on reducing barriers to full practice; leveraging state licensure renewal processes to identify workforce shortages and better understand the geographic distribution of providers; and asking Health Care Innovation program awardees to consider how workforce roles will change as the delivery system changes.

outcomes across settings and episodes of care. Measurement will capture information at the levels of the individual clinician, group/facility, and population/community.

QUESTIONS AND COMMENTS

If researchers could determine the link between a credential and an important outcome, would CMS scale and spread it?

Whelan replied that, from the CMS actuaries’ point of view, before the “scale and spread” and implementation of regulations can occur, evidence of “success” (measured as cost savings or quality improvements) is also sought. For example, nurse practitioners and certified nurse midwives receive reimbursement for their services in some programs, and new Medicaid regulations permit states to reimburse non-licensed clinicians for some community health work.

Jack Needleman commented that billing data are used to determine Medicare and Medicaid costs, but to assess changes in care delivery, especially when it is necessary to determine who is actually delivering the care, better data systems are needed.

Earlier presenters discussed the difference between signaling measures and human capital measures; they concluded that one of the key problems with the former measure is that signals can be weak.

Needleman stated that sometimes a weak signal exists because the differences among individuals (or organizations) who choose to obtain credentials and those who do not are not significant. Moreover, there may not be large performance differentials between credentialed and noncredentialed groups. Similarly, many noncertified individuals may have the same knowledge and skills as certified individuals, but simply choose not to pursue certification.

McHugh emphasized that it may be important to understand the differences that do exist between the two groups. For example, why do some people choose to obtain a certification and others do not? Often a population of employees is simply divided into “credentialed” and “not credentialed,” and analysts make a lot of assumptions about what that means. Needleman added that credentialing data on physicians include “board eligible,” as well as “board certified” categories, both of which differ from the category of physicians who did not engage in either process. He suggested an analogous approach might be useful in nursing.

Weak signals are better than no signals, and with a large enough data set, signals can often be identified, Romano added. To some extent, there may be a feedback loop, in that when the importance of a credential-related signal becomes recognized as useful or valuable, employers increase their incentives and rewards for pursuing that credential.

Could researchers use programs like NICHE (Nurses Improving Care for Healthsystem Elders)⁴as test-beds for examining credentialing effectiveness, or is it necessary to look at larger, more comprehensive service systems?

Monsen responded that, ideally, research would take both approaches, but acknowledged that funding may not be available for both. Romano said that identifying the key processes that lead to desired outcomes enables identification of the characteristics of certification programs that are most likely to affect those pathways. If research indicated that identifying high-risk patients and modifying factors to reduce this risk improves outcomes, then certification programs can target development and maintenance of the most relevant skills to reduce risk.

__________________

⁴The NICHE program, Bolton said, has a specific patient population, has data about advanced-practice nurses certified in gerontology and about staff nurses, and shows early evidence of the nurses’ ability to identify patients at risk and intervene early (see NICHE, 2014).