Future Directions of Credentialing Research in Nursing: Workshop Summary (2015)

3

Strengthening Data and Health Informatics for Credentialing Research

DATA HARMONIZATION FOR CREDENTIALING RESEARCH

Ronda Hughes, Marquette University, and Murielle Beene, U.S. Department of Veterans Affairs

Nursing credentialing research is limited by data sets that are “insufficiently accessible and inadequately standardized across credential types and credentialing organizations” (Hughes et al., 2014, p. 1). To better understand the impact of nurse certification or nursing credentialing on different outcomes, researchers must have access to current, standardized, and interoperable data sets (Hughes et al., 2014).¹

Harmonizing Meaningful Data

In health care, a major challenge as a researcher or as a clinician is to generate meaning from existing data, began Hughes. Research on the impact of nursing credentialing is more than noting the presence of a dichotomous variable—whether someone has a certification or not. It requires being able to determine whether there are intervening variables that mediate, moderate, or modify the effects of having a credential. Can existing variables be used to answer research questions in a manner that is actionable and generalizable? Ideally, researchers want to determine

__________________

¹The first two presentations drew largely from the IOM Perspective paper The Significance of Data Harmonization for Credentialing Research (Hughes et al., 2014).

whether meaningful differences exist between a practitioner who has a specific certification and one who is highly competent but lacks a certification, said Hughes.

Adequate data standardization is a persistent challenge. In general, data collection by numerous credentialing and certification organizations is idiosyncratic, which complicates attempts to link databases for comparison. Different organizations have their own data sets, and may select and collect variables for purposes unrelated to advancing knowledge about the value of credentialing. Even within robust databases, content and quality of content may vary significantly, with some data being more recent than other data.

Linking such varied data to outcomes is a further challenge. Observational data often lacks the granularity necessary to investigate credentialing impact, noted Hughes. For example, claims data are retrospective and cannot be used to identify which health care providers influence care to individual patients. Electronic health records (EHRs) may permit closer analysis of the care process, but Hughes again cautioned that the number of health care providers who interact with a patient during a care visit or inpatient episode make attribution of outcomes to a specific person, whether credentialed or not, next-to-impossible.

Data accessibility is another limitation, though databases may be more accessible in the future. For the most part, certification data are not easily accessed by researchers because the data are considered intellectual property of credentialing organizations and employers, explained Hughes. Even if organizations are willing to share data, financial and procedural requirements create additional barriers. In spite of these challenges, researchers are optimistic about increasing data availability.

Attaining Data System Interoperability

Beene began by stating that, as a science, informatics is at the intersection of information science, health science, and computer science and may present opportunities to develop a common data information model in nursing credentialing research. Eventually, U.S. health care data systems are meant to be interoperable, ensuring that data can be exchanged among them, but interoperability is in its early stages.

Various levels of system interoperability exist (see Table 3-1). Currently, some U.S. information systems represent Level-2 operability,

TABLE 3-1 Levels of Data Interoperability and Characteristics

SOURCE: Hughes et al., 2014.

with unstructured data that are electronically exchanged and viewable, said Beene. To answer questions about what credentials and characteristics make a difference in health care provision and health outcomes, systems need structured data that are coded using a standardized terminology and can be electronically exchanged across systems.

Many barriers exist to optimizing interoperability among data systems. First, cost is the barrier upgrading these data environments. Second, the field lacks a common data model that uses consistent definitions across organizations, systems, and databases.

However, “in a service-oriented data architecture, interoperability and the exchanging of data meaningfully in their proper context can be achieved,” said Beene. Cloud computing may prove helpful in data aggregation. To help increase interoperability and encourage more robust data systems, harmonization could start with a basic, minimum set of data elements, Beene suggested.

HARMONIZATION AND PERFORMANCE MEASURE DEVELOPMENT TO EVALUATE CREDENTIALING

Patricia Dykes, Brigham and Women’s Hospital

According to Dykes, analytic approaches are insufficiently data-driven due to many factors related to data harmonization and meaningful performance measures, including the number of groups involved in individual and organizational credentialing; the minimum and voluntary nature of many standards; the multiple streams of (imperfect) data from different sources; the inability of current data platforms to capture, store, and organize different types of data in ways that support manipulation and analysis; and the absence of a common data model. EHRs may only perpetuate, not solve these problems, if inconsistent across systems.

To advance nursing credentialing research, Dykes suggested two research priorities: (1) establish a common credentialing data model that defines required data, how they will be used, and relationships between individual data points; and (2) identify existing measures and develop new metrics to evaluate credentialing and establish relationships between credentialing and outcomes.

A Common Data Model

A common data model could facilitate discussion of important metrics (including definitions) in nursing credentialing and promote standardization of data and data collection procedures across organizations to improve interoperability, she continued. Table 3-1 could be used as a foundation for such a data model because it identifies different categories of data that could be incorporated. Box 3-1 includes examples of some key questions that could influence the content and structure of a common data model for nursing credentialing research.

Developing a common data model will require the input of large groups of stakeholders to determine relevant data elements. Adopting a big data approach may be useful to help focus research questions and identify relevant metrics and variables based on the Expanded Conceptual Model (see Figure 2-2). Table 3-2 provides an example of leveraging that model to build a common data model that could capture the relationships among data elements, research questions and measures, and data sources in the context of organizational Magnet certification and patient falls.

TABLE 3-2 Example of a Credentialing Research Data Model

NOTE: CMS = Centers for Medicare & Medicaid Services; ICD = International Classification of Diseases; LOINC = Logical Observation Identifiers Names and Codes; MU = Meaningful Use; SNOMED CT = Systematized Nomenclature of Medicine–Clinical Terms.
SOURCE: Dykes, 2014.

Identifying Measures and Metrics

Identifying what data are necessary to answer vital research questions is critical. Data related to credentialing are available, but distributed across multiple organizations and databases. Although there is agreement about some necessary data points, more information is needed. For example, at the individual level, it would be useful to collect data on the education level of nurses and the accreditation status of their nursing schools. At the organizational level, it would be useful to know which Meaningful Use Stage 1 and 2 objectives have been met and to what extent the organization has adopted electronic records.

Identifying process measures to evaluate nursing credentialing is more difficult. Nurses do not consistently document their interventions. This lack of adequate data, particularly nursing process and intervention data, in a structured coded format, hinders evaluation of the impact of nursing care on patient outcomes.

Outcomes data exist in numerous places, such as in electronic health records, administrative databases, and incident reporting systems. At present, these data often are not available electronically because they are not consistently in a structured, coded format, and priorities for developing more systematic outcome data have not been set.

Given the amount of available data, it becomes important to define a process for transforming and aggregating data from various databases, said Dykes. Who will contribute, validate, and manage these data is unclear. Establishing both human and technological information networks across data streams and multiple stakeholders will require incentives and funding. These networks assist efforts to fill several large-scale data gaps, including the lack of high-quality nursing process and outcome data.

Going forward, it might be useful to identify critical nursing-sensitive and credentialing-sensitive research questions and metrics (including independent and dependent variables), develop data sources, and collect data, such as those identified in Figure 3-1. Once these data are aggregated and if a common data model exists across organizations, big data analytics can be used to generate hypotheses, which can lead to multi-site research studies.

DEVELOPING, TESTING, AND REFINING MEASURES OF NURSE-SENSITIVE QUALITY OF CARE

Nancy Dunton, Kansas University School of Nursing

Dunton focused her presentation on describing data available in the National Database of Nursing Quality Measures^® (NDNQI) (which is a proprietary database that includes nurse-sensitive data on the structure, process and outcomes of care) and on how to develop valid and reliable measures.

The NDNQI Database

Dunton explained that the NDNQI includes two data streams:

1. Quarterly clinical and staffing data that include information from chart review, prevalence surveys, incident reports, patient census, payroll, and from human resources data on the education and certification of nurses.
2. A survey of nurses who spend more than 50 percent of their time in direct patient care and who have been in their current work group for at least 3 months (Dunton, 2014). The questionnaire includes a variety of questions related to education, nursing specialty certification, and credentialing.

NDNQI data uses the nursing care unit as the unit of analysis, rather than individual nurses or patients. For example, NDNQI includes measures on the prevalence of certified nurses on the unit or in a work group. NDNQI also collects data on registered nurse (RN) specialty certification through the clinical and staffing sources and the nurse survey. The NDNQI RN Survey questionnaire collects data from RNs and advanced practice registered nurses (APRNs) with current certifications in a nursing specialty that is granted by a national nursing organization.

Both NDNQI data sets can be used to identify whether a specific factor, such as certification, is associated with a nursing outcome or process about which it also collects data, although few associations have been discovered so far. The strongest association observed has been between critical care/cardiac care certifications and blood stream infections. In some instances, when certification appeared to have a significant association with a care improvement (e.g., reduced pressure ulcer rates), the cor-

relation became insignificant after controlling for education and years of practice.

There are some obvious limitations in collecting data about nursing credentialing. For example, data reporting is voluntary and many hospitals do not report all measures. Some hospitals report particular measures more frequently than do others. Dunton also explained that nurses do not always provide accurate information about certifications. NDNQI has learned to ask questions about education, credentialing, and hospital-issued certificates before asking questions about national nursing specialty certifications.

Measure Development and Evaluation

The NDNQI measure development process is designed so that evidence collected supports a submission to the National Quality Forum (NQF) for potential consensus adoption. Database developers assess the importance of the prospective measure—whether a performance gap across hospitals exists, whether the measure relates to a high volume or high cost service, and whether the measure aligns with national health care priorities. In addition, evidence is collected on the reliability and validity of measures.

Measure development is triggered by a variety of factors, including national policy issues, hospital requests, or from peer-reviewed publications on nursing processes or nurse-sensitive patient outcomes. Once a topic area is identified, the next step is to conduct literature reviews, looking for existing measures and guidelines (such as those endorsed by the NQF and the Agency for Healthcare Research and Quality [AHRQ]). If a new measure is needed, the NDNQI measure development process includes consulting with topical experts about proposed measures, developing draft guidelines and data collection forms (which include the proposed numerator and denominator, inclusion and exclusion criteria, and potential collateral data). These documents are reviewed by the experts. The next step in measure development is to conduct pilot testing in volunteer hospitals. Results from pilot testing are used to confirm data availability and data collection feasibility. The results of the pilot study can be used to refine and revise the guidelines and forms. After development, the new measure is built into NDNQI’s data capture and reporting systems. Post-implementation activities involve quality assurance

checks and ongoing monitoring. New measures, in particular, are tracked for large variations over time and for problems in accessing needed data.

As an example, the process for developing NDNQI’s recently released e-measure for pressure ulcer incidence and prevention took 2 years and resulted in a 60-page guidance document. Initially, nine hospitals have successfully implemented the measures, which is promising, Dunton remarked.

The final step is to conduct reliability and validity studies after a measure has been in place for at least 1 year and data collection has stabilized. For nursing-sensitive process and outcome measures, NDNQI develops multi-level models and tests to see whether nursing workforce characteristics are significantly associated with the new process or outcome measure.

Additionally, maintaining valid and reliable measures requires continual quality assurance and measure evaluation. Over time, measures are further refined or enhanced to maintain alignment with the state of the science.

INFORMATICS FOR DECISION MAKING

Patricia Flatley Brennan, University of Wisconsin–Madison

Emerging technologies are changing the rules in which health care operates, Brennan stated. Many health care quality initiatives naturally target episodes of care, driven by compensation models, in part. But achieving true patient-centered care requires expanding the focus of performance measurement or credentialing beyond the walls of a health care institution—on the episode of health rather than the episode of health care provision. Brennan then urged the nursing credentialing community to weigh patient life experiences when considering which responsibilities that individual and organizational credentialing capture. This broad perspective challenges researchers to think about data and practice differently and to consider different reference points for credentialing. “Are we credentialing an individual for life or credentialing a team for the moment?” Brennan asked.

Brennan emphasized the importance of the interoperability of information systems, if patient-centered data collection is to be achieved. The field has strategies to formalize, aggregate, and interpret data in order to assess the characteristics of an individual or organization, as well as impact of the certification. Interoperability will be essential for continuing

to assess the process of care across a number of different points, provided by practitioners whose accountabilities extend beyond a brief patient encounter. Medical informatics has taken some fledgling steps toward this, although the priority put on meeting Meaningful Use requirements has slowed the process.

Improved health information technologies may enable a more nimble certification cycle. The provision of health care services is constantly evolving, and certification programs need to account for these changes in their examinations. Brennan suggested that some certification requirements may not adequately reflect the changing knowledge and skillsets required to practice in today’s health care environment. For example, telemedicine encounters or remote nurse call centers may require different knowledge management experience, critical judgment, and interpersonal skills than acute care hospital settings. Certification programs need to reflect these new realities.

Health care provision is also increasingly team oriented, and there may be opportunities for certification to measure team performance. Credentialing of teams is different from having a team whose individual members have various certifications. Team credentialing would signal whether a particular team is effective in certain care domains—care of people with chronic illnesses or of families with a seriously ill child, for example. Challenges associated with dynamic team compositions within a health care setting (e.g., changing expertise or skills required to care for patients, shift changes, and normal day-to-day exigencies) may be alleviated through different kinds of personal tracking technologies, such as radio-frequency identification sensors or programmable or wearable devices. However, adoption of technological innovation will also raise new issues.

Brennan urged the audience to consider how credentialing can reflect efforts to improve patient-centered care. Should patients have some kind of say in the credentialing process, and, if so, how should they be compensated for their participation? Should patients have input into the measurement of “outcomes,” including whether they like the outcome, and is it the one they wanted? Should patient-reported outcomes be included in EHRs?

As tools for knowledge management and information access evolve, certification examinations should also evolve to reflect these changes, said Brennan. Certification programs need to consider not only how to make the best use of current technologies, but also how to incorporate technologies in the future.

QUESTIONS AND COMMENTS

As the country moves toward a health versus an illness approach and an increasing amount of care is delivered outside hospitals, how do we measure outcomes in the community?

Hughes reiterated that a formidable challenge to measuring outcomes in the community is the lack of interoperability of information systems across care settings. Even insurance data lacks information about what transpires during face-to-face encounters with clinicians. Measures need to be developed for outpatient settings that complement inpatient measures.

Brennan said attempts are under way to develop a multi-level, systems framework for health care–related data. A wider focus for collecting data on the patient experience may be needed, taking into account more of the patient’s life and health and linking that information to population health. On the other hand, a narrower focus that views certification as an episodic rather than a continuous monitoring activity may be desirable for learning about professional certification. In the future, certification may be viewed not as a persistent attribute, but something verified through sampling and spot checks—an approach used in quality engineering.

Will it be necessary for human resources data systems to become more structured, or will data be captured from different streams (including from free-text portions of EHRs) and structured after the fact, in some reliable, valid way?

Dykes responded that “big data” offer the possibility of using multiple, new methods of aggregating and mining data, and of recognizing patterns, so that always having structured, coded data becomes nonessential. A good place to start would be to identify a core data set related to some of the most important research questions. The entire process is likely to be incremental. Developing partnerships with clinicians, EHR vendors, and other stakeholders will be important in designing a way forward.

At present, researchers first specify the data they want and then try to find it in their health records and data systems, Dunton added, which requires a substantial amount of judgment and testing, and can thus affect reliability. Brennan suggested there was a need to think more broadly about necessary data elements and systematic sampling strategies to be more feasible. Brennan continued, existing text-based data systems will be extremely difficult to convert to standardized terms, although that

process would be aided by development of some good natural language processing software.

How can data systems identify credentials and their value in community-based settings?

Dykes said at the organizational level, data can be collected from different sources related to individual clinicians and their credentials. However, decisions are needed to identify the data steward: the certifying organization (from which employers can pull the data) or the employer, asked Dykes. Brennan said there is not always an employer relationship at the site where a clinician is working. For example, public health nurses may work in a particular clinic, but be employed by a larger entity, such as county government. If there is an umbrella organization or a head nurse to pull information together, that may be a channel for obtaining certification information, Brennan concluded.

As more health care providers become accountable care organizations, which are required to use EHRs, will patient-level data become increasingly available?

A useful step would be to try to increase the amount of information that patients are willing to share, Hughes said, by providing incentives to report data and by minimizing concerns about privacy, confidentiality, or potential misuse of data. Brennan said many health systems offer patients Web-based portals in which they can enter personal and even clinical data (from out-of-system providers) that are not standardized, do not become part of their clinical record, and are not accessible to clinicians. This data wall needs to be broken down, she said.

Dykes reported that a current project at Brigham and Women’s Hospital allows patients and families to provide feedback by entering information on their goals of care, potential concerns, and ratings of the provider team. This becomes part of their interdisciplinary plan of care. However, not every patient and family can or wants to do this.

Do certification examinations and their preparatory materials need some mechanism for continual updating, in order to improve predictive validity and achieve better alignment among health system needs, the education system, and the credentialing process?

Based on feedback received by his organization, WorkCred (an affiliate of the American National Standards Institute), participant Roy Swift stated that many health systems share the common belief that recent

graduates and credentialed professionals have a skills gap. The Quality and Safety Education for Nurses project² includes a range of competencies needed by future nurses that should be incorporated into education in both undergraduate and graduate training, he said.

__________________

²This project has defined six competencies needed by nurses, so that they have the knowledge, skills, and attitudes required for improving the quality and safety of health care systems: patient-centered care, teamwork and collaboration, evidence-based practice, quality improvement, safety, and informatics (see Case Western Reserve University, 2014).

This page intentionally left blank.