One session of the workshop was devoted to a set of three breakout groups that examined strategies to improve measurement and the integration of data at the (1) individual (2), community and (3) state and system levels. This final chapter of the workshop summary describes the major points discussed by those three groups. It also includes observations made by workshop participants in general discussions of the major issues raised at the workshop.
MEASUREMENT AT THE INDIVIDUAL LEVEL
As children age and develop, measurement issues at the individual level change. In addition, children are situated within families and broader social systems that both influence and respond to their development. These complexities create challenges in measuring the strengths and the needs of children as well as services, processes, and resulting outcomes.
The breakout group began by discussing several gaps in measurement. Several important features of families are not currently well measured, such as family structure, family functioning, parental mental health, and child and family strengths. Some children may be more biologically susceptible to adversity, but individual-level markers are not currently measured or incorporated into decision making or datasets. Better integration of preschool and school data into measurements from other sectors could yield indicators of how a child is doing socially, academically, cognitively, and emotionally.
In the measurement of services and processes, a team approach could fill existing gaps, especially with colocation and coordination of care. With
regard to outcomes, brief and straightforward outcome measures could reflect key principles. For example, is a child on a trajectory to graduate from high school? Other issues involving measurement gaps include the information that can feasibly be derived from primary care and other service systems, the ability of data and measures to cross-talk among systems, the gradual transition from the parent or caregiver to the child or adolescent as the source of information, and whether data are readily available or are difficult to access.
The breakout group also discussed obstacles to data integration. Regulations deriving from the Health Insurance Portability and Accountability Act (HIPAA) and the Family Educational Rights and Privacy Act (FERPA) can reduce integration and cross-talk, despite opportunities to use data covered by these regulations productively. Integration is also an issue between research studies and the implementation of interventions in practice. Differing languages and measures point to a need for more systematic and standardized measures in research, policy, and practice, including measures of implementation fidelity.
Developmental changes also can be a factor in data integration. For example, young people who are aging out of care systems can lose services; they also can gain legal rights and greater control over data. Such transitions complicate the challenge of using data productively. Similarly, differing cultures and languages across professions can have a strong influence on the ability to share data and set measurement priorities. For instance, in the context of adult health care, behavioral health may refer to chronic disease self-management, self-efficacy, management of substance abuse, or other personal attributes.
Recent data breaches have increased concerns about the loss of control over data and how data might be used. For example, if a parent mentions in one context that he or she has mental health issues, will that information be disseminated in ways that cannot be controlled and could have harmful consequences? In general, ethical issues in measurement and data use require greater emphasis and study, members of the breakout group observed.
Data sharing requires infrastructure, time, energy, and a willingness to overcome a natural reluctance to share potentially sensitive information, but no system now exists to finance such sharing. Basic questions such as who will be entering data into a system, who cleans up the data, and who makes the data accessible remain unanswered. Families may be able to enter some data into a system, but who will work with families to determine what they want, need, and trust?
Determining data collection approaches, terms, and measures will require having different systems talking with each other to develop shared understandings. Workforce development and training also will be required to understand cross-sector approaches and overcome cultural differences,
breakout some group participants observed. The U.S. Department of Health and Human Services could advance this agenda by prioritizing measurement and assessment, by creating an office focused on child health, and by organizing a White House conference on children and youth to reinvigorate public support for improving youth services.
Measures could be bundled to provide greater insights into system issues. For example, the Affordable Care Act (ACA) includes provisions that could advance needed steps, though the ACA has few provisions directed specifically toward children in this area.
Families need to feel that they own and trust measurement processes if they are to participate fully. Could service providers work with families so they are involved in entering and using data? This would allow the building of a culture in which measurement tools contribute to learning and to the building of relationships between service providers and family members. Data collection could also be structured in such a way as to counter the fragmentation of health care if data were made available not just to single providers but to teams of service providers in ways that maintain the trust of individuals.
Individuals and families often do not have ready access to the services they need. For example, would it be possible for them to get better access to providers, and especially specialists, during the hours and days of the week that are convenient to working families? Measures could be developed to study and improve this issue.
Finally, the breakout group on measurement issues at the individual level discussed the need for public use measures. Assessment tools that are copyrighted or otherwise restricted can limit measurement capabilities. With greater availability and numbers of tools, the most appropriate assessment for a particular need could be identified and used. Similarly, common measures that are usable across research, policy, and practice could increase the scope, influence, and power of those measures. One approach, for example, could be to extend the lexicon developed by the Agency for Healthcare Research and Quality (AHRQ) around behavioral and mental health and use it across domains and sectors.
MEASUREMENT AT THE COMMUNITY LEVEL
The breakout group on issues at the community level talked about the definition of the term community. Most often, communities are defined geographically, but they could also be defined socially, economically, culturally, technologically, or along many other dimensions, and different forms of community call for different measurement tools.
A major issue discussed by the breakout group was that a great deal of data are already being collected. These data originate in a large number
of programs, jurisdictions, systems, and levels of government. They include economic data, public health data, education data, health data, census data, and data from the foster care, juvenile justice, and other social service systems. Data also can come from unconventional sources; for example, the website Zillow provides a wealth of community-level data.
However, this great wealth of data is highly fragmented, which results in an inability to form a holistic view of children’s health and well-being. Data collected at the federal, state, and local levels are not well connected, and public- and private-sector data are isolated from each other. Greater integration, dissemination, and use of these disparate data sources could help achieve many of the objectives sought by workshop participants while reducing duplicated effort.
Some members of the breakout group noted that funding could be directed explicitly toward the analysis and use of data and not just the collection of data could capitalize on the data that exist. Greater emphasis on the use of data in funding streams could incentivize such a change; another possibility would be to make future funding dependent at least in part on the use of the data generated by current funding. One interesting proposal discussed by the breakout group was to integrate all of the existing forms of data for a single community and determine both the potential and remaining gaps in such a dataset.
Greater integration of data will require attention to privacy issues, some members of the breakout group noted. Privacy issues may evolve as people begin to create their own personal data vaults and as a result of new policies. Greater public understanding of data issues also could change public attitudes toward privacy.
Considerable work is being done on the development of new and better assessment tools at the community level. However, some forms of data, such as positive indicators of children’s health and well-being or various forms of contextual and environmental data, are still notably lacking. As with existing measurement tools, the data from new measures could be used more efficiently and effectively to solve existing problems and identify strengths.
The community has a vested interest in the health and well-being of its children, even more so than do state or federal governments. This interest in children could be the foundation for greater trust and collaboration in building community-oriented data systems.
MEASUREMENT AT THE STATE AND SYSTEMS LEVEL
Integration of data is a particularly pressing concern at the state level, noted many participants in the third breakout group. State-level departments that deal with issues affecting children’s health and well-being
may not want to share information with each other, recognizing only the potential downsides rather than the benefits of such sharing. However, some states have forged quite successful data-sharing systems.
An interesting project would be to survey state systems to determine how they are sharing data, how they are disseminating data, and whether they have developed systems that could be disseminated more widely across the states. A variety of issues could be examined in such a project, from general ethical concerns to practical issues such as developing an adequate data-sharing infrastructure or building relationships between researchers and state agencies to collect and use data.
States currently collect and have access to a wide variety of data, but gaps remain. Examples mentioned by breakout group members include predictor variables, like risk and protective factors; the specific services individual children are receiving; longitudinal information on children; and the early identification of problems. Children need safe, stable, and nurturing relationships, one breakout group member mentioned, yet little information is collected directly about these relationships.
States have an opportunity as the health care system adopts electronic health records to use data from these records to further children’s health and well-being. In particular, young children often have been overlooked in the past and could be a point of emphasis in collaborative efforts. States would need to work with software developers to understand what types of data need to be collected and how those data could be collected rather than grafting data systems onto electronic health records once they are up and running.
The breakout group discussed the interactions among state-level personnel and the frontline personnel who are often collecting data at the local level. Frontline personnel need to understand the purpose of the data being collected to build support for the data collection system and to provide input on how data can be used. They also need technical assistance from the states if they are to gather information accurately and reliably. Finally, communities need to receive information back from the state to maintain and extend their data collection efforts, according to some members of the breakout group.
Finally, several presenters and participants in the workshop made comments that elaborated on or extended the observations of the breakout groups.
Felisia Bowen, Rutgers School of Nursing, emphasized the need to develop culturally appropriate instruments, given the diversity of the U.S. population. The best way to do so, she added, is to invite the members of
culturally distinct groups to share in the development of instruments, which also educates them in the process and increases the likelihood of sharing.
The children in U.S. schools speak many different languages, observed Laurel Leslie, Tufts University School of Medicine, and many live in poverty, which can be expected to have effects on their health literacy. As Hendricks Brown of Northwestern University pointed out, a tool is not adequate if it leaves out large portions of the population that have less access to health care.
Andy Shih of Autism Speaks pointed to the need to conduct field trials of new instruments in a range of cultural settings, as was done with a recent diagnostic screening tool for autism that can be administered by nonspecialists. Conducting field trials in different cultures makes it possible to identify elements that may be transcultural so that a tool can be generalized.
Robert Goerge, University of Chicago pointed out that many of the barriers to better measures are not technical but social and cultural. Laws and regulations in the United States protect privacy and independence, which limit what can be done with data after it is collected. As Mary Ann McCabe, George Washington University, added, these considerations differ among data types. For example, deidentified administrative data tends to be viewed differently than personally identifiable health care data, particularly in sensitive areas such as mental health.
David Keller observed that one way to develop new tools is to require the use of such tools in payment systems. For example, value-based payment formulas have been tied to tool development in adult health care, though the same measures have not been applied in pediatrics. In general, added Brown, actionable information from a more comprehensive and integrated data system could support the investments in infrastructure needed to build and sustain such a system. Brown also pointed to the opportunities to use qualitative information more strategically, particularly in monitoring the implementation and adaptation of programs.
Harold Pincus observed that, in health care, one challenge is to redesign the workforce to establish base measurements of care, and an electronic health record (EHR) can be built around that. In that case, an EHR would be usable not just by clinicians but by patients and parents. The Office of the National Coordinator for Health Information Technology is working on developing standards for behavioral health EHRs, and other groups are thinking about how patient portals can be used more effectively.
Jeff Schiff, Minnesota Department of Human Services, pointed more generally to the potential of new technologies to gather patient-reported outcomes. “All of us will, at some time in the next 15 minutes, look at our cell phones,” he said. “We have to start thinking about different modalities of patient-reported outcomes. And then not just look at patient-reported
satisfaction or patient-reported well-being but how people are actually functioning in their families and in their communities.”
Finally, Schiff emphasized the power of forceful advocacy. “If we can be a little impatient and a little persnickety about this and get to be a thorn in a bunch of people’s shoes, we may be able to move forward,” he said. Legislators tend to react more forcefully to stories than to data, Schiff said, so he always tries to talk about data in the form of stories. At the same time, a little bit of data can go a long way, especially if it drives home an important point. “If we can get the data to be used, we can make a difference.”
This page intentionally left blank.
