8

Systems-Level Issues

Key Points Raised by Individual Speakers

•  International comparisons can reveal the potential ways in which characteristics of child protective systems affect outcomes.

•  Differential response systems are an alternative to traditional child welfare systems designed to produce better outcomes with reduced costs.

•  Experimentalist reforms go even further by devolving responsibility for tailoring treatment programs and monitoring results, with review by peers, to individual case workers.

The final session of the workshop examined several issues at the broadest and most multidisciplinary level—the systems level. International comparisons of systems can reveal the key features of any one system and possible ways to modify those features. System-level analyses also can suggest both incremental and radical reforms designed to achieve particular goals.

A CROSS-NATIONAL VIEW OF CHILD
PROTECTIVE SYSTEMS

International comparisons can reveal not just the differing characteristics of child protective systems, but potential ways in which those characteristics affect outcomes. They can highlight differences in the incidence, prevalence, exposure, risk, and burden of child maltreatment. They shed light on different approaches to monitoring and evaluation, which in turn can be related to the epidemiological measures of a system.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 81
8 Systems-Level Issues Key Points Raised by Individual Speakers  International comparisons can reveal the potential ways in which characteristics of child protective systems affect outcomes.  Differential response systems are an alternative to traditional child welfare systems designed to produce better outcomes with reduced costs.  Experimentalist reforms go even further by devolving responsibility for tailoring treatment programs and monitoring results, with review by peers, to individual case workers. The final session of the workshop examined several issues at the broadest and most multidisciplinary level—the systems level. Interna- tional comparisons of systems can reveal the key features of any one sys- tem and possible ways to modify those features. System-level analyses also can suggest both incremental and radical reforms designed to achieve particular goals. A CROSS-NATIONAL VIEW OF CHILD PROTECTIVE SYSTEMS International comparisons can reveal not just the differing character- istics of child protective systems, but potential ways in which those char- acteristics affect outcomes. They can highlight differences in the incidence, prevalence, exposure, risk, and burden of child maltreatment. They shed light on different approaches to monitoring and evaluation, which in turn can be related to the epidemiological measures of a system. 81

OCR for page 81
82 CHILD MALTREATMENT RESEARCH, POLICY, AND PRACTICE They reveal valuable information on costs, the distribution of resources, and the responsiveness of clients and systems to the supply of and de- mand for services. Finally, they reveal different approaches to decision making, including the nonrational factors that affect the psychology of decision making including decision-making thresholds. John Fluke, vice president of the Children’s Innovation Institute at the American Humane Association, summarized a recent article in Lancet that examined the policy implications of cross-national differ- ences in the responses of child protective systems (Gilbert et al., 2012). Before discussing the Lancet article he referred to two sources of data that have been used in cross-national approaches: self-report data and administrative data. The World Health Organization has identified self- report data as the standard for understanding the epidemiology of child maltreatment. Examples of instruments that collect such data are the Parent-Child Conflict Tactic Scales, the Child Abuse Screening Tool developed by the International Society for Prevention of Child Abuse and Neglect, the Juvenile Violence Questionnaire, and the Multiple Indi- cator Cluster Study developed by the United Nations Children’s Fund (UNICEF). The article in Lancet drew primarily from administrative da- ta, though the availability of such data depends on the existence of a functioning service delivery system. In some countries, information from health care systems can be a valuable addition to sparse or missing data from service delivery systems. As an example of self-report data, Fluke cited international results from the UNICEF Multiple Indicator Cluster Survey on physical disci- pline. It found that physical punishment of children is less likely in households that share the belief that physical discipline is not necessary in child rearing, whereas physical discipline occurs more often in house- holds that do see physical discipline as necessary. Such data indicate the feasibility of achieving a “broadband view from an international perspec- tive across many low- and middle-income countries,” said Fluke. A Six-Nation Comparison The Lancet study looked at policies and practices in six high-income countries or parts of countries: England, Manitoba (in Canada), New Zealand, Sweden, the United States, and Western Australia. These coun- tries are facing similar challenges and have relatively consistent enumer- ation methods that make it possible to look at multiple indicators. In particular, the study looked at children younger than 11 years old and drew data from hospital admissions and CPS agencies.

OCR for page 81
83 SYSTEMS-LEVEL ISSUES The analysis did not find consistent upward or downward significant trends in most forms of child maltreatment behaviors and reports when comparing the six countries. However, comparisons of measures across countries were more revealing. For example, the rate of violent deaths, drawn from health care data, was more than fivefold higher in the United States than in Sweden. In general, low levels of maltreatment indexes in Sweden and high levels in the United States are consistent with extremes of child poverty and support for parenting, Fluke said. Also, hospital admissions for various forms of maltreatment, gath- ered from analysis of International Classification of Diseases (ICD) codes, were comparable between Sweden and Western Australia, but the rate for the United States was substantially lower. This raises “intriguing questions,” said Fluke, about access to health care and the effects of uni- versal health care services on use. The study did not find a consistent decrease in the maltreatment indi- cators it gathered. While policies might be effective in protection of some vulnerable groups of children, they may be failing to reach others, particularly younger children, Fluke said. Future Research Using Cross-National Analysis The study emphasized the importance of measuring multiple indica- tors, especially for administrative data. It also highlighted the importance of consistent methodologies, especially when comparing trends, and the need to cross-link data more systematically. All countries have limited measures, and these limits are in different domains. In general, variability in social services data is greater than for healthcare data. Also, it can be difficult to separate cultural factors, such as attitudes, from measurement biases. Fluke concluded by listing some of the questions that cross-national analyses raise for him. How can methodologies be improved over time? Can the results be clearly linked to different policies? Can cross-national insights lead to better understanding of national data? Can cross-national interpretations of data lead to better understanding of policies and their impacts? What can cross-national analyses reveal about the access and availability of services?

OCR for page 81
84 CHILD MALTREATMENT RESEARCH, POLICY, AND PRACTICE ALTERNATIVE CHILD WELFARE SERVICES APPROACHES An innovation that has occurred since the 1993 NRC report has been the development and implementation of differential response systems, which are also known as alternative response, dual track, multitrack, or multiple response systems. Differential response provides a flexible ap- proach to child welfare service delivery, explained Richard Barth, dean of the School of Social Work at the University of Maryland. Referrals with severe child maltreatment or imminent risk of further abuse are as- signed to an investigative pathway, which is sometimes called the formal pathway. Referrals initially classified as low or moderate risk enter a noninvestigative pathway, sometimes called the assessment pathway, which emphasizes family engagement and services outside of the court’s purview. About a third of the states have a third pathway, sometimes called the prevention pathway, which provides some degree of services for screened-out cases or cases that do not meet the statutory require- ments for child maltreatment. A key feature of differential response systems is the capacity to reas- sign families to another pathway. For example, if a service agency were to perceive a higher risk, a family could be reassigned to the traditional child welfare pathway. This can be a problem in some differential re- sponses systems if barriers exist to the reassignment of cases. Also, differential response programs will create significant problems for child protection if the use of data about the prior involvement of the family with CPS is limited because prior reports are among the best pre- dictors of the seriousness of the threat to children. States should be dis- couraged from destroying information about cases that have ended up in alternative response. Pathway responses need to be codified, said Barth, so that alternative responses are delivered in a consistent way that can be described. At this time, there is no “best practice” for families who are in any of these al- ternative pathways. Also, noninvestigative pathways are generally volun- tary, and the cases do not have a determination of maltreatment. As many as 20 states, and possibly more, have differential response systems, Barth said, with additional systems in pilot stages (Figure 6). With recent legislative changes this number may be closer to 30 states.

OCR for page 81
85 SYSTEMS-LEVEL ISSUES 20 18 Statewide 16 Pilot 14 Number of States 10 10 13 12 12 10 10 5 6 6 8 10 9 8 4 4 6 2 2 4 88 7 2 66 66 66 1 55 5 5 2 2 44 2 2 1 1 0 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Year FIGURE 6 Implementation of differential response systems either statewide or as pilot programs since 1993. SOURCE: QIC-DR, 2011, p. 8. Purposes of Differential Response Differential response has several purposes, said Barth. It provides improved matching of cases to services, rather than matching progress through the child welfare system to the availability of evidence. It also enables better engagement of clients without the involvement of the court. Of course some people insist that court involvement is essential to engaging clients, Barth added, but even court-involved clients often do not complete basic requirements such as parent training. Differential response can reduce court and service costs by reserving resources for severe cases and forensically managed cases. Also, it can produce better engagement of community agencies in serving clients connected with child welfare services because alternative response sys- tems usually have close ties with community agencies. Differential re- sponse can support child maltreatment prevention with high-risk cases.

OCR for page 81
86 CHILD MALTREATMENT RESEARCH, POLICY, AND PRACTICE The overall goal of differential response, said Barth, is for children and families to “get services they need when they need them. They would get help that feels like help.” Existing Research on Differential Response Several randomized controlled trials have been completed on differ- ential response systems, and others are under way. Other studies have used quasi-experimental designs or administrative data to determine whether children who went down an alternative pathway were safe. These studies have found that differential response does not result in increased reports of harm to children, though the studies with direct as- sessment of harm have not been able to rule out the possibility that in- creased harm might occur. Differential response also appears to increase family engagement. “Families report very positively about it,” said Barth, and “for the most part, workers report very positively about it.” Child welfare workers who have the choice to work in the alternative track often choose to do so, and they often report greater satisfaction un- der the noninvestigative pathway, even though differential response typi- cally does not reduce their workload. Future Research on Differential Response Barth provided a list of questions that would be useful targets of re- search on differential response; many are taken directly from or based on questions from a literature review funded by the Children’s Bureau on differential response in child protective services (QIC-DR, 2011).  What is the response, and what do differential response workers do?  Which aspects of differential response implementation are plau- sibly linked to improving outcomes for children and families?  Are the positive effects on families due to assignment to a non- investigative pathway, so that families are not further harmed by involvement in the child welfare system, or to actual provision of services?  How do criteria for assignment influence the effectiveness of the noninvestigative pathway (e.g., should some cases only be served under the formal/forensic system)?

OCR for page 81
87 SYSTEMS-LEVEL ISSUES  How does participation in the noninvestigative pathway differen- tially affect families with different demographic, social, or cul- tural characteristics?  Does assignment to a noninvestigative pathway affect child and family well-being beyond safety?  What is the total cost-effectiveness of differential response when costs to other service and support systems are considered?  Will the key findings for child and family outcomes hold up un- der more rigorous evaluation designs?  What is the impact on the child welfare system as a whole when multiple pathways are incorporated into an agency’s response to allegations of maltreatment?  What changes in administrative data collection and analysis will best capture the impact of differential response?  Given the limited evidence that cases are being referred from differential response back to CPS agencies, why is this not occurring? A critical challenge, said Barth, is to change the distinction between substantiation and nonsubstantiation. Descriptions should be more accu- rate so that, for example, “sexual abuse” has categories that give infor- mation about the abuse, including its severity. “This is a real opportunity, as we rethink the reporting that is required for alternative response sys- tems, to also include measures of severity.” Barth also pointed out that differential response systems may be a place to experiment with concepts of child-centered social services homes or medical homes. Cases could be treated as either open or not open, with a behaviorally anchored severity scale that transcends victim- ization. Such a system would change thinking about recurrence and about how people are brought into child welfare systems. ROLE OF CLASS-ACTION SUITS IN BUILDING EVIDENCE-BASED CHILD WELFARE SYSTEMS In his presentation at the workshop, Charles Sabel, the Maurice T. Moore Professor of Law and Social Science at Columbia Law School, professed to be engaged in a classic bait and switch. He said that he would discuss class-action lawsuits as a way to influence child welfare systems, but such suits are neither necessary nor sufficient for improving

OCR for page 81
88 CHILD MALTREATMENT RESEARCH, POLICY, AND PRACTICE the organization of child welfare services. They mitigate the most horrif- ic circumstances, he acknowledged, but they generally do not produce sustainable improvement in organizations, and their cost-benefit balance is unclear. However, structural litigation offers one great advantage if it can open up spaces for experimentation, including ongoing experiments in several states that have created a new model of welfare provision. More- over, federal efforts increasingly reflect and reinforce such reforms. These innovations deserve to be more rigorously studied, Sabel said. The Problem with Bureaucracy The problem is easy to state, said Sabel: Big bureaucracies do not work. The reason is that the lowest level of “street-level” bureaucrats— the case workers, teachers, police officers, and so on—exercise discre- tion in an unobservable way. They have many rules to follow, and they decide which rules to apply. “No one sees them doing the rule applica- tion. And if you try and review what they do later on, you repeat the same problem. Different judges or administrative judges come to differ- ent conclusions about what was done.” This discretion sets intrinsic limits to the utility of large organiza- tions as instruments of public policy, Sabel observed. Adding more regu- lations makes the organization less responsive to dynamic changes in its environment and offers even more discretion to street-level bureaucrats. Public bureaucracy is even worse because successive changes in upper level administration lead to a hodgepodge of conflicting instructions. Furthermore, public-sector unions can accelerate the rate at which these problems compound. The early class-action lawsuits contributed to this problem because they imposed more rules. They were preoccupied with deadlines, quanti- tative measures, and specific procedural and documentation require- ments. The New Public Management (NPM) movement of the 1990s tried to break this logjam, but it, too, defined narrow goals to allow for precise measures of progress and clear incentives, sometimes embedded in contracts with private parties. In this way, NPM fragmented and sepa- rated services even as demands for coordination and integration were growing.

OCR for page 81
89 SYSTEMS-LEVEL ISSUES An Alternative to Street-Level Bureaucracy There is an alternative to this approach, said Sabel. It is to acknowledge and authorize low-level discretion rather than try to elimi- nate it. In this model, the case worker is tasked not with determining which clients are eligible for which programs. Rather, the case worker devises, in consultation with the client and a team of expert service providers, a plan that brings the relevant resources to bear on the client’s problems. As a condition of this autonomy, the frontline worker (or, increasingly, the multiprofessional, frontline team) provides a detailed report on the client’s progress under the plan and evaluates progress by agreed met- rics. The plan and monitoring reports are in turn reviewed by a group of peers in the light of experience with comparable situations. These re- views can prompt revisiting of the metrics and decision-making rules. This system makes the topsy-turvy world of street-level bureaucracy accountable and capable of learning from diverse experiences. Errors can be corrected and innovative solutions identified and generalized. “Instead of hiding the discretion, you make it more transparent. People have to say what they are doing, and they have to say it in front of peers who understand their situation very well.” Peer review creates a mechanism for dynamic or forward-looking accountability. The frontline worker is accountable when he or she can justify the actions taken as in the best interest of the client given the overarching purposes of the organization and the range of results obtainable in similar cases. This approach in- volves compliance with peer judgments rather than rule accountability. When this requires deviation from the rules, the rules are reexamined in light of the higher purposes they serve. It also fosters institutional learn- ing, allowing identification and correction of local error, detection of dead-end policies, and scrutiny and eventual generalization of promising successes. Experimentalist Reform in Child Welfare This system, known as experimentalist or pragmatist reform, has been partially implemented in several states. In Utah, for example, the key instrument is called a quality service review (QSR). The QSR begins with selection of a stratified random sample of cases. An agency official and an outside reviewer examine each case over 2 days, including a file review and interviews with the child, family members and other caregiv- ers, professional team members, and others. The reviewers score the

OCR for page 81
90 CHILD MALTREATMENT RESEARCH, POLICY, AND PRACTICE cases in terms of two sets of indicators: a “child and family status” that measures the well-being of the child and his or her family, and a “system performance” that reflects the capacity to build teams, make assessments, formulate and update plans, and execute those plans. The initial scoring is then refined in a series of meetings, with a final report setting out the aggregate scoring, identifying recurring problems, and illustrating these from specific cases (see Sabel and Simon [2011] for additional discus- sion of the QSR). “The idea is that systematic problems—a breakdown in one place in the system—are evidence of systematic failures in the organization of routines that are producing bad decision making,” said Sabel. Participa- tion in this process is a way of training the people in the system, so long as they are open to changing how they normally do things. Similarly, QSR data function as a measure of performance and as a diagnostic tool of systemic reform. The scores can be compared over time and (in prin- ciple, though not yet in practice) across states, giving rough but service- able indications of where attention and remedial effort should be focused. The second experiment Sabel mentioned is the federal Child and Family Service Reviews (CFSRs). Launched in 2000, the CFSR com- bines review of aggregate case processing and outcome data with qualita- tive peer review of a sample of cases. It has become less punitive and more diagnostic and remedial, and key federal officials see it as a form of “continuous improvement.” CFSR has prompted some states to start qualitative monitoring. In fact, some states have used QSR-type data to challenge the findings of federal CFSR review. “That’s the test that a system is working—that it induces reflection on the lower level that compels reconsideration on the higher level.” Limits and Future Opportunities Today, the audit sample of the CFSR is small even by QSR stand- ards. The performance measures blur process and outcome in a way that impairs their diagnostic value, and their results are not compiled or dis- closed in a way that facilitates comparison across states (Noonan et al., 2008). Most importantly, the CFSR currently selects its own cases for review, while a more plausible procedure would re-review a sample of the cases from the state’s process. That way, federal efforts would lever- age the states’ process and integrate with them directly. But these flaws seem remediable, said Sabel; for example, the CFSR could complement innovative state reforms.

OCR for page 81
91 SYSTEMS-LEVEL ISSUES So far, QSR and experimentalist child welfare services lack system- atic evaluation of outcomes with random assignment of cases. That might seem incompatible because random assignment evaluation measures the effect of a precisely defined and consistently applied proto- col while the aim of experimentalist service provision is to adjust the protocol as the treatment continues. But the “guidance-based” or “treat- ment to target” strand of the evaluation literature measures the effects of organizational arrangements for customizing therapy. If so, QSR-type reviews can serve as a useful starting point for rigorous evaluation. Better integrated with an improved system of state QSRs, the CFSR could become a national framework for rebuilding the child welfare sys- tem from the bottom up, Sabel concluded. It could also provide Congress with a forward-looking form of oversight that would enhance accounta- bility while improving the capacity of state systems to learn. Discussion In response to a question about whether potential liability is a factor in the decision making of child welfare workers, Sabel said questions of liability can never be completely eliminated. However, one way to inter- pret the firing of child welfare directors is that it connotes greater public accountability for these systems. “After decades where the liability was always pushed down and the people on the top never left, there is a win- dow in which they are being held accountable. If it persists too long and there is nothing but churning, people will turn away in despair and the results will be devastation. [But] this is a moment when that high-level accountability has to be turned to good purpose.” LEADERSHIP AND MANAGEMENT IN CHILD WELFARE AGENCIES At the beginning of the final discussion session, David Sanders, the executive vice president of systems improvement at Casey Family Pro- grams, offered reflections based on his experience as someone who has spent many years directing large public agencies as well as several re- flections on earlier presentations.

OCR for page 81
92 CHILD MALTREATMENT RESEARCH, POLICY, AND PRACTICE Leadership in Child Welfare Agencies Sanders spent 13 years as a child welfare director in large urban set- tings, first in Hennepin County (Minneapolis and suburbs) and later in Los Angeles County. He discussed the demands on child welfare direc- tors in large urban settings and implications for leadership of the agen- cies. He said that he was asked to run child protection in Hennepin County after a crisis that resulted in the firing of everyone above him in the organizational structure. He added that, at the time, “I had no experi- ence with child protection. I had never set foot in a family’s home, but I will tell you that this is not an uncommon experience for many people in child welfare.” In Los Angeles County, he was the fifth director in 5 years, and the ninth in the 20 years that the department had existed. Though he was not fired or forced out of his job, most of the other direc- tors of the Los Angeles County child welfare agency that came before or after him were fired because of crises in the child welfare system. This pattern is common throughout the country, he said. He noted that this situation often results in leaders with little preparation for the job, who find the position is fraught with challenges, and who lead “a day to day existence.” Continuous and sustainable improvement is difficult in such a situation. “This issue of leadership turnover and leadership preparation, and of strengthening and sustaining leadership, is absolutely critical,” he said. Sanders also discussed leadership and management issues that arise from the perception that public expectations are not being met. In both systems that he led, child safety was the initial priority. In Los Angeles County, for example, the agency investigated 170,000 complaints of abuse and neglect every year, and there was zero tolerance for any error. He described how most of the agencies’ effort was dedicated to these investigations and to foster care, with little attention and resources avail- able for implementing evidence-based practices that will improve out- comes for children. Although child safety is critical, he said, it is also critical to expand the focus to think about what well-being looks like. Finally, Sanders noted that as director he had to spend at least 50 percent of his time dealing with the media, elected officials, and other non-practice-related issues. It is critical, he said, to think more about how systems can be supported to make the changes necessary to incorporate innovative practice.

OCR for page 81
93 SYSTEMS-LEVEL ISSUES Management in Child Welfare Agencies Sanders emphasized many important issues that previous presenters raised, including issues of worker turnover, distress, capacity to take on new initiatives, capacity to think differently, and management capacity to think differently. He also touched on the role of labor unions in child welfare agencies. In Los Angeles, all changes in workload had to be ne- gotiated. For example, if implementing a new program meant even one extra hour of time over the course of a year, it would have to be negotiat- ed with the union. This meant that the implementation of new evidence- based practices would inevitably be subject to trade-offs and considera- tions about which issues to raise with the union. Sanders also discussed the impact on workers of implementing new evidence-based practices. Workers will stay in organizations where they believe they are doing effective work, but many also feel overwhelmed. Careful thought needs to be given to how to introduce changes in such a setting. Finally, Sanders discussed the implementation of evidence-based practice in the larger context of system-reform. “How [do] you make sure that it really is incorporated into the planning and management and not seen as separate work?” He noted that the challenge is not whether or not there is a good set of evidence-based practices, but rather how to im- plement them across a large organization with many workers. In Los An- geles, he said, they focused on a small set of outcomes and a small set of strategies. He also suggested implementing evidence-based practice by blending research with child welfare services. Research results can pro- vide support to leadership and justify systemic reforms. “It is a way of incorporating evidence-based practice much more quickly into child wel- fare agencies than any other structure.”

OCR for page 81