Lessons Learned from Other Policy Domains
The workshops provided an opportunity to learn from experience with performance measures in other domains of public policy. Some faced challenges similar to those confronting the early childhood care and education field, even though specific solutions may not be directly applicable. This chapter provides highlights of presentations by representatives from the Maternal and Child Health Bureau, the Family and Youth Services Bureau, and the National Governors Association.
MATERNAL AND CHILD HEALTH BUREAU
At the federal level, the Maternal and Child Health Bureau (MCHB) is charged with the primary responsibility for promoting and improving the health of the nation's mothers and children. Part of the Health Resources and Services Administration of the Public Health Service of the U.S. Department of Health and Human Services, MCHB administers Title V of the Social Security Act, enacted by Congress in 1935. Title V is the oldest existing federal-state partnership, providing funds through the Maternal and Child Health Block Grant as well as through other programs. To measure effectiveness, MCHB necessarily had to assess the individual states' maternal and child health programs as a gauge of the Title V program nationally.
MCHB built its performance measurement system over a two-year period, said Gary Carpenter of the MCHB Division of State and Commu-
nity Health. It created a committee involving representatives from the State Maternal and Child Health and Children with Special Health Care Needs programs; academics in both public health and maternal and child health; staff of the National Governors Association; staff of the National Association of Maternal and Child Health programs; and staff of the Health Resources and Services Administration. This diverse and sizeable team determined a set of core performance measures states could use to show the progress of their maternal and child health programs.
The MCHB committee initially struggled over whether to start the process from the top down by setting national goals and then arraying state performance measures under them, or to start from the bottom up by amassing examples of performance measures from state maternal and child health programs as well as from academia, professional associations, and institutions and then developing national goals that reflected those measures.
Taking the latter approach, the committee collected maternal and child health measures and organized them by populations served, availability of data, and consistency of measures. The initial list of measures approached 300, with duplication as well as gaps. Next, the committee set criteria for paring down the list and filling in gaps, including the ability of the states to measure changes in the maternal and child health programs and the availability of desired data. Through its Office of State and Community Health, MCHB took the lead in guiding the committee to select a final set of 18 national performance measures that went into the new guidance to states.
One year after the committee began its work, draft measures were presented to all state maternal and child health directors for their review and comment. Once state comments were collected, a single document was compiled that outlined the process for capturing the data required by the Title V legislation, incorporated the revised performance measures, and allowed the states the flexibility to add their own performance measures based on their perceived needs. When the first draft was completed, 17 states volunteered to pilot test the new procedures.
During the pilot test, MCHB guided development of an electronic reporting system to allow the states to submit their applications and annual reports and to capture all the quantitative data necessary for performance measurement as well as qualitative programmatic information. All 59 jurisdictions in the program received intensive training in the new procedures. The guidelines and database on the MCHB block grant program has become a useful tool for a number of audiences including Congress, the
Department of Health and Human Services, states, and communities. (For more information, please contact < www.mchb.hrsa.gov/>.)
FAMILY AND YOUTH SERVICES BUREAU
The Family and Youth Services Bureau (FYSB) is an agency within the Administration on Children, Youth, and Families of the U.S. Department of Health and Human Services. FYSB helps individuals and organizations provide comprehensive services for youth in at-risk situations, as well as for their families. The primary goals of FYSB programs are to provide positive activities for youth, to ensure their safety, and to maximize their potential to take advantage of available opportunities.
Currently, FYSB primarily administers three programs. The Basic Center Program provides emergency shelter for youngsters who have run away from home, often from situations of abuse or neglect, with the aim of reuniting youngsters with their families within three to five days when possible. The Transitional Living Program for Homeless Youth offers housing and a range of services including job training, counseling, and life-skills training to homeless youth between the ages of 16 and 21. The Street Outreach Program for Runaway and Homeless Youth establishes and builds relationships between staff of local youth service providers and street youth to help young people leave the streets. Local grantee programs provide a range of services directly or through collaboration with other agencies, specifically those working to protect and treat young people who have been or who are at risk of being subjected to sexual abuse or exploitation.
According to Gilda Lambert, associate commissioner of FYSB, the FYSB began the process of establishing youth development performance measures in 1997. Because the types of organizations that receive FYSB funds vary widely in size and scope, developing measurements feasible for all represented a formidable task. FYSB chose to follow a human services model for performance measurements. Like MCHB, the Bureau first convened a technical advisory group of representatives from major youth-serving organizations, foundations, and federal agencies. After getting their advice, FYSB gathered information from a wide variety of sources, including children, parents, program providers, and grantees.
Out of that two-step process, FYSB drafted measures for its three programs. At the time of the workshop, next steps called for refining the draft measures and pilot testing them. As the maternal health experience showed, pilot testing allows for critical adaptation of measurement tools to estab-
lished systems for collecting information from grantees, and for carrying out the legally required monitoring.
Lambert noted that FYSB continued to wrestle with major issues raised throughout the workshops. One is determining how to balance desirable outcomes for young people with what programs actually offer so they can fairly be held accountable for specific outcomes. She emphasized the benefits of bringing youth-serving organizations into the performance measures development process to find common ground and to focus on positive youth development. (For more information, please contact <www.acf.dhhs.gov/programs/fysb/>.)
NATIONAL GOVERNORS ASSOCIATION CENTER FOR BEST PRACTICES
The National Governors Association (NGA) Center for Best Practices helps governors and their key policy staff develop and implement innovative solutions to governance and policy challenges facing them in their states. The center works extensively in the areas of welfare, welfare to work, juvenile crime, and early childhood education and has developed performance measurements that correspond to these programs.
Evelyn Ganzglass, Director of Employment and Social Services Policy Studies at NGA's Center for Best Practices, presented information about the organization's work in performance standards and measurement. She noted that the organization is partnering with federal agencies in developing the performance measurement system for the Personal Work and Responsibility Act. Performance measures are a management tool for use at the system, program, and individual provider level to improve the performance of the system. Thus, they need to be clear and understandable and to reflect the mission of the system, the program, the provider, and the institution that delivers the services—much easier to say than to do.
In the case of the workforce development system, for example, many of the programs are state funded and services—which include employment and training programs, adult education, and vocational education at the postsecondary level—are delivered by a wide array of agencies in a number of settings. Reconciling accountability for the vocational education component, which is linked to education reform, into the main objective of workforce development has proved challenging. Many providers also are responsible to a variety of social service systems and must report on different measures, some of which are incompatible or are in conflict This situa-
tion is analogous in some respects to the challenges faced by child care and early childhood services, which encompass objectives such as care and supervision, school readiness, and aspects of the workforce and welfare-to-work system.
Ganzglass said she saw a clear shift away from process measures and standards to outcome measures, but clearly both exist and they need to coexist. In the workforce development system, lessons from the private sector on continuous improvement are being studied. Performance measures drive behavior so it is important to articulate measures and to set a time for starting the system so that data collection systems can be put in place. Performance standards are most effective in influencing performance if there are consequences, she noted. In vocational education, some states were not willing to take action for poor performance at the local level. Once incentives were put into place, however, much more attention was paid to the quality of the data and to the performance standards themselves. Under the 1998 Workforce Investment Act (P.L. 105-220), performance bonuses actually bring together three separate programs to improve performance across the system. She pointed out that performance measures that are feasible in a block grant program are going to be very different than what's appropriate in, for example, Head Start where there is much closer reporting and accountability. (See Chapter 4 for a discussion of Head Start performance standards.)
In sum, lessons from other policy domains suggest three points on which to focus. First, involve stakeholders from beginning to end. Second, set a date for implementation and start measuring, fine-tuning over a period of time. Third, assign a strong decision maker to lead the effort.