CHAPTER 8
Data and Technical Assistance Resources

  • Participates in volunteer activities

  • Has a positive outlook

  • Reads for pleasure

  • Misses school regularly

  • Has been arrested

  • Participates in a gang

Over the past decade, communities, cities, states, and nations have become increasingly interested in knowing what percentage of their youth population is doing well and what percentage is doing poorly (Brown and Corbett, forthcoming; Kingsley, 1998). To estimate these numbers, scientists and policy makers have generated both a growing list of social indicators of well-being and problem behaviors and set about gathering the data. As a result, there are now data and related technical assistance resources to support youth development work. This work, however, has also produced a keen awareness among practitioners of the limitations of these resources and the need to increase their quality, breadth, and availability in the coming decade (MacDonald and Valdivieso, 2000).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 228
Community Programs to Promote Youth Development CHAPTER 8 Data and Technical Assistance Resources Participates in volunteer activities Has a positive outlook Reads for pleasure Misses school regularly Has been arrested Participates in a gang Over the past decade, communities, cities, states, and nations have become increasingly interested in knowing what percentage of their youth population is doing well and what percentage is doing poorly (Brown and Corbett, forthcoming; Kingsley, 1998). To estimate these numbers, scientists and policy makers have generated both a growing list of social indicators of well-being and problem behaviors and set about gathering the data. As a result, there are now data and related technical assistance resources to support youth development work. This work, however, has also produced a keen awareness among practitioners of the limitations of these resources and the need to increase their quality, breadth, and availability in the coming decade (MacDonald and Valdivieso, 2000).

OCR for page 228
Community Programs to Promote Youth Development In this chapter we review these resources in light of the youth development framework developed in Part I and the variety of practical applications for which the data and resources are needed. Our review of data and resources includes: (1) relevant administrative and vital statistics data that are commonly available at the community level, (2) community surveys and topic-specific instruments that can be used to track aspects of youth development not well covered by administrative data, and (3) selected national surveys focusing on youth. We then discuss data collection to support implementation and reflective practice in individual community programs for youth. Finally, we review the efforts of key national intermediary organizations that have developed resources and technical assistance materials for use by local youth development efforts interested in social indicator data. The chapter concludes with suggestions regarding: (a) the need for enhanced access of community youth programs to social indicator data and to the training needed to use them effectively, (b) the development and fielding of new measures, and (c) an expanded role for national intermediaries to support these goals. USES OF SOCIAL INDICATOR DATA The uses of social indicator data for community programs for youth and youth initiatives include needs assessment, service targeting, goals tracking, program accountability, and reflective practice to improve program and policy effectiveness over time. Any assessment of available data resources to support this work needs to be understood in the context of their use. Needs and Resource Assessment Assessing the needs of youth in a community is a common starting point for many community youth initiatives. Whole communities may do a general assessment using available data to identify the areas of greatest need for their youth, which can then be addressed in a coordinated fashion by multiple programs and agencies. Organizations (e.g., the United Way, the YWCA, a local synagogue) can also use such data to shape their own programs, targeting their efforts in areas of greatest need (United Way of America, 1999). Strengths in the community (cultural, financial, etc.) that could be mobilized to meet the needs are often identified as part of the same process. Most communities are limited to available administrative data

OCR for page 228
Community Programs to Promote Youth Development sources, such as service program data, health surveillance, vital statistics, school performance data, decennial census, police reports, and so on. Because of the limited nature of these data, some communities are fielding their own surveys for a more complete assessment. In many cases, however, community programs for youth are started in response to needs that are obvious to those who work with the youth without consulting any data resources. Nonetheless, however, a more formal documenting of need can help even in these cases to elicit the cooperation and interest of other actors both inside and outside the community, including funders. Tracking Progress Toward Goals Virtually all youth initiatives have some goals that can be translated into social indicator language. Community-wide youth initiatives often set goals to improve selected dimensions of youth well-being in specific and measurable ways. These goals then serve to focus the activities of multiple organizations in the community. In Tillamook County, Oregon, for example, the community agreed to focus on lowering the teen birth rate, which was then one of the highest in the state. Many local organizations, some with very different ideas and strategies for addressing the issue, worked to reduce teen births in the county. Over a period of four years, the rate was cut by 70 percent, giving the county the lowest teen birth rate in the state (National Campaign to Prevent Teen Pregnancy, 1997). The federal Healthy People 2010 initiative is another useful example. Participating states and communities adopt specific goals across a host of health outcomes to be achieved by the year 2010.1 Public and private health organizations focus their efforts on one or more of these goals so that, together, they can make measurable progress at the community or state level. Social indicator data are the primary source used to assess this progress. Individual programs can also use social indicator data as tools for setting and tracking progress toward their specific goals. Accountability Social indicator data are increasingly used by funders to hold programs and initiatives accountable for measurable results in many areas, 1   For more information, visit <http://web.health.gov/healthypeople>

OCR for page 228
Community Programs to Promote Youth Development including youth well-being and development. Successful improvement in the targeted areas of development may be met with increased funding. Failure to meet specified goals may result in the provision of technical assistance to overcome problems or in the loss of funding or autonomy. Such accountability practices are becoming increasingly common in state and local educational assessment efforts, for example (Brown and Corbett, forthcoming). Community-level indicators are rarely used for this purpose, as individual programs are not generally expected to have a community-wide impact on youth well-being. Instead, they are held accountable for outcomes to program participants. Reflective Practice Community programs for youth can also use social indicator data to monitor their own success and to guide program refinement (i.e., they can use social indicator data as part of reflective practice). At its most formal level, programs can develop a detailed model relating program activities to interim and long-term project goals for participating youth (United Way of America, 1999; Gambone, 1998; Weiss, 1995). In the best case, such a model of the links between program activities and youth outcomes will be based on existing research (when it exists), theory, and the shared beliefs of those running the program. Both program activities and youth outcomes can then be measured and tracked over time. Failure to produce the expected results could indicate inadequate implementation of some part of the program, or it may call into question one or more of the underlying assumptions of the model. Practices, the model, or both can then be reevaluated as a result and the programs can be modified. In many respects reflective practice functions like program evaluation, even though it lacks the methodological rigor required to draw firm causal inferences about the relations between program activities and youth outcomes (see Chapter 7 for discussion of evaluation methodologies). However, the level of certainty required to qualify results as scientific knowledge is not needed to produce good guides for responsible program management. And programs can sometimes incorporate some of the practices associated with experimental evaluations into their reflective practices. For example, they can compare the social indicator data from their catchment area with social indicator data being collected longitudinally in other comparable catchment areas (stimulating a quasi-

OCR for page 228
Community Programs to Promote Youth Development experimental design, provided that the data are being collected over time beginning before the program change is put into place). DATA SOURCES Communities and youth programs interested in tracking indicators of youth development have two major sources of data to draw from: administrative and related data sources (e.g., school records, crime reports, social service receipt, vital statistics, decennial census) and surveys. Every community already has data relevant to youth development from its administrative data collection efforts, although there can be substantial differences across communities in the accessibility of those data to the public, even among the agencies that collect them. Few communities go to the added effort and expense of collecting survey data, although this is an excellent way to achieve a complete picture of the status of youth and the social factors affecting their development. Over the past decade, however, the number of communities conducting surveys has increased substantially. In this section we review the data resources available to communities through the lens of the youth development framework developed in this report. This framework identified four outcome domains of development—described as personal and social assets—in Chapter 3 and eight social setting domains—described as features of positive developmental settings in Chapter 4. We also include “negative outcomes and behaviors” as a separate outcome domain. We examine the types of indicator data available in each of the domains in our framework for commonly available administrative data, and for three of the more advanced survey instruments used for assessing youth development at the community and state levels. These are reviewed in terms of their coverage across the domains, data and measurement quality, and the extent to which their measures are grounded in the scientific literature. Topic-specific research instruments and national surveys are also discussed.2 The section finishes by considering issues of public access to the data generated by these sources. 2   Several major federal publications series, not reviewed here, provide regularly updated trend data on children and youth across a wide variety of domains. These include: Trends in the Well-Being of America’s Children and Youth (<http://aspe.hhs.gov/hsp/99trends>)

OCR for page 228
Community Programs to Promote Youth Development Administrative and Related Sources Every community collects a substantial amount of data on their children and youth. Administrative sources include school records, educational assessments, vital statistics (birth and death records), police data on reports and arrests, child welfare and public assistance records, health surveillance systems, emergency room admissions data, and so on (Coulton and Hollister, 1998; Coulton, 1995). In addition, data from the decennial census provide detailed economic and demographic information on youth, their families, and their neighborhoods. These data have a number of advantages in addition to their ubiquity. First, someone has already paid for their collection. If they are not already available to the public, they can often be made available at a relatively modest cost. Second, many of these sources are capable of generating indicator estimates down to the neighborhood level. This is particularly important for community programs for youth, many of which serve limited geographic areas within the larger community. Neighborhood-level data allow programs to make strategic location decisions to areas of greatest need, assess needs and strengths in neighborhoods they already serve, and track changes in these characteristics over time. The last is particularly important for community programs for youth when their goals extend beyond program participants to include changes at the neighborhood level (e.g., reductions in gang activity). Third, administrative data provide information that cannot be gathered using existing community surveys on youth development. Standardized education assessments, for example, are important measures that are not covered by such surveys. Neighborhood characteristics (e.g., crime levels, teen birth rates) derived from administrative reporting systems provide indicators of social settings that are more objective in the sense that they are not dependent, as the surveys are, on youth perceptions. Some potential problems with these data should be kept in mind. First, many of these sources can have problems with data quality and consistency (Coulton and Hollister, 1998; Coulton, 1998). For example, Youth Indicators 1996: Trends in the Well-Being of American Youth (<http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=96027>) America’s Children: Key National Indicators of Well-Being: 2000 (<http://www.childstats.gov>)

OCR for page 228
Community Programs to Promote Youth Development changes in police policy for the disposition of juvenile offenses may result in dramatic increases or decreases in arrest rates without any change in the actual rate of juvenile crimes committed. Even census data, which are very reliable overall, have been shown to substantially underestimate the number of both black and white children in large urban settings (West and Robinson, 1999). Second, data sources often use different and incompatible geographic units when they report (e.g., school catchment area, census tract, health district, or other specially defined service area). This can limit the ability to draw on multiple data sources to produce a picture or profile for particular neighborhoods. Other problems can include a lack of separate estimates for important population subgroups (e.g., Hispanics), long time lags between data collection and release, and a lack of public accessibility. In many communities, estimates that could support planning efforts outside local government never make it beyond the walls of the agencies and departments that collect them. Table 8–1 presents the types of indicator data available from administrative data and from the three community survey instruments we reviewed, sorted by the domains in our youth development framework. In the youth outcome domains, administrative data are quite strong in the areas of cognitive development, negative outcomes or behaviors, and physical health. They are weak sources of indicator data in the areas of psychological, emotional, and social development. And, with the exception of the cognitive development domain, the indicators are heavily weighted toward negative outcomes. This should not be surprising given the reliance on crime, vital statistics, health surveillance, and social service receipt data, all of which focus primarily on negative events. Among the social setting domains that influence youth development, administrative data are strongest in the safety, social norms, and opportunities for skill building domains. They are weak to nonexistent in the structure, emotional and intellectual support, opportunities for efficacy, and integration domains. Community-Level Surveys of Youth For this review we have chosen three of the best known and most widely used surveys of youth at the state and community levels. All three surveys are based entirely on youth reports, are commonly administered

OCR for page 228
Community Programs to Promote Youth Development in the classroom, are relatively inexpensive to administer and process, and take an hour or less to complete. These are: Profiles of Student Life: Attitudes and Behaviors; the Student Survey of Risk and Protective Factors, and Prevalence of Alcohol, Tobacco, and Other Drug Use; and the Youth Risk Behavior Survey. Profiles of Student Life: Attitudes and Behaviors The Profiles of Student Life: Attitudes and Behaviors (PSL-AB) survey, developed by the Search Institute, has been administered in over 1,000 communities since 1989. It is designed for youth in grades 6 through 12 and is based on a comprehensive framework grounded in the youth development literature (Scales and Leffert, 1999). The framework includes eight asset areas: four internal (commitment to learning, positive values, social competencies, and positive identity) and four external (support, empowerment, boundaries and expectations, and constructive use of time). There are 40 assets in all, as well as multiple measures of thriving and risk behaviors. The questions used in the survey were culled primarily from national surveys. About half of the assets are measured as scales based on three or four items. A psychometric analysis of these scales revealed Cronbach’s alpha coefficients ranging from to .31 to .82, with about two-thirds of the scales exceeding the .60 level, a common cutoff point used in research. The validity of the assets is primarily face validity based on their relationship in the research literature to the promotion of healthy behavior, the prevention of risk behaviors, or both. Analyses based on PSL-AB data revealed strong relationships between the number of assets a youth has and the prevalence of risk and thriving behaviors. These analyses are based on large but nonrepresentative and disproportionately white samples across multiple communities (see Leffert et al., 1998, for details). Although the assets and their representative measures are grounded in existing academic research, the research base is, as its designers freely admit, rather thin or mixed for a number of the assets, such as empowerment, positive values, cultural competence, self-esteem, and sense of purpose (Leffert et al., 1998). Clearly, some of the measures used in the PSL-AB race ahead of the underlying science, and a number of the constructed scales fall well be-

OCR for page 228
Community Programs to Promote Youth Development TABLE 8–1 Youth Development Outcomes and Social Settings Measures in Administrative Data and Community Surveys   Community Administrative Data Profiles of Student Life: Attitudes and Behaviors (PSL-AB) Student Survey of Risk and Protective Factors, and Prevalence of Alcohol, Tobacco, & Other Drug Use (SSRP) Youth Risk Behavior Survey (YRBS) Youth Outcomes   Cognitive development School grades; standardized test scores/assessments; high school dropout or completion rates Achievement motivation; school engagement; time doing homework; reads for pleasure Academic achievement; school engagement   Psychological development Suicides Moral character (caring, social justice, integrity, honesty); self-efficacy; self-esteem; mattering; positive outlook; depression; attempted suicide Moral character (attitudes toward negative behaviors, drug use); honesty with parents; depression Sad/depressed; attempted suicide, suicide ideation; vomit or take laxatives to keep from gaining weight Emotional development   Takes personal responsibility; plans ahead; makes choices Rebelliousness   Social development Participation rates in school clubs, organizations, and Volunteer/service activities; participation in creative Skills negotiating with parents; peer pressure Hours watching TV

OCR for page 228
Community Programs to Promote Youth Development   other extracurricular activities activities (music, theater, and other arts); participation in sports, clubs, or other organizations, in or out of school; empathy, sensitivity, and friendship skills; cultural diversity (knowledge of and respect for other groups); peer pressure resistance skills; hours watching TV resistance skills regarding drinking   Physical development Participation rates in school sports and physical education; death rate by cause; violent youth crime victimizations; incidence of school violence; prenatal care receipt by teen mothers Participation in sports activities; physical health behaviors; whether a victim of violence   Vigorous exercise; strengthening exercise; physical education classes; sports team activity; height, weight; safety (helmet, seat belt use); exercise-related injury; threatened or injured with a weapon; forced to have intercourse; tried to quit smoking; pregnancy prevention and condom use; nutrition (detailed); dieting; timing of last routine physical exam; pre

OCR for page 228
Community Programs to Promote Youth Development   vention of pregnancy, AIDS, or other sexually transmitted diseases discussed at exam; timing of last visit to dentist; use of sun screen/sun block of SPF15 or higher Negative outcomes and behaviors Teen birth rate (total and nonmarital); teen rate of STD; teen arrest rate, by type of arrest (violence, drugs, truancy); school suspension or expulsion; school dropout Drug use; sexual activity; anti-social behaviors; violence; driving and alcohol; school problems; gambling; attitudes toward drug use, sexual activity Drug use (detailed); school suspension, arrested, carried handgun, fighting; sold drugs; gang membership; perceived risk in using drugs; intention to use drugs as an adult; engaged in negative risk-taking activities Drug use (detailed); driving and alcohol; carry weapon, gun; fighting; physical fighting with boyfriend/ girlfriend; sexual activity (detailed); drugs and intercourse; pregnancy Social Setting   Appropriate structure   Clear rules and monitoring (family, school); neighbors monitor young people’s behavior Clear rules and monitoring (family); youth consulted in family decisions that affect him/her; youth perceives opportunities to shape school activities and rules Ever asked to show proof of age when buying cigarettes? Physical and Reports of abuse/neglect Feels safe in home, school, Youth feels safe at school; School or trip to school

OCR for page 228
Community Programs to Promote Youth Development on indicators of developmental quality because of their essential and unique importance for youth development programs. Although many community programs for youth recognize the value of data on developmental quality, most lack the staffing, knowledge, and other capacity to measure it. Even if programs had wanted to do so, until recently they would have searched in vain for good measurement tools applicable to youth development programs. The development of such tools has been a major challenge to the field. As a more common understanding of how community programs for youth need to be structured to promote positive development has emerged, there have been notable advances in meeting this challenge. The work of Public/Private Ventures (P/PV), in conjunction with other partners in the youth development community, exemplifies recent progress in developing indicators of developmental quality for any youth activity, regardless of its specific goals. Building on Gambone and Arbreton (1997), P/PV has developed a sophisticated set of forms and scales to assess seven dimensions of developmental quality, each with subcategories. They are adult-youth relations, peer support, quality of staff presentation, behavior management, opportunities for decision making and leadership, youth engagement in the activity, and the quality of the space or location. These seven dimensions encompass the eight factors that facilitate youth development identified in this report. Observers use the instrument first to record information on an activity, focusing on observations of positive and negative behaviors for each subcategory. They then use the observations to evaluate each dimension of the activity’s quality. The instrument seeks both quantitative and qualitative observations and judgments. After using it to record three independent observations and assessments of the activity, observers prepare an overall assessment and recommendations for improvement. To foster uniform implementation of the evaluation instrument, observers receive a manual that provides detailed instructions on how to observe and assess activities and fill out the forms. The instrument does not produce a summary measure of an activity’s overall developmental quality. Rather, it yields a rich picture of the degree of success a program is achieving along multiple dimensions that contribute to developmental quality. Independent observers have used the instrument to evaluate activities in the San Francisco Beacons Program and will use it to evaluate Extended Service Schools, supported by the DeWitt Wallace Reader’s Digest Fund. In the future, executive directors and other supervisors

OCR for page 228
Community Programs to Promote Youth Development could use it to evaluate their organization’s activities. If supervisors decide to share the evaluation forms with their activities’ leaders and tell them that the criteria on the forms are what they were going to be evaluated on, the forms could help leaders see that an activity can mean much more to youth than just an opportunity to do whatever its specific goal happens to be—making pottery, playing basketball, learning computer skills, etc. Efforts are under way to more fully automate data collection and processing via hand-held computers. Although still a work in progress, this instrument represents cutting-edge practice. Even if a program scores high on factors that promote developmental quality, it needs to know whether each participating youth experiences meaningful involvement with its positive development environment. Thus, programs also need indicators of participation over time by individual youth to measure the “dosage” of developmental factors. Yet the best that most programs currently do is collect daily attendance information. Youth, of course, may simultaneously attend more than one program, switch programs over time, or mix program participation with involvement in other activities (e.g., extracurricular school activities, music lessons, religious training) that may also contribute to positive development. High-quality dosage data therefore cannot be based entirely on individual agency records but require tracking individual youth across programs and other activities. The Community Network for Youth Development (CNYD), a local intermediary organization based in San Francisco, has advanced the use of data for reflective practice through its Youth Development Outcomes Project. CNYD grounds its work with agencies in a coherent, research-based youth development framework developed in conjunction with the Community Action for Youth Project (CAYP). CNYD brings together funders and youth-serving agencies to build consensus and agency capacity around issues of assessment, accountability, and best practice to promote healthy youth development. The outcomes project led to the creation of two surveys—one for junior high and high school youth, the other for elementary school-children—that provide concrete measures of young people’s program experiences. The questions seek to capture the extent to which participants experience the supports and opportunities that facilitate healthy youth development. The dimensions of developmental quality tapped in this survey overlap with those in the P/PV instrument but are not identical. The survey also gathers information on the duration and intensity of

OCR for page 228
Community Programs to Promote Youth Development a youth’s participation in the program. Findings that revealed program shortcomings helped engage funders and agencies in discussions of why programs were falling short in some areas and how resources might be more effectively deployed. A complementary self-assessment form completed by agency staff examines organizational practices in areas viewed as critical to supporting a program’s developmental quality. These include creating safe, reliable, and accessible activities and spaces; providing a range of diverse, interesting skill-building activities; ensuring continuity and consistency of care; high, clear and fair standards; and several others. Taken together, the participant and agency instruments provide comprehensive data to help organizations create focused strategies for improving the developmental quality of their programs, given their resource constraints. If used skillfully in a process of quality improvement, such data can raise the cost-effectiveness of an agency’s management and programming activities. The general capacity building process and the specific instruments show potential for replication, and the youth survey may be adapted for publication on the web. The CNYD and P/PV approaches to assessing developmental quality share a common theoretical framework. They overlap in the constructs of developmental quality they examine and how they operationalize those constructs. But there is an important difference between the two. P/PV focuses on assessing the practices of specific activities (e.g., making pottery) in terms of their developmental quality, whereas CNYD aims at assessing how well the overall organization fosters positive youth development via specific activities as well as its general organizational environment. Both have value. Evidence in Roth (2000) suggests that youth-serving organizations are likely to welcome assistance for building their capacity to assess the developmental quality of their programs and act on those assessments. Roth surveyed executive directors of youth-serving organizations about the goals and characteristics of their programs. Many of the program goals reported by directors match up well with the personal and social assets of positive youth development discussed in this report. She also found that many program characteristics reported by directors are closely related to our list of features of positive developmental settings.

OCR for page 228
Community Programs to Promote Youth Development Qualitative Data for Community Programs for Youth Thus far this chapter has focused on quantitative social indicators and measures of program implementation, operation, and impact. This focus reflects the fact that most resources for data collection and program evaluation have been channeled in quantitative directions. Like experimental evaluation methods, quantitative data are often viewed as more objective, easier to understand, and more highly valued by funders and policy makers than qualitative data. Nonetheless, as has been suggested in the last few chapters, qualitative data can also play important roles in the design, implementation, and evaluation of community programs for youth and aid in the understanding of the process of positive youth development. Methods to generate qualitative data on programs and the social and cultural context in which they operate include direct observation; open-ended interviews with key informants, staff, and participants; focus groups; ethnographic studies; the use of diaries kept by informants; and detailed case studies. Correctly understood, qualitative data neither are inferior to nor substitute for quantitative data (Lin, 2000). Rather, by trading breadth for depth, qualitative data can complement statistical evidence in several respects (Sherwood and Doolittle, 2000). Data from field research can play an important role in planning the specific way services are provided. For example, qualitative research on the determinants of successful mentoring relationships is fairly consistent about what practices make for effective mentoring (Sipe, 1996). Such findings can be translated into guidelines for coaching and supervising mentors on an ongoing basis. Identifying better mentoring styles would be very difficult with standard quantitative survey methods. Focus groups can be used to help design services that more fully respond to the realities of the intended clients’ lived experiences and motivations. Hence, such services are more likely to be used (see Furstenberg et al., 1992 as an example of this use of focus group data; see Branch et al., 1998 as an example in the youth program area). Detailed interviews and observations of line staff and participants are essential to determine if actual program operations are, in fact, those intended by the program planners. Such information allows evaluators to decide whether the quality of program services was sufficient to produce a fair test of whether the services made a difference. It can offer insights about the reasonableness of the underlying model and about problems likely to arise during implementation and ongoing operation.

OCR for page 228
Community Programs to Promote Youth Development These insights would also provide lessons for replication. For example, Plain Talk, a community program aimed at improving adolescent knowledge of sexuality and use of methods to prevent pregnancy and sexually transmitted diseases, took a long time to implement. Field research showed that community lay trainers needed more time than initially envisioned to become sufficiently comfortable with sexuality issues before they were ready to engage other members of the community (Walker and Kotloff, 2000). The P/PV instruments discussed earlier collect both qualitative and quantitative information for assessing the developmental quality of youth activities. The quantitative data provide a general view of which aspects of an activity are running satisfactorily and which may need improvement or intervention by senior management. But only the qualitative data can provide information on the specific behaviors of staff and youth and the details of daily activities that would allow the program managers to take concrete steps to improve operations. Qualitative data may yield more complete, more nuanced, and context-specific understandings of the nature of correlations and causal relationships in the process of youth development that have been uncovered by quantitative data (e.g., by using some of the major datasets discussed in this chapter). Similarly, qualitative data can help explain subtle causal mechanisms through which a program works (or fails to work) —the “how,” whereas quantitative analysis typically shows only “how much” the program affected an outcome of interest. Such data can play a key role in formal, extensive impact studies if gathered early in the project and used to develop hypotheses that are investigated further through subsequent quantitative work. If time permits, additional qualitative research can then elucidate the findings from the quantitative hypothesis testing and suggest yet newer hypotheses—and so on through an iterative process that may yield much richer understanding than reliance on only one kind of data. Even community programs for youth that do not undergo rigorous impact evaluation can supplement small-scale quantitative assessment of program quality with data from focus groups, individual interviews, and careful observation of program operations. The latter can aid in interpreting quantitative findings as well as suggest new questions for future quantitative assessment. Qualitative data can allow exploration of how program services are understood and experienced both by participating youth and by those who choose not to participate. Neighborhood youth may have some idea of what motivates them, what they found stimulating in a program,

OCR for page 228
Community Programs to Promote Youth Development what drew them to it, why they were prepared to give it time over a period of months or perhaps years, and whether they think that participating in it affected them in particular ways. Such data can also help program operators, funders, and policy makers understand what parents care about in programs their children may attend. They can clarify various stakeholders’ expectations for a program and what would constitute success in their eyes. Program research based on qualitative data has important limitations as well. Samples are often small and not necessarily representative, even if drawn from program participants. Verifying conclusions from qualitative data is difficult. So is generalizability. Qualitative research is costly to do well. Telling stories, however interesting or compelling they may be, is not a substitute for rigorous analysis of qualitative data. Despite these and other limitations (Lin, 2000), qualitative data can play important roles in the design and analysis of community programs for youth. National Intermediary Organizations A number of national groups have become important in assisting community programs to enhance their capacity to collect and use data in their design, planning, and evaluation efforts. A number of these are focused specifically on youth development programs. These groups are playing an increasingly important role in the education of youth program staff around youth development concepts, in organizing available data-related materials for use by such groups, and in consulting with individual programs to develop data collection and analysis strategies to support planning and reflective practice. They are providing essential information and services to the youth program community. The Youth National Outcome Work Groups, mentioned earlier, provides more than detailed information on measures relevant for youth development. They support work groups in the areas of child, youth, family, and community development. In addition to information on measures, they provide easy to understand primers for each domain of well-being covered by the work group and an extensive bibliography for those who are interested learning more about a particular domain.8 8   For additional information, visit <http://ag.arizona.edu/fcr/fs/nowg/ythindexintro.html>

OCR for page 228
Community Programs to Promote Youth Development National Youth Development Information Center is a web-based information center set up by the National Collaboration for Youth to provide comprehensive practice-related information to youth— serving organizations throughout the country at low or no cost. The site includes a section on outcome measures and program evaluation, which lists major guides on youth outcome measures and evaluation approaches. There is also a section that provides youth-related statistics and collections of statistical data.9 The Youth Development Mobilization Initiative, Center for Youth Development and Policy Research, Academy for Educational Development has a goal to ensure long-term institutional support for youth development by creating a communication network between policy makers and practitioners at the local level. It is currently developing a project, Youth Development/Community Indicators: On the Plus Side, which seeks to maximize the development of community-level social indicators using existing administrative and related data resources to support positive youth development planning and policy. It intends to work with several localities (including their current partners in Albuquerque, New Mexico, Hampton, Massachusetts, and Milwaukee, Wisconsin), as well as a national advisory board of youth development experts in this effort, to develop best practices that can be followed by other communities.10 The Aspen Roundtable on Comprehensive Community Initiatives is comprised of 33, substantive experts, policy officials, and program heads who examine and discuss issues surrounding the strengthening of Comprehensive Community Initiatives. The roundtable’s web site features a catalogue of measurement instruments related to community research. This feature, called Measures for Community Research, is one of the first resources of its kind and will serve as a clearinghouse for the collection and distribution of instruments and other tools related to key community-level outcomes. This resource includes a separate section on youth-related measures.11 National Neighborhood Indicators Project is working with local institutions in a dozen cities to develop neighborhood-level indi 9   For additional information, visit <http://www.nydic.org.> 10   For additional information, visit <http://www.aed.org/us/cyd/ydmobilization.html> 11   For additional information, visit <http://www.aspenroundtable.org>

OCR for page 228
Community Programs to Promote Youth Development cator data systems that can be used to support a variety of development activities throughout the community. Partners include groups in the cities of Atlanta, Baltimore, Boston, Chicago, Cleveland, Denver, Indianapolis, Miami, Milwaukee, Oakland, Philadelphia, Providence, and Washington, DC. In addition to facilitating a variety of peer support activities, the project is developing a National Neighborhood Data System, which will provide easily accessible data at the census tract or zip code level for major metropolitan areas, as well as a series of handbooks and other tools on the use of information in community capacity building.12 SUMMARY Community programs for youth benefit from ready access to high-quality data that allows them to assess and monitor the well-being of youth in their community, the well-being of youth they directly serve, and the elements of their programs that are intended to benefit those youth. Programs may use social indicator data for needs assessments, service targeting, goals tracking, program accountability, and in support of reflective practice to improve program and policy effectiveness over time. They also benefit from information and training to help them use these data tools wisely and effectively. In this chapter we have reviewed surveys and other measurement instruments, the data generated from these tools, and related technical assistance resources that can be used by youth development programs; we were pleasantly surprised to find a relatively rich set of tools and information. Every community already has a great deal of data collected through its social programs and educational systems, its vital statistics and health surveillance systems, and the decennial census. Some communities have been more active than others in developing these data resources to guide policy and program development for children and youth. A systematic review of the potential of these data sources to yield useful indicators could provide a guide for communities seeking to maximize the use of their own data resources at a reasonable cost.13 Even when exploited to their full potential, administrative, vital statistics, and related data sources can cover only limited geographic areas 12   For additional information, visit <http://www.urban.org/nnip> 13   The Center for Youth Development, through its proposed On the Plus Side initiative, intends to produce such a review.

OCR for page 228
Community Programs to Promote Youth Development and only some components of a youth development framework. Adding local survey data in diverse communities, as has been done in a number of states and individual communities, can help create a more complete picture. A number of states and many individual communities have fielded youth surveys at the local level. Steps should be taken to expand opportunities so that more communities can benefit from such data. For example, the Centers for Disease Control and Prevention, which sponsors the Youth Risk Behavior Surveillance survey, could expand its use in additional metropolitan areas (it is currently fielded in 16 major metropolitan areas) and work with interested states to expand its use down to the school district level. Furthermore, in order to make it a more useful tool for positive youth development, the Centers for Disease Control and Prevention could work to develop optional modules that focus on the domains of development and social settings in which it is currently weak. Research on many aspects of positive youth development is still in its early stages, with the result that many of the indicators used to represent youth development outcomes in existing surveys are not well tested. Areas in need of particular attention include life skills, social, emotional, and psychological development, as well as most of the domains of developmental settings. Most of the psychometric work that has been done on indicators used in the two most complete community surveys on youth development (the Profiles of Student Life: Attitudes and Behaviors and the Student Survey of Risk and Protective Factors, and Prevalence of Alcohol, Tobacco, and Other Drug Use) are based on cross-sectional data and, in the case of the PSL-AB, on nonrepresentative samples. Longitudinal surveys allow one to explore a crucial aspect of social indicators, namely their relationship to future development as one moves from adolescence into adulthood. The Values in Action initiative, led by psychologists Martin Seligman and Christopher Peterson, is attempting a rigorous classification of strengths and virtues that will include definitions, existing research and available measures, promotional and inhibiting factors, and other valuable information that should provide a useful information base to guide future research and measurement development in this area. There is evidence that youth-serving organizations have an interest in building their capacity to assess the developmental quality of their programs. To produce useful process evaluations, performance monitoring, and self-assessment, however, community programs for youth

OCR for page 228
Community Programs to Promote Youth Development need valid, reliable indicators of the developmental quality of the experiences they provide. The evaluation research community, working with practitioners, needs to build on recent progress made in improving the quality and scope of these indicators. Research is needed to determine whether appropriate indicators vary depending on the characteristics of the specific youth population served by a program. As understanding of the determinants of positive youth development improves, the indicators should be periodically revisited and, if necessary, revised. To this end, methods for collecting information on individual youth participation in one or more community programs also require development and implementation. At a minimum, youth-serving organizations should track individuals’ attendance and their involvement in specific activities. Data on individuals’ participation across programs and in informal activities that affect developmental assets can be gathered and incorporated into basic and evaluation research. Geographic information system databases can greatly enhance the utility of social indicator data for community programs for youth by allowing users to draw on multiple data sources to pinpoint areas of need at the neighborhood level. This is particularly important to community programs for youth, which often serve particular neighborhoods within the larger community. An increasing number of communities are putting their data into GIS systems, and some, like those participating in the National Neighborhood Indicators Project, are making them available to all community organizations. Funding to optimize systems in communities with functioning GIS systems will support the needs of local community programs for youth by increasing the relevant data available through the system and the ease with which it can be accessed. Many community programs also lack staff knowledge and the funds to take full advantage of social indicators as tools to aid in planning, monitoring, assessing, and improving program activities. Individual programs and communities would benefit from opportunities to increase their capacity to collect and use social indicator data. Greater access to professional knowledge and advice on data and measurement issues, via the Internet and through individual consultation, will allow community programs for youth to be more effective and efficient as they design their own monitoring and evaluation strategies. This will require support from funders to develop the internal expertise and external consultants needed. National and local intermediaries, like the Center for Youth Development, and state organizations, like the Indiana Youth Institute, also pro-

OCR for page 228
Community Programs to Promote Youth Development vide invaluable information, support, and professional advice to state and local youth programs. Efforts to move to Internet-based systems for documenting and disseminating successful assessment tools and protocols, administering assessment instruments, inputting the responses, and analyzing the data would also simplify, streamline, and lower the costs of collecting and inputting program data and also deserve support.