National Academies Press: OpenBook

Parenting Matters: Supporting Parents of Children Ages 0-8 (2016)

Chapter: Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness

« Previous: Appendix A: Public Session Agendas
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×

Appendix B

Clearinghouses Used to Identify Interventions with Evidence of Effectiveness

THE SUBSTANCE ABUSE AND MENTAL HEALTH SERVICES ADMINISTRATION’S NATIONAL REGISTRY OF EVIDENCE-BASED PROGRAMS AND PRACTICES1

The National Registry of Evidence-based Programs and Practices (NREPP) provides publicly available electronic access to information on more than 350 substance abuse and mental health interventions. NREPP’s registry and review system was established to provide information to the public about evidence-based programs and practices that are available for implementation. NREPP’s registry, run by the Substance Abuse and Mental Health Services Administration (SAMHSA), includes only interventions that undergo NREPP’s review process, which provides information on research quality and impacts on individual outcomes. However, it should be noted that some, but not all, of the evidence presented online has been reviewed. Therefore, NREPP’s registry is not fully comprehensive, and specific interventions are not recommended or supported. Instead, the registry is intended to be a tool for use in designing an intervention to meet specific needs.

The information provided by NREPP includes a program profile that gives

  • a description of the program, the population(s) served, and the program’s major components and goals;

___________________

1 This section was compiled from information on SAMHSA’s Website (http://nrepp.samhsa.gov) and NREPP’s database (http://nrepp.samhsa.gov/AllPrograms.aspx) [October 2016].

Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
  • key study findings and ratings for outcomes (both positive and negative);
  • a compilation of evaluations of the effectiveness of the program;
  • dissemination and implementation information; and
  • references.

The information is updated over time.

Typically, interventions that are candidates for inclusion in the registry are submitted by developers or other interested parties or found through environmental scans, such as literature searches by staff, or through agency nominations. NREPP then screens the interventions to determine whether they are eligible for review. To be eligible, an intervention must meet three minimum requirements:

  1. The intervention’s research or evaluation must either measure mental health or substance abuse outcomes or behavioral health-related outcomes for those with or at risk of mental health issues or substance use problems.
  2. Evidence of outcomes must have been found in a minimum of one experimental or quasi-experimental study.
  3. Results of the study/studies must have been published in a professional publication such as a peer-reviewed journal or included in an eligible comprehensive evaluation report.

NREPP recently revised its review criteria and ratings. The new review process is intended to improve the quality of the reviews themselves as well as the information they yield. Programs that are eligible for review are rated as effective, promising, or ineffective. These new ratings are intended to make it easier for users to find evidence-based programs that can address their specific needs. From September 2015 through June 2019, NREPP will be re-reviewing all programs currently in the registry.

Previously, programs were given a rating for the quality of research for each outcome assessed, as well as for the program’s overall readiness for dissemination, on a scale of 0 to 4, with 4 being the highest rating. Higher scores indicated stronger, more persuasive evidence. Outcomes were rated individually since programs could aim to achieve more than one outcome (e.g., decreased substance use and improvement of parent-child relationships), and the evidence for each outcome could differ. A brief description of the criteria used to rate programs is provided in Box B-1, as until the updated reviews have been completed, the results of the previous review process will be the only information available.

Now, new interventions that qualify for the registry undergo a review process that begins with information gathering and a literature search for

Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×

relevant evaluation studies and eligible outcomes that meet minimum criteria. Eligible outcomes presently include mental health, substance abuse, and wellness. Next, an expert review performed by two certified reviewers measures the rigor of the study and the impact on outcomes. The outcomes are reviewed using an NREPP outcome rating instrument and are judged on the basis of four dimensions: rigor, effect size, program fidelity, and conceptual framework (see Box B-2).

After all eligible measures or effects have been rated, the scores for each outcome are calculated, an evidence class for each measure is determined, and an outcome rating is determined (see Figure B-1).

Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×

First, the evidence class is determined based on the evidence score (a combination of the rigor and fidelity dimensions) and the effect class (based on the confidence interval of the effect size). The evidence classes are as follows:

  • Class A: highest-quality evidence with confidence interval completely within the favorable range
  • Class B: sufficient evidence with confidence interval completely within the favorable range
  • Class C: sufficient or highest-quality evidence with confidence interval spanning both the favorable and trivial ranges
  • Class D: sufficient or highest-quality evidence with confidence interval completely within the trivial range
  • Class E: sufficient or highest-quality evidence with confidence interval spanning both the harmful and trivial ranges
  • Class F: sufficient or highest-quality evidence with confidence interval completely within the harmful range
  • Class G: limitations in the study design preclude reporting further on the outcome
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Image
FIGURE B-1 Diagram of how the final outcome rating is determined for the National Registry of Evidence-based Programs and Practices.
SOURCE: SAMHSA’s National Registry of Evidence-based Programs and Practices (2016). Available: http://nrepp.samhsa.gov/04a_review_process.aspx [August 2016].

Next, the outcome rating is determined on the basis of the outcome scores and the conceptual framework. The outcome scores are calculated from the evidence classes of each component measure, and the rating of the conceptual framework is a determination of whether a program has clear goals, activities, and a theory of change. The possible outcome ratings are

  • Effective: strong evidence of a favorable effect
  • Promising: sufficient evidence of a favorable effect
  • Ineffective: sufficient evidence of a negligible effect OR sufficient evidence of a possibly harmful effect
  • Inconclusive: study design limitations or a lack of effect size information precludes reporting further on the effect

BLUEPRINTS FOR HEALTHY YOUTH DEVELOPMENT2

Blueprints for Healthy Youth Development is a registry of evidence-based programs for positive youth development created by the Center for the Study and Prevention of Violence at the University of Colorado Boulder. The registry is intended to be a source of information for decision makers investing in programs with a goal of promoting positive youth development. Positive youth development includes academic performance and success, health and well-being, and positive relationships. Programs can be family-, school-, or community-based and can have a variety of different goals (e.g., violence prevention, reduced school delinquency, reduced sub-

___________________

2 This section was compiled from information on the Blueprints Website. Available: http://www.blueprintsprograms.com/ [October 2016].

Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×

stance use, improved mental and physical health, improved self-regulation, higher educational achievement). The registry is intended to provide users with information on evidence-based programs with high standards. Thus far, more than 1,300 programs have been reviewed, with less than 5 percent having been found to meet the review criteria. For programs that have met the criteria, users can find a program description as well as information on outcomes, target population, risk and protective factors, training and tech-

Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×

nical assistance, evaluation methodology, program costs, funding strategies, benefits and costs, and references.

Program reviews are conducted by staff and then an advisory board of seven youth development experts to determine whether a program meets the criteria of (1) evaluation quality, (2) intervention impact, (3) intervention specificity, and (4) dissemination readiness. (See Box B-3 for more detail.) Programs meeting these criteria have demonstrated at least some effective-

Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×

ness at changing targeted behavior and developmental outcomes. These programs are then added to the registry with a rating of Model or Promising.

Model Programs

Model programs meet higher standards than those met by Promising programs and offer greater confidence in the program’s ability to modify behavior and developmental outcomes. These programs are suggested for use in large-scale implementation, such as at the national or state level. Model programs meet the four criteria above and two additional requirements:

  1. Evaluation Quality: Evidence is required from two high-quality randomized controlled trials (RCTs) or one RCT and one quasi-experimental evaluation.
  2. Intervention Impact: A minimum of one long-term follow-up (at least 12 months after the intervention has ended) on at least one outcome measure must indicate that results have been sustained following the intervention. Data on sustainability are required for both program and control groups. For interventions designed to extend over many years, evidence is required that effects have been sustained after several years of participation in the program even though participation is continuing and will be accepted as evidence of sustainability.

Programs rated as Model that also have been independently replicated in a high-quality manner are designated Model Plus.

Promising Programs

Promising programs meet the four criteria elaborated in Box B-3 and are recommended for local community and system adoption. Promising programs do not have to meet the additional evaluation quality and intervention impact requirements for Model programs listed above.

THE CALIFORNIA EVIDENCE-BASED CLEARINGHOUSE FOR CHILD WELFARE3

The California Evidence-Based Clearinghouse for Child Welfare (CEBC), funded by the California Department of Social Services’ Office of Child Abuse Prevention, is a database of child welfare-related programs in-

___________________

3 This section was compiled from information on CEBC’s Website. Available: http://www.cebc4cw.org/ [October 2016].

Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×

tended to provide information and resources for child welfare professionals. The mission of the clearinghouse is to advance the effective implementation of evidence-based practices for children and families involved in the child welfare system. CEBC provides descriptions of and information on research evidence for specific programs, as well as implementation guidance. All programs are categorized by topic area. Assignment to topic areas is based on clear definitions and requirements that programs must meet. Requirements also are specified for which program outcomes the research evidence must demonstrate for programs to be rated within each topic area.

The CEBC review process starts with the selection of topic areas, which is performed annually by an advisory committee. A list of possible programs to be included in each topic area is then generated based on information from topic experts and literature searches conducted by staff. Each of these programs is then contacted with a list of screening questions. If the program passes the screening, it receives a questionnaire to complete, and a literature search is conducted for any relevant published peer-reviewed research literature. A program outline is then created, and study outcomes are summarized from the research literature. Outcomes of focus relate to child welfare and include safety, permanency, and child/family well-being. Program outlines that meet one of the five categories of the CEBC Scientific Rating Scale (see Figure B-2) are then sent to raters (usually the topic expert and two staff). The scale’s purpose is to assess each program based on the available research evidence.

Scientific Rating Scale

The Scientific Rating Scale is a 1 to 5 rating of the strength of the research evidence supporting a program. A scientific rating of 1 represents a program with the strongest research evidence, while a 5 represents a con-

Image
FIGURE B-2 Scientific Rating Scale for the California Evidence-Based Clearinghouse for Child Welfare.
SOURCE: California Evidence-Based Clearing House for Child Welfare (2016).
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×

cerning practice that appears to pose substantial risk to children and families. A rating of 2 indicates the program is supported by research evidence, 3 indicates promising research evidence, and 4 indicates that the evidence fails to demonstrate effect. Specific criteria for each rating are presented in Box B-4. Some programs currently lack strong enough research evidence to be rated on the Scientific Rating Scale and are classified as NR (Not Able to Be Rated). A rating of NR does not mean a program is not effective.

Program ratings are evaluated on an ongoing basis as new research is published, and programs are rerated if necessary. Intermittent re-reviews are conducted to look for new published, peer-reviewed research on programs already rated. Program representatives also can submit new published, peer-reviewed studies to initiate the re-review process at any time.

Child Welfare System Relevance Levels

In addition to its assigned rating, each program included in the database is reviewed to determine how child outcomes are addressed in the program’s research evidence. The topic expert and staff review the target population and goals of the program to determine a Child Welfare System Relevance Level of high, medium, or low. Programs rated high are designed or commonly used to meet the needs of children and families receiving child welfare services. Those rated medium are designed or commonly used to serve children and families similar to child welfare populations and likely include current and former child welfare participants. Finally, programs rated low serve children and families with little or no apparent similarity to child welfare participants.

Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 401
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 402
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 403
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 404
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 405
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 406
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 407
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 408
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 409
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 410
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 411
Suggested Citation:"Appendix B: Clearinghouses Used to Identify Interventions with Evidence of Effectiveness." National Academies of Sciences, Engineering, and Medicine. 2016. Parenting Matters: Supporting Parents of Children Ages 0-8. Washington, DC: The National Academies Press. doi: 10.17226/21868.
×
Page 412
Next: Appendix C: Table of Parenting Interventions »
Parenting Matters: Supporting Parents of Children Ages 0-8 Get This Book
×
Buy Paperback | $74.00 Buy Ebook | $59.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Decades of research have demonstrated that the parent-child dyad and the environment of the family—which includes all primary caregivers—are at the foundation of children’s well- being and healthy development. From birth, children are learning and rely on parents and the other caregivers in their lives to protect and care for them. The impact of parents may never be greater than during the earliest years of life, when a child’s brain is rapidly developing and when nearly all of her or his experiences are created and shaped by parents and the family environment. Parents help children build and refine their knowledge and skills, charting a trajectory for their health and well-being during childhood and beyond. The experience of parenting also impacts parents themselves. For instance, parenting can enrich and give focus to parents’ lives; generate stress or calm; and create any number of emotions, including feelings of happiness, sadness, fulfillment, and anger.

Parenting of young children today takes place in the context of significant ongoing developments. These include: a rapidly growing body of science on early childhood, increases in funding for programs and services for families, changing demographics of the U.S. population, and greater diversity of family structure. Additionally, parenting is increasingly being shaped by technology and increased access to information about parenting.

Parenting Matters identifies parenting knowledge, attitudes, and practices associated with positive developmental outcomes in children ages 0-8; universal/preventive and targeted strategies used in a variety of settings that have been effective with parents of young children and that support the identified knowledge, attitudes, and practices; and barriers to and facilitators for parents’ use of practices that lead to healthy child outcomes as well as their participation in effective programs and services. This report makes recommendations directed at an array of stakeholders, for promoting the wide-scale adoption of effective programs and services for parents and on areas that warrant further research to inform policy and practice. It is meant to serve as a roadmap for the future of parenting policy, research, and practice in the United States.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!