Review of Four CARA Programs and Preparing for Future Evaluations (2023) / Chapter Skim
Currently Skimming:
8 Preparing for Future Evaluations
8 Preparing for Future Evaluations
Pages 125-136
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 125... ...
8 Preparing for Future Evaluations INTRODUCTION Over the past 5 years, the committee has reviewed the four Comprehensive Addiction and Recovery Act (CARA) programs and the information that the Substance Abuse and Mental Health Services Administration (SAMHSA)
|
From page 126... ...
The committee chose to propose recommendations that could be applied to the future program development and evaluation efforts of federal programs broadly, rather than just to those conducted by SAMHSA. So, while the earlier chapters of this report focus on the four CARA programs and the evaluation effort conducted by the National Academies in partnership with SAMHSA, this chapter looks beyond these specific entities or this particular evaluation effort.
|
From page 127... ...
Figure 8-1 presents a visual depiction of the actors involved in evaluations of federal programs, their relationships, and which part of the process the committee's recommendations would impact. Recommendation 1: The committee recommends that Congress, when mandating evaluations, confer with the implementing agency and evaluation experts to align expectations with feasibility and resource considerations.
|
From page 128... ...
permitting causal inferences and rely on detailed quantitative measurements that have been systematically collected over time. Evaluation Misalignment in the CARA Evaluation In the case of the CARA evaluation, the National Academies (the evaluator in this case)
|
From page 129... ...
This limited options for the National Academies to design an appropriate evaluation. Given the early development state of the projects, conducting a formative evaluation of how each of the four CARA programs were implemented is likely to have yielded more helpful information for Congress; such an evaluation would have been better aligned with the program goals and characteristics, as well as with the information available to the committee.
|
From page 130... ...
To facilitate alignment, Congress should provide funding as early as possible to allow for ample time to coordinate with the implementing agency on how to best scope and plan the required assessment activities before grantees begin their implementation process.4 If brought in early, the parties can work together to help tailor data collection and evaluation methodology (e.g., research design, sampling, types of data required and most appropriate means of collecting those data, and analytic approach)
|
From page 131... ...
Additionally, the pre-existing mission and culture of the implementing agency can impact its ability to support an evaluation.5 In particular, some federal agencies may be more practice-oriented than research-focused in their mandates. This may mean that an agency could have less staff expertise relevant to evaluations, data collection systems that are less suited to evaluations, or less interest in participation.
|
From page 132... ...
. The legislation defined evaluation as "an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency." In response to this guidance, in 2022, SAMHSA's Center for Behavioral Health Statistics and Quality (CBHSQ)
|
From page 133... ...
For example, applying for and receiving OMB approval for new instruments or larger samples may take a year or more, and is thus infeasible with quick deadlines for funding and startup of grant programs. Data Collection Systems in the CARA Evaluation For example, in the CARA experience, SAMHSA's off-the-shelf reporting tools such as the Government Performance and Results Act (GPRA)
|
From page 134... ...
Data Sharing Agreements in the CARA Evaluation For example, in the CARA experience, many SAMHSA grantees were initially unaware that they would be participating in a cross-site evaluation of the CARA programs or unaware of the specific data collection requirements. This was likely because, with the exception of the PPW-PLT program, SAMHSA's Funding Opportunity Announcements for the CARA programs did not mention this possibility.
|
From page 135... ...
Technical Assistance to CARA Grantees In the case of the CARA evaluation, grantees described numerous challenges to participating in data collection and utilizing the collection tools mandated by SAMHSA.10 They highlighted the need for additional support and communication with SAMHSA grant program officers and suggested that TA and peer-to-peer sharing across grantees might have benefited their work. Strategies shared in such a manner could even be shared more broadly to groups providing similar services.
|
From page 136... ...
the evaluator were involved and funded prior to implementation, (3) the GPRA and DSP-MRT data collection tools were edited or that new ones fitted specifically to the program goals were created and utilized, (4)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.