National Academies Press: OpenBook

Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop (2017)

Chapter: 8 Key Themes and Next Possible Steps

« Previous: 7 Cheerleaders, Naysayers, Large and Small Evaluators: Fostering Support and Inclusion
Suggested Citation:"8 Key Themes and Next Possible Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/24831.
×

8

Key Themes and Next Possible Steps

Russ Whitehurst (chair, steering committee) drew the participants’ attention back to the scope of the workshop: evaluation of federal programs intended to affect human behavior. He added that U.S. taxpayers make a decision to fund those programs with the goal of improving opportunities and reducing identified problems and that failure to use their money in a way that can contribute to those goals is a disservice to them. He referenced the opinion of Jon Baron (Arnold Foundation) about evaluations revealing the low success rates of certain programs, but he said that incremental progress is still progress.

Whitehurst said that while the evaluation principles that are currently in place are very sound, they need legislation to help give them permanence and stability. He sees the legislation taking one of two forms:

  • drafting legislation on an agency-by-agency basis that supports the creation of independent research and evaluation agencies and affords them protections and statutory guidelines (such as that for the Institute of Education Sciences), or
  • aligning with the Paperwork Reduction Act, creating separate legislation and giving the U.S. Office of Management and Budget (OMB) some general authority over this function, similar to that which is in place for the statistical agencies.

Whitehurst acknowledged that funding had been addressed several times throughout the workshop, telling the group that, across the board, budgets for evaluation are comparatively smaller than the money allotted to

Suggested Citation:"8 Key Themes and Next Possible Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/24831.
×

other functions in the same programs. He thinks that budgeting for evaluation would need to be included in any legislation.

Whitehurst once again mentioned OMB as a key player in the practice of implementing evaluation principles. He added that the leading agencies have a role to play as well, but he suggested that the inclusion of Congress in the specifics of implementation could ultimately do more harm than good. He reiterated the importance of peer review as a system that can hold the producers of the work responsible for its quality, and he reminded participants of OMB’s prior practice of putting a process in place to rate the quality of evaluation efforts as another accountability measure. Baron added that the proper use of peer review and techniques like specifying confirmatory versus exploratory hypotheses could also be leveraged to influence similar processes in scholarly journals, whose peer review protocol is sometimes not as rigorous.

Judith Gueron (member, steering committee) reminded Whitehurst and the participants about the earlier discussion on the tension that arises between focusing on rigor and making evaluations useful. Whitehurst acknowledged that while the issue is a real one, it is subjective, and a high program failure rate can adversely affect stakeholders’ opinion of success. He reinforced Baron’s point about the need to evaluate program components as opposed to entire programs—especially for larger, more established operations—in an effort to mitigate that tension.

Miron Straf (Virginia Polytechnic Institute and State University) countered Whitehurst and Baron’s point, saying that the message he would like to see projected to the evaluation community would be an encouragement to move away from the myopic approach of focusing on an effect size of a single intervention and to look at the social programs as part of a complex system. He suggested looking at the major drivers and also considering the use of passive or “big data” as reinforcement. He also compared social program analysis to advances in medical research and agreed with Whitehurst that it might be wise to consider how to integrate an evaluation style that focuses on continuous improvement.

Naomi Goldstein (Administration for Children and Families) brought together the discussants’ summary points by saying that while peer review can be valuable, she believes it is a practice, rather than a principle, and falls under the larger umbrella of quality control—a very important principle to be considered. Whitehurst thanked participants for sharing their thoughts, and supporting the committee’s view that the field of federal program evaluation is an important enterprise that will need support to continue to grow, improve, and be recognized for the important contribution it makes to the U.S. population and the government.

Suggested Citation:"8 Key Themes and Next Possible Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/24831.
×
Page 41
Suggested Citation:"8 Key Themes and Next Possible Steps." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/24831.
×
Page 42
Next: References »
Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop Get This Book
×
Buy Paperback | $50.00 Buy Ebook | $40.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In October 2016, the National Academies of Sciences, Engineering, and Medicine convened a 1-day public workshop on Principles and Practices for Federal Program Evaluation. The workshop was organized to consider ways to bolster the integrity and protect the objectivity of the evaluation function in federal agencies—a process that is essential for evidence-based policy making. This publication summarizes the presentations and discussions from the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!