A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix H Summary of Previous Title X Evaluations and Reviews Several evaluations of specific aspects of Title X have been conducted. These include both government reviews and evaluations conducted by nongovernmental organizations at the request of the Office of Population Affairs (OPA) or the Office of Family Planning (OFP). The purpose, scope, primary findings, and major recommendations from these assessments are summarized below, in chronological order. General Accounting Office In 1981, the U.S. General Accounting Office (GAO) provided testi- mony before Congress summarizing the major findings from a series of GAO reviews of family planning activities under Title X. Between 1970 and 1981, GAO issued eight reports to Congress and one to the District of Columbia’s Department of Human Services. The reports did not provide a comprehensive evaluation of the effectiveness of Title X, but concluded that the program had not served many in the target population effectively. This conclusion was based on a series of interviews conducted with welfare recipients, many of whom desired to use family planning services but were unaware of the existence of local clinics, as well as interviews with a sample of clinic clients that suggested high client turnover (GAO, 1981). Several areas for improvements in program management were identified. They included streamlining services for clients using oral contraceptives and col- lecting fees more aggressively; consolidating and improving coordination of the four different programs administered by the Department of Health and Human Services (HHS) that fund family planning services; improving data 295 

296 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM collection and monitoring of grantees; improving procedures for grants and contracts awards; and ensuring the appropriate use of funds authorized for program implementation research. GAO indicated that these improvements could reduce costs or enhance the effectiveness and efficiency of Title X. Assistant Secretary for Administration and Management In 2004, the Office of Grants Management and Policy, under the direction of the Assistant Secretary for Administration and Management, conducted a review of the family planning grant program pre-award activi- ties, which include selection of instruments, development of the program announcement, solicitation of applications, review of applications, and monitoring of post-award activities (Office of Grants Management and Policy, Assistant Secretary for Administration and Management, 2004). The evaluation focused on how the program complied with HHS regulations and policies, whether the pre-award process effectively maximized competi- tion, and whether post-award administration was adequate. Results of the evaluation identified areas for improved compliance, such as utilizing the grants.gov system more fully, improving filing of communi- cation with grantees, and standardizing the maintenance and organization of grant files. In terms of competition, it was recommended that term limits for grant application reviewers be established, that supplemental requests across regions be reviewed to ensure objectivity, and that OPA review practices to ensure that competition is maximized. Several recommenda- tions were also made in the area of post-award administration, including forwarding copies of site visit reports, as well as the protocols used and the documentation provided, to the grants office for inclusion in the official grant file; developing a post-award monitoring instrument to record contact with grantees; verifying and documenting that appropriate financial moni- toring is conducted; and developing a mechanism to verify that organiza- tions eligible for new or continuation grants are up to date with applicable audit requirements. Some additional recommendations outside of the main focus areas were also made, such as establishing a specific amount of time a reviewer could serve under the same program. The protocol used to conduct program reviews of grantees was identified as a best practice. Research Triangle Institute In 2005, the Research Triangle Institute (RTI) provided a review of evaluations relevant to the Title X program to assess evaluation activi- ties, determine evaluation needs and topics, and identify approaches to guide future evaluations (RTI International, 2005a,b). After identifying 

APPENDIX H 297 490 potentially relevant published and unpublished studies and excluding studies such as those that were not evaluations or were not specific to the Title X program, the RTI review included 29 published and 39 unpublished studies. The majority (69 percent) of the reviewed studies focused on the financing, costs, organizational structure, and operations of the Title X program. The remaining 31 percent of the studies focused on utilization and demand characteristics of reproductive health care. The studies were further grouped under various topics within these major categories, as well as by evaluation type, population studied, and geographic location of the evaluation. The review also identified differences found between published and unpublished studies; for example, published studies were more likely to be output, process, and outcome evaluations, while the majority of unpub- lished studies were process evaluations and needs assessments. RTI noted a lack of studies in several areas that are highly relevant to the Title X program. Few of the studies reviewed examined demand for and utilization of family planning services; organizational or operational issues related to service provision; or supply and demand issues for racial and ethnic minorities, migrants, people with limited English proficiency or low literacy, the uninsured and underinsured, young men, or other high-risk populations (for example, homeless individuals). No studies examined the effects of family and school ties on adolescents’ sexual behavior, partner- ships with community- or faith-based providers that work with vulnerable populations, education in extramarital abstinence, and HIV/AIDS risk pre- vention counseling—topics from the 2005 list of program priorities. A systematic examination of the quality of each of the studies included in the review was undertaken, and the gaps and needs thus identified were discussed. Recommendations were made for OPA to encourage researchers to conduct studies in areas where research is currently lacking, as well as to establish guidelines to improve the quality of unpublished studies. Other recommendations included taking steps to make it easier to identify relevant published Title X studies (for example, by encouraging the use of Title X identifiers in data collection efforts and creating a dedicated medical subject heading term for Title X), as well as implementing a system to catalogue OPA-funded evaluations. Additional suggestions were made for strength- ening the quality and impact of Title X program evaluations, including forming an external Title X evaluation work group to, among other things, establish priorities for national and cross-regional evaluations; encourag- ing the use of logic models as part of the design and implementation of demonstration or other pilot project initiatives; encouraging support for and involvement in the collection and analysis of Title X–relevant national- level data; and encouraging Regional Training Centers to collaborate on the development of a methodology for evaluating training needs for cross- regional OPA initiatives. 

298 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM Office of the Inspector General After concern was raised that organizations receiving Title X funds might not be fully complying with state reporting laws regarding sexual abuse, the Office of the Inspector General (OIG) reviewed the process used by OPA to inform and monitor grantees with respect to these requirements. The OIG assessment included a review of documents issued by OPA and its grantees, including Program Guidelines, assessment tools, and training materials; a written survey of the 10 Regional Offices; and structured in- person and telephone interviews with representatives from the OPA Central Office, regional program consultants, and representatives from each of the Regional Training Centers. OIG concluded that “OPA has informed and periodically reminds Title X grantees of their responsibilities regarding State child-abuse and sexual-abuse reporting requirements in its reviews and site visits of grantees” (OIG, 2005, p. 1). Health Systems Research, INC. In 2005, Health Systems Research, Inc. assessed clinical specialty train- ing for Title X clinical service providers (Health Systems Research, Inc., 2005). OPA requested the assessment to help improve clinicians’ capacity to provide effective and high-quality family planning services. The assessment consisted of a literature review, a workgroup meeting, and interviews with the two Title X clinical specialty training programs (each of which provides training for half of the country’s regions). The staffing challenges identified included an aging provider population, difficulty in recruiting and retaining providers, and inadequate funding. Training challenges identified included “a lack of hands-on training opportunities, difficulties in finding preceptors, complex client needs and changing demographics, difficulty recognizing and assessing training needs, and funding, geographic, and timing issues.” (Health Systems Research, Inc., 2005, p. 1). Suggested strategies for addressing training challenges included (1) using the existing infrastructure to expand training opportunities (for example, by standardizing training technologies and sharing information about model programs); (2) utilizing alternative resources (for example, by collaborating with other federal training programs that provide training in similar areas); (3) building a new infrastructure to provide more opportunities for training (for example, by creating a pool of clinical trainers that could travel to sites to provide training, or by establishing model clinics in each region); and (4) addressing recruitment and retention issues (for example, by providing salary adjustments for clinicians with advanced training). Recommenda- tions were also made regarding the key components of an ideal program for Title X clinical specialty training, including creating roles for “key players” 

APPENDIX H 299 (such as a pool of expert clinical trainers who would travel among clinics to provide training or the staff of a national coordinating center); develop- ing training content, including a national standardized curriculum cover- ing the Title X core competencies and service requirements; enhancing the delivery of clinical training to increase accessibility and maximize the use of resources; and creating a national coordinating body to assess training needs and develop a standardized evaluation that would be conducted on an ongoing basis. Program Assessment Rating Tool As described in Chapter 1, the Office of Management and Budget (OMB) performed an evaluation of Title X under the Program Assessment and Rating Tool (PART) process (OMB, 2009). The program received a rating of Moderately Effective. The program subsequently developed three long-term measures that are reported on annually: (1) increasing the number of unintended pregnancies averted by providing Title X ­ family planning services, with priority for services to low-income individuals; (2) reducing infertility among women attending family planning clinics by identifying chlamydia infection through screening of females aged 15–24; and (3) reducing invasive cervical cancer among women attending family planning clinics by providing Pap tests according to nationally recognized standards of care. In addition to requesting the present evaluation, OFP has developed performance goals for key program activities in response to the PART process. The program has developed, implemented, and established targets for all annual and long-term performance goals, including its efficiency measure, which is to maintain the actual cost per client below the medical care inflation rate. REFERENCES GAO (U.S. General Accounting Office). 1981. Results of GAO reviews of family planning activities under Title X of the Public Health Service Act. Washington, DC: GAO. Health Systems Research, Inc. 2005. Assessment of clinical specialty training for Title X ­clinical service providers: Executive summary. Washington, DC: Health Systems ­Research, Inc. Office of Grants Management and Policy, Assistant Secretary for Administration and Man- agement. 2004. Review of grants management practices, policies, and procedures in the Office of the Assistant Secretary for Public Health and Science (OPHS), Office of Popula- tion Affairs (OPA), Family Planning Services Program, CFDA 93.217. Washington, DC: U.S. Department of Health and Human Services. OIG (Office of the Inspector General). 2005. Federal efforts to address applicable child abuse and sexual abuse reporting requirements for Title X grantees. http://www.oig.hhs.gov/ oei/reports/oei-02-03-00530.pdf (accessed April 2, 2009). 

300 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM OMB (Office of Management and Budget). 2009. Detailed information on the family plan- ning assessment. http://www.whitehouse.gov/omb/expectmore/detail/10003513.2005. html (accessed March 6, 2009). RTI International (Research Triangle Institute International). 2005a. Review and ­ synthesis of evaluations relevant to the Title X program. Research Triangle Park, NC: RTI International. RTI International. 2005b. Synthesis of data relevant to the Title X program. Research Triangle Park, NC: RTI International.