National Academies Press: OpenBook

A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results (2009)

Chapter: 5 Collection of Data to Measure Program Outcomes

« Previous: 4 Program Management and Administration
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 145
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 146
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 147
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 148
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 149
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 150
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 151
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 152
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 153
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 154
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 155
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 156
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 157
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 158
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 159
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 160
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 161
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 162
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 163
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 164
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 165
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 166
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 167
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 168
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 169
Suggested Citation:"5 Collection of Data to Measure Program Outcomes." Institute of Medicine. 2009. A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results. Washington, DC: The National Academies Press. doi: 10.17226/12585.
×
Page 170

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 Collection of Data to Measure Program Outcomes Under the auspices of the Office of Population Affairs (OPA), the Office of Family Planning (OFP) uses a variety of measures to provide program- matic information that is both timely and responsive to a wide range of stakeholders concerned with the program’s scope, quality, and reach. This chapter addresses the ability of these measures to assess the program’s goals, processes, and outcomes. It begins by summarizing the Office of Management and Budget’s (OMB) Performance Assessment Rating Tool (PART) evaluation process for Title X. It then presents a series of evaluation questions and an evaluation framework used by the committee to assess the sources and types of data collected by OFP. Next is a review of current sources of data for program assessment. The committee’s evaluation frame- work is then used to assess OFP’s measures and goals. This is followed by an assessment of the data collection infrastructure for the Title X program. The chapter ends with conclusions and recommendations for improving the Title X evaluation system as a whole, drawing on recommendations from earlier groups as well as this committee. PART PROCESS FOR EVALUATING TITLE X There have been several evaluations of specific aspects of the Title X program, including both government reviews and evaluations conducted by nongovernmental organizations at the request of OPA (see Appendix H for a summary of findings and recommendations from these evaluations). The present study was prompted by OMB’s evaluation of Title X under the PART process (OMB, 2005) (the PART is presented in Appendix E). The 145

146 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM program received a rating of Moderately Effective (rating categories include Results Not Demonstrated, Ineffective, Adequate, Moderately Effective, and Effective). The highest scores were achieved in the areas of Program Purpose and Design and Program Management, with lower scores in the areas of Strategic Planning and Program Results/Accountability. The com- mittee concurs that strategic planning is an area for improvement (see the discussion and recommendations in Chapter 3). As required by the OMB evaluation process, OFP defined three long- term annual performance measures, described in Chapter 3: (1) increasing the number of unintended pregnancies averted by providing Title X ­family planning services, with priority for services to low-income individuals; (2) reducing infertility among women attending family planning clinics by identifying chlamydia infection through screening of females aged 15–24; and (3) reducing invasive cervical cancer among women attending family planning clinics by providing Pap tests according to nationally recognized standards of care. In addition to defining these three long-term measures, OFP had earlier developed, implemented, and established targets for all annual and long-term performance goals, including its efficiency measure, which is to maintain the actual cost per client below the medical care inflation rate. These measures, along with quality-of-care indicators, are assessed in this chapter. committee’s Evaluation Questions The committee examined two key issues. The first is what data collec- tion infrastructure is currently in place for the Title X program. The second is how the data collection effort and its infrastructure can be improved. The following questions, which emerged from the committee’s charge, the PART process, and previous program evaluations, framed the committee’s assessment of the measures around these two issues: • What types of data are being used by OFP for monitoring and assessing the Title X program? • Does the existing portfolio of data collection approaches ade- quately capture Title X’s activities? • What modifications should be made to the data collection system and planning? committee’s Evaluation Framework To answer the above questions, the committee developed an evaluation framework that focuses on Title X’s primary mission—to provide indi­viduals the information and means to exercise personal choice in determining the

COLLECTION OF DATA 147 CLIENT NEED STRUCTURE PROCESS OUTCOMES Predisposing Enabling Client Factors Factors Health Behaviors Services Received Outcomes Client System Process of Care: Service Use Modeled Characteristics Characteristics Client–Provider Performance Improved Encounter Indicators for Title Clinical X Priority Areas Outcomes FIGURE 5-1  Conceptual framework for Title X evaluation. number and spacing of their children (see Figure 5-1). The committee’s conceptual framework structures this final chapter. Discussion of how the framework can be used to evaluate the program is detailed below. The committee believes such a framework can help establish a more integrated and comprehensive evaluation approach for Title X by linking the program’s assessment to its stated goals and priorities. Consistent with Title X’s commitment to ensuring adequate quality of care, the commit- tee’s evaluation framework draws on well-tested models for evaluation of the quality and utilization of health services—Donabedian’s Quality Model (Donabedian, 1968, 1980) and Andersen’s Health Care Utiliza- tion Model (Andersen and Davidson, 2007). The use of this framework offers an opportunity to place the discussion of quality of care within the broader context of national and various state health policies and Title X financing as enabling factors for the Title X program, patient care, and education and outreach. The goal of the framework is to help OFP maintain a cohesive and pro- active evaluation program focused on quality improvement and the ability to document outcomes. The framework provides a logic model emphasizing measures linking (1) the clients to be served; (2) the resources, facilities, and personnel required to serve them (structure); (3) the services actually provided (process); and (4) the results of those services (outcomes). Having such a model allows planners to determine returns on various investments or interventions. The committee acknowledges the challenging task of identifying, at a minimum, meaningful and prioritized short- and longer- term outcomes that are valid indicators and can readily be measured in a clinic setting. As with any data collection, it is necessary to recognize the resource and staff costs associated with complete and timely data collec- tion at the local level. Moreover, longer-term outcomes may be difficult to capture adequately without significant investments in client tracking and data collection.

148 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM The committee’s framework encompasses: • Client need, or predisposing factors (column 1)—These factors include characteristics of clients that motivate them to use services and that Title X should consider in defining its profile of services. • Structure, or enabling factors (column 2)—These system character- istics enable patient access to health care services and facilitate the delivery of quality services important to client outcomes. • Process, or health behaviors/services received (column 3)—These process factors pertain to the interaction between clients and pro- viders, as well as the services received and how they align with established service use performance indicators for the Title X priority areas. • Outcomes (column 4)—These factors refer to improvements in clinical outcomes that occur as a result of clients’ interactions with the system of services and outreach. Depending on the evaluation question, measures relevant to the Central Office, Regional Offices, grantees, delegates, clients, populations in need, and barriers to care may be included in the data collection. For example, Title X’s Central Office may be interested in answering questions pertaining to overall trends in the numbers and profiles of clients served across the country. At the delegate level, there may be interest in questions that pertain to comparisons among similar types of delegates or to contrasts among dif- ferent types of delegates in various geographic areas. Current Sources of Data for Program Assessment OFP currently uses a variety of data sources for monitoring and assess- ing the Title X program (see Chapter 4). The Family Planning Annual Report (FPAR) is the primary source of annual, uniform reporting by all Title X grantees (see Appendix G for the FPAR data elements). Informa- tion from the FPAR is important to OPA for several reasons. First, FPAR data are used to monitor compliance with statutory requirements, regula- tions, and operational guidance set forth in the Program Guidelines, which include giving priority in the provision of services to low-income persons (Section 1006(c) of Title X of the Public Health Service Act, 42 USC § 300) and ensuring that Title X grantees and their subcontractors provide a broad range of family planning methods and services (Section 1001(a) of Title X of the Public Health Service Act, 42 USC § 300).  Second, OPA uses FPAR data to comply with accountability and federal performance requirements for recipients of Title X family planning funds as required by the 1993 Government Performance and Results Act (GPRA).

COLLECTION OF DATA 149 Current GPRA performance goals for the Title X program include as pri- orities the provision of family planning services to low-income individuals, access to and utilization of cervical and breast cancer screening, and access to on-site HIV testing at Title X–funded clinics. Finally, the program relies on FPAR data to monitor performance, respond to inquiries about the program from policy makers and Congress, and support program planning. The FPAR allows OPA to assemble compa- rable and relevant program data to answer questions about the character- istics of the populations served by Title X clinics, utilization of the services offered, the composition of revenues, and program impact. FPAR data are the basis for objective grant reviews, program evaluation, and assessment of program technical needs (RTI International, 2008). Each year, FPAR data are analyzed to produce the Family Planning Annual Report National Summary. This report provides analyses of the FPAR measures by demographics; social and economic profile; method use, by gender; and cervical, breast, and sexually transmitted disease (STD) screening. It also includes information on staffing, family planning encounters, and revenue. The Research Triangle Institute (RTI) conducted analyses of the FPAR measures to produce the 2006 National Summary (RTI International, 2008). Although the committee anticipated that its site visits and the testimony of grantees and delegates would elicit reports of the burden imposed on clinic staff by meeting the FPAR requirements, a more complex picture emerged. Many interviewees reported that complying with FPAR require- ments was not burdensome, especially after they switched to electronic health records. Others, however, indicated it was difficult to meet these requirements; indeed, the committee was told that the FPAR requirements deterred some clinics from seeking Title X funding. Finding 5-1. Sites vary in their capacity to meet the FPAR data collection requirements. Another source of information that OFP uses to evaluate the Title X program is the Comprehensive Program Review (CPR), conducted every 3 years by OFP’s Regional Offices. This review is intended to ensure that grantees are complying with Title X policy and program requirements, as well as OMB regulations. The Program Review Tool (PRT) used in these CPRs contains key questions on administration, financial management, clinical services, and outreach/information (see Appendix J, Table J-2, for data collected by the PRT). Grantee-monitoring site visits are then conducted by the Regional Offices to check on compliance and to follow up on issues identified previously in the CPR, grant application, and/or needs assessment. The CPR activities suggest that there is considerable

150 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM system capacity for and commitment to evaluation. The question the com- mittee raises is whether this capacity could be used more strategically to strengthen the evaluation and performance of the program. OFP might further structure this evaluation visit, with input from evaluation experts, RPCs, grantees, and delegates. This group would define core goals and provide tools/approaches that would be used during site visits. Finding 5-2. The Comprehensive Program Review represents an underutilized opportunity for systematic assessment across the Title X program. Another source of data is the National Survey of Family Growth (NSFG), which Title X partially supports on an annual basis. This large- scale survey asks a nationally representative group of women a series of questions related to reproductive behavior and health. Key variables most relevant to Title X include (1) whether family planning medical or con- traceptive services were received, (2) contraceptive(s) received, (3) type of family planning medical services received, (4) type of setting where services were received (e.g., hospital, health maintenance organization facility, com- munity clinic), (5) location (i.e., name, address) of the facility, (6) whether the location was clients’ regular source of care, (7) how the service was paid for (e.g., copayment, Medicaid, insurance, free), and (8) whether services were paid for by clients on a sliding-scale basis. Data on the Title X program that may be useful for evaluation purposes are also available from several other sources. These include, for example, contracts that are funded by HHS 1 percent evaluation funds, the Service Delivery Improvement grant program, family planning research cooperative agreements, and demonstration projects aimed at addressing key program initiatives (see Appendix I). In 2008, OPA spent approximately $9 million on research and evalua- tion activities. The goal is to identify emerging needs in family planning, as well as to develop and evaluate service innovations that respond to identi- fied needs, with the implied intent of disseminating findings to grantees. Findings from OPA-funded research are also important for monitoring service delivery needs and the provision of care by clinics funded under the program. Finding 5-3. The National Survey of Family Growth and other research grant programs funded by OPA hold potential for more targeted research to enhance evaluation agendas for Title X. Moni- toring to ensure wider dissemination of OPA-supported research findings to the provider community and use of these findings is key to ongoing efforts to improve service delivery.

COLLECTION OF DATA 151 Applying the committee’s Framework to Assess OFP’s Measures and Goals The committee used the evaluation framework presented in Figure 5-1 to evaluate whether the measures currently used by OFP adequately assess the program’s progress and effectiveness in meeting its goals. The primary data sources reviewed were the FPAR and to a lesser degree the CPR/PRT. Table 5-1 shows the data currently being collected that are relevant to each of the framework’s domain columns. Also listed are indicators and approaches the committee recommends be added (presented in italics). The evaluation framework was also used to guide the committee’s recommendations for improving the evaluation of Title X patient care and counseling, as well as to help in identifying emerging needs for outreach and education. OFP’s collection of data elements is much stronger for some columns of the framework than others. On the one hand, OFP already collects data on key characteristics of clients served, several critical system characteristics, and services performed. On the other hand, OFP does not systematically collect data on key process and outcome variables. The next section reviews the strengths and gaps under each of the frame- work’s domains—client need, structure, process, and outcomes. Recommen- dations for filling the identified gaps are presented at the end of the chapter. Adequacy of the Data Collection Infrastructure Client Need Consistent with column 1 of the committee’s evaluation framework, a key goal of the Title X program is to provide services to low-income, uninsured individuals; ethnically/racially and linguistically diverse women and men; and adolescents. Individuals with certain predisposing factors clearly are at risk of being unable to obtain needed reproductive health ser- vices. They include those without the economic means to pay for services, low-income individuals whose insurance plans do not cover contraceptive services, those who are linguistically and geographically isolated, those with limited knowledge regarding available contraceptive methods, and those who may have limited support in planning for the number and spacing of their children. Thus it is critical that the FPAR clearly identify the extent to which the program is reaching its target populations. Strengths The FPAR successfully captures information on the key characteristics of Title X clients, and the data collected suggest that the program is reaching

152 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM TABLE 5-1  Committee’s Evaluation Framework Applied to the Title X Program CLIENT NEED STRUCTURE PROCESS OUTCOMES Predisposing Enabling Client Factors Factors Health Behaviors Services Received Outcomes Service Use Modeled Process of Care: Performance Improved Client Client System System Process of Care: Client–Provider Service Use Indicators for Title X Modeled Clinical Characteristics Characteristics Characteristics Characteristics Client–Provider Encounter Performance Priority Areas Improved Outcomes Encounter Indicators for Title Clinical DERIVED DERIVED DERIVED FROM DERIVED FROM FPAR X Priority Areas NEW Outcomes FROM FPAR FROM FPAR FPAR PROPOSED 1. Enumeration of MEASURE TO 1. Income 1. Title X 1. Mandated care is services provided, BE 2. Insurance funding/other state of the art including screenings, COLLECTED 3. Age revenue for testing, and BY CPR/PRT 4. Gender each clinic DERIVED FROM contraceptives 5. Ethnicity 2. Title X CPR/PRT 2. No. of users receiving 1. Low-income (Latino) program testing and other women 6. Users with requirements Administrative data: services; no. of achieve their limited English 3. Ratio of positive test results family proficiency staffing to 1. Range of client planning goals 7. Race patient services offered by Examples of (Gregory, 8. Income— encounters qualified staff Service Use 2009 [see percent of a. No. of 2. Procedural outline Performance Appendix K]) poverty full-time to offer client indicators guidelines equivalents services and Defined by Family Derived 9. Insurance (FTEs) who document them in Planning from PART covers family are medical the client’s medical councils of and NSFG planning versus other record america POPULATION clinical 3. Written plan for DATA New service client education 1. Increased family Proposed providers 4. Report that planning services to 2. Decreased Measure to b. Nonclinical counseling, history, low-income clients to number of Be service and exam services decrease number of unintended Collected providers comply with Title X unintended pregnancies, By CPR/PRT 4. Limited requirements pregnancies particularly information re 5. Quality assurance a. 80% of among 1. Client interpreters program ongoing contracepting male low-income knowledge, during visits and female clients individuals intendedness, (not who return to clinic visit agenda necessarily continue any staff) method for 10–14 DERIVED 5. No. of months unless FROM CPR/PRT delegates seeking a supported by pregnancy 1. No. of clients Title X b. 90% of female 6. Service clients seeking planning sites contraception do supported by not report a Title X positive pregnancy test within 15 months of receiving contraception

COLLECTION OF DATA 153 TABLE 5-1  Continued Service Use Modeled Process of Care: Performance Improved Client System Client–Provider Indicators for Title X Clinical Characteristics Characteristics Encounter Priority Areas Outcomes DERIVED New Proposed 2. Increased screening 3. Reduced FROM CPR/PRT Measures To Be of females aged 15– infertility Collected By 24 for chlamydia among Administrative CPR/PRT infection women by data: a. 5% of female 7 identifying 1. Evaluate monthly clients under 25 chlamydia 1. No. of clinic the range of receive at least infections sites contraceptive one test for through 2. Compliance products available, chlamydia within screening of with including emergency 14 months females aged administrative contraception, to b. 100% of 15–24 requirements assess budget all female clients 4. Reduced for Title X site impact with a positive invasive structure, 2. Wait time for test for chlamydia cervical having written scheduling visit by are retested at the cancer among goals and an reason for visit first visit that women by evaluation 3. Continuity of care takes place 90 providing Pap plan, facilities, at the same site if days or longer tests staffing, policy needed after treatment; for language 95% of those assistance, etc. Patient-based measures who are retested 3. Personnel and test negative clinic 4. Care is patient- 3. Increased services to management centered and reduce invasive systems respectful cervical cancer (such 4. Client care 5. Clear information is as HPV protocols offered (bilingual immunization and 5. Training and counseling offered Pap tests, to be technical for those with defined by an expert assistance limited English committee); 6. Financial proficiency) benchmarks to be management 6. Patients feel determined using system welcomed by evidence-based 7. Systems to reception and guidelines involve the clinical staff during 4. Increased screening community all calls and visits for HIV/AIDS (to 7. Services are be added if perceived as recommended by an confidential expert panel); benchmarks to be determined using evidence-based guidelines NOTE: Italicized text represents indicators and approaches the committee recommends be added to OFP’s data collection system.

154 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM its target populations. Important demographic data provided by the FPAR include each client’s income, insurance coverage, gender, race/­ethnicity, and age and the number of users with limited English proficiency. In addition, the PRT used by the Regional Office or its designated representative col- lects relevant data as part of the CPR conducted every 3 years. These data include the total number of clients served, as well as the number served by each Title X delegate. In January 2005, Title X revised the FPAR to include new data elements, such as user health insurance coverage ­status, English proficiency, and contraceptive use by males (RTI International, 2006). These data elements help answer a variety of questions pertaining to the profile of the clients using Title X services. Gaps The FPAR does not provide client-level data on knowledge and preg- nancy intendedness. For example, if a clinic wanted to improve its education and support activities, it would need to collect data on clients’ knowledge of available contraceptive methods and their pregnancy intendedness. However, the clinics are not required to collect this information. OPA should explore the feasibility of gathering these data at the time of the CPR. It might be possible to use the CPR and site visit cycle to sample clients’ knowledge and intentions related to their visit agendas at the time of their clinic encounters if the representatives were provided the tools, through expert advice, neces- sary to obtain more information about knowledge, pregnancy intention, and satisfaction with provider interaction. These results then could be reviewed by the site visit team at the time of the CPR. It would be useful if the FPAR could collect more group-specific data in order to detect the effect of small changes in the performance indica- tors based on race and ethnicity, age and gender, level of income, and level of education. For example, this type of data could include unduplicated number of family planning users by age and gender; ethnicity and race; income level; limited English proficiency; primary contraceptive method; and number of gonorrhea, syphilis, HIV, and chlamydia tests. If there were a decline in the percentage of African American users, it would be useful to determine by group-specific data whether the decline was among teenagers, young adults, or older users who might have high STD rates and might not prefer birth control pills as their primary method. This information could help target outreach to be more specific and provide better program direc- tion. It would also be possible to identify any racial disparity in who, for example, obtains a Pap test or has an STD, or high infertility rates based on age, income, or educational level. It would also be useful to know the proportion of unmet need within the Title X target population, both overall and in different regions of the

COLLECTION OF DATA 155 country. This information, along with pregnancy intendedness information, could help inform Title X outreach planning for different regions. However, the collection of these data would require a new data collection instrument. Any expansion of the CPR to accomplish these goals would need to be sci- entifically valid, conducted in a way that preserves client confidentiality, and required for all grantees. To date, OFP has relied primarily on the NSFG to gather population-level data, and it might be possible for the NSFG to collect these types of data. Other data currently collected under cooperative agreements with OPA specifically examine the population in need of pub- licly funded family planning and how much of that need is met by family planning clinics, whether funded through Title X or not, at the state and county levels. The use of these data should also be explored. Finding 5-4. OFP collects the data needed to affirm that it serves the target populations for the Title X program. Data needed to affirm client knowledge and pregnancy intendedness; resources available to support clients’ childbearing decision making; and the proportion of unmet need within the Title X target popula- tions, both overall and in different regions of the country, are not being collected. However, these data could be gathered by sampling c ­ lients served or expanding the NSFG instead of requiring that this information be obtained on every client served. Structure A client’s ability to access quality services and the processes used to offer services are influenced by public policy; funding; staff training and availability; and facility factors such as location, hours of operation, com- fort, and privacy. Strengths FPAR. To ensure that there is adequate and appropriate staffing, the FPAR documents the number of full-time equivalents, categorized as medical versus other clinical service providers and as nonclinical service p ­ roviders. Data on the number of family planning encounters are also required. Together, these data provide the needed ratio of staffing to patient encounters. The FPAR also collects data on the sources of revenue for each Title X clinic so that federal grants, payment for services, and other sources of revenue are itemized. These data provide an opportunity to (1) calculate and compare the staff load within and across clinics, and (2) construct a cross-sectional and longitudinal profile of revenue sources and track how required service changes or expansions relate to gaps

156 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM in available revenue. Additional data elements include total number of delegates, clinic sites, and subcontractors supported through Title X, as well as service planning sites. All of these elements should continue to be measured in their current format. CPR. The CPR gathers substantial information about compliance with administrative requirements, such as commitment to voluntary participa- tion in services, confidentiality of services, and appropriate facilities. This information is obtained from written goals, objectives, policies for quality language assistance, and the like. The CPR also assesses whether clinic management is consistent with Title X guidance, whether there are written protocols for client care, and whether the required continuing education has been completed. In addition, the CPR examines the financial management system, as well as the quality of educational and informational program components. (The regulations require that grantees have a community board and a community education program, and provide for community participation in the development or selection of materials.) Finding 5-5. Both the FPAR and CPR collect important data needed to evaluate the adequacy of the Title X clinic structure and compliance with administrative guidelines to meet program goals. Gaps While both the FPAR and CPR collect substantial information on structural factors in clinic settings, it is not clear how the two sources are used to inform each other. For example, although the FPAR collects some information on the number of family planning users who have limited English proficiency, more specific information is needed about the avail- ability or quality of interpreters or the bilingual nature of staff to determine whether the needs of these clients at any given clinic are being met. It is not clear from reviewing the PRT whether or how an assessment could be conducted using current data on the availability of interpreters for each patient requiring such services. Additional analyses are needed to compare client characteristics and the enabling factors or system characteristics that are in place by region and by type of delegate. Financial information is thorough in the PRT, but it is not clear whether a profile of revenue sources, both cross-sectionally and longitudinally, is used to track how the required services change in relation to available revenue. For example, did a site’s inadequate revenue limit the type of con- traceptives or the number of months’ supply of contraceptives that clients could receive in certain periods of the year?

COLLECTION OF DATA 157 Finding 5-6. The Title X data collection tools gather key informa- tion on structure, such as revenue, staffing, and other enablers for services and number of services. However, it is unclear that these data are used effectively to examine the relationships between financing and services. Process A critical ingredient once clients access care is the type of care they receive. Title X mandates require delivery of state-of-the-art care and iden- tify specific services that must be provided, including screenings, testing, and contraceptive provision (Table 5-1, column 3). Strengths The PRT includes a variety of measures aimed at documenting that protocols are in place for delivering high-quality care. Thus, checklists elicit information on such factors as whether the full range of family planning ser- vices and Food and Drug Administration–approved contraceptive methods are provided to eligible clients by qualified and trained personnel. In addition, client chart protocols documenting the array of services provided during the initial visit are required, and there are extensive requirements for client con- sent; protocols for emergency care; and referrals, for example, for prenatal care. The PRT also notes whether clinic protocols are in place, such as those for follow-up for women and their partners when a chlamydia test is posi- tive. The FPAR collects data on the number of Pap tests performed that had an atypical squamous cells or high-grade squamous intraepithelial lesion or higher result, as well as the number of family planning users who obtained a clinical breast exam and were referred for further evaluation. The number of tests provided for chlamydia, gonorrhea, syphilis, and HIV is also collected. Gaps A weakness of both the FPAR and the CPR is that neither systemati- cally gathers information directly from clients about their family planning agenda or their experience at the clinic. Without this information, it is dif- ficult to know the extent to which clients achieved their family planning agendas; whether their visits were client-centered; to what extent their most important reproductive counseling, education, and support needs were met by the visit; whether they felt services were confidential and offered respectfully; whether the information offered was clear, particularly when an interpreter was needed; or how welcomed clients felt during initial and follow-up phone calls, as well as in their interactions with reception staff,

158 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM clinicians, and counselors during their visits. The lack of this information leaves a gap in the evaluation of the quality of care provided by Title X clinics. While the committee acknowledges that patient agendas are not the only consideration when public health matters, such as preventing the spread of STDs, are involved, it is useful to determine even in a represen- tative sample of clients whether clients had negative experiences or were required to receive services that would deter their return. Various ­methods, including telephone interviews, surveys, and community focus groups, could be used to collect data directly from clients. Although the FPAR captures client service data, it collects no specific quality measures for those services. The FPAR does not appear to collect data on many aspects of the process of care, such as the length of wait time to schedule a visit for different types of care (emergency contraception, initial family planning, pregnancy testing, fertility counseling, HIV testing) or what contraceptive methods are available at the time of a client’s ­family planning visit. This latter issue is a critical one given that some of the most effective, long-lasting methods are more expensive, and the commit- tee heard testimony that tight clinic budgets often limit the availability of these methods for clients who most need and desire them. Equally impor- tant, the reporting system does not capture the length of time a client’s contraceptive supply can last before a return visit for refills (if needed) is necessary. This, too, is a critical issue since barriers to accessing desired methods relate to adherence and subsequent pregnancy prevention. Given OFP’s commitment to quality performance indicators, these data gaps are especially noteworthy. There are other important indicators of quality for which the FPAR does not collect information. These include compliance with age- and g ­ ender-specific screening protocols; the extent to which appropriate tailor­ ing of protocols occurs, given both client characteristics and client-­identified needs; the wait time before clients receive test and other screening results and follow-up when merited by results; and continuity of care. For exam- ple, do clients see the same clinician in subsequent visits when possible, so that continuity is maximized? While the FPAR does identify specific services that must be provided, including screenings, clinical tests such as laboratory tests and Pap smears, and contraceptive provision and collects data regarding the number of abnormal test results, data are not available on referral and treatment provided as a consequence of those results. Thus, while the FPAR reports service use, reporting on the numbers alone does not provide sufficient data on the process of care. The PRT currently offers little insight into the individualization of care based on the different characteristics of clinics or client populations. For example, while outreach may be required to market clinic services to

COLLECTION OF DATA 159 targeted populations in some areas, it may not be needed at all for a clinic that has more patients than it can comfortably serve. Similarly, efficiency indicators in one clinic may not apply in another where the client popula- tion comprises adolescents who require more time for education and exams, or where the understanding and needs of clients who lack English language proficiency require special attention. It is important to document how coun- seling and education efforts are tailored to respond to the client and his or her partner’s childbearing decision making. This issue is particularly critical as the program serves ethnically/racially and culturally diverse clients. Their values regarding the overall number and timing of their children need to be considered in assessing the role Title X can play in ensuring the availability of appropriate services, including planning for desired children and avoid- ing unintended childbearing. To minimize the burden on staff of collecting information on the above process indicators, a plan for stratified random sampling by type of visit and test results could be designed with the assistance of an external group of scientific and clinical advisors. One possibility would be to integrate this data collection into the CPR. Having different types of Title X sites gather client information nationwide in a staggered fashion would make it possible to collect clinic-specific feedback from a restricted number of clients at any one time and also contribute to a national data picture. A multimethod approach would provide useful and complementary information, as sug- gested by Gregory (2009). As noted above, different methods could be used to collect data directly from clients, including telephone interviews, surveys, exit interviews, and community focus groups, depending on the specific evaluation question and the sampling plan for clinics selected to capture different types of clients and visits. In addition, standardized patients could participate and observe the complete trajectory of patient contact, from the initial phone call for an appointment through the completed service delivery, as suggested by Gregory. Finding 5-7. A number of indicators of the quality of the process or services provided are not systematically addressed by either the FPAR or CPR. In addition to considering gaps in data needed to evaluate the process of care, it is necessary to consider whether clinics have sufficient quality improvement indicators. One example would be service use performance indicators for Title X priority areas. As noted above, clinics enumerate the number of times they offer specific services in their FPAR. However, these data are not translated into a quality improvement indicator that clinics could use to compare performance for the current and previous years. This

160 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM would not be intended as an outcome measure but more as a formative evaluation indicator to improve the quality of service delivery. As noted earlier, OFP identified three performance measures through the PART process: (1) increasing family planning services to low-income clients to decrease the number of unintended pregnancies, (2) increasing screening of females aged 15–24 for chlamydia infection to reduce infer- tility, and (3) increasing Pap tests to reduce invasive cervical cancer. It is important to acknowledge the difficulty of connecting process indicators with outcomes. Nevertheless, the service performance indicators associated with these outcomes are valuable antecedents as quality improvement tools in themselves. To address quality improvement and management issues, OFP could establish benchmarks or standards for each of its priority outcome mea- sures. Table 5-1 provides examples of service use performance indicators for the Title X priorities using benchmarks from the Performance Measurement System (PMS) developed by the Family Planning Councils of America, Inc. (FPCA) and partially funded by OFP. OFP could closely examine the FPCA PMS as well as other potential benchmarks and related outcome measures with the assistance of an external panel of experts. Relevant Healthcare Employer Data and Information Set (HEDIS) and Centers for Disease Con- trol and Prevention (CDC) standards and goals could be used to reevaluate the benchmarks against which service quality can be measured. By using longitudinal within-clinic return visit data, clinics (especially those with electronic systems) could begin providing a broader picture of service per- formance quality directly linked to OFP goals. As an example, the goal of reducing infertility through chlamydia screening offers an opportunity for specific antecedent benchmarks against which service provision can be evaluated. According to the 2006 FPAR, 56 percent of all Title X female clients were aged 15–24; of those female clients 24 years and younger, only 51 percent were tested for chlamydia (RTI International, 2008), although HEDIS recommendations require that 100 percent of sexually active women in this age group be screened. To achieve the goal of reducing infertility among women, incremental targets for reaching national standards for rates of chlamydia testing and screen- ing among women aged 15–24 could be established. Rates for follow-up with treatment and for remaining negative at a follow-up visit also need to be tracked against benchmarks based on national standards (recognizing that whether one is negative on follow-up depends on both treatment and intervening exposure). Since some older women and men may be tested for chlamydia, OFP could consider how these groups fit into the Title X pri- orities. OFP might consider altering this goal to focus on the reduction of STDs among the groups most likely to be at risk, since the primary goal is to achieve reductions in infertility. Data could also be added to determine

COLLECTION OF DATA 161 what percentage of users are treated for STDs. As STDs are a reportable disease, local and state health departments have the results of positive STD tests and treatment. Family planning agencies could consider linking through information technology to local and state health departments for this information. As part of this process, expert consultation will be needed to determine whether the performance measures themselves should be modified. OFP’s goal of reducing cervical cancer through Pap tests deserves further review by an expert panel in light of HEDIS and CDC recommendations (see also Chapter 3). Guidelines for Pap tests were revised in 2002 and now state that a Pap test is not indicated until a woman has been sexually active for 3 years or reaches age 21, whichever occurs first. Guidelines for manage- ment of abnormal Pap tests were revised in 2007. Follow-up of abnormal Pap tests for adolescents in particular changed significantly. Unless there is a very abnormal test (i.e., high-grade lesion or worse), the follow-up should be at 1 year. Both of these changes have led to a decrease in Pap tests required, yet a gap continues to exist between program requirements and recommended practice. OFP could use an expert committee of clinicians and scientists to examine its primary approach to reducing cervical cancer. A goal of early immunization with the human papillomavirus vaccine for Title X female clients up to age 26 and age-appropriate Pap screening for other female clients should be considered. This could be accomplished in conjunction with the Vaccines for Children program. With the input of an expert panel, benchmarks for the immunizations and follow-up could be established, similar to those for chlamydia testing, and parallel national goals could be incorporated. With respect to the possible addition of an outcome related to HIV/AIDS, in 2006 CDC revised its guidelines to recommend that all individuals aged 13–64 in health care settings undergo HIV screening. HIV-positive status has crucial implications for pregnancy planning, as well as early intervention for the disease itself. According to the 2006 FPAR, Title X providers performed 1.3 HIV tests for every 10 family planning users. This is a measurable item that should have greater importance in the Title X program. OFP could therefore consider adding a goal regarding HIV testing and include this goal in its discussions of measures and bench- marks for service performance quality improvement. OFP could evaluate quality of care using national guidelines and bench- marks. Particular attention should be paid to how guidelines vary with the age of the client. For example, while OFP currently requires that adoles- cents receive breast examinations, the U.S. Preventive Services Task Force has concluded that the evidence is insufficient to recommend for or against routine clinical breast exams alone to screen for breast cancer, especially among the young (USPSTF, 2003). According to the National Breast and

162 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM Cervical Cancer Early Detection Program, 85 percent of breast cancers are identified in women over age 50. Mammograms detect 90 percent of breast cancers; the sensitivity of a clinical breast exam is 40–69 percent. Because the majority of Title X users are in an age group at low risk of breast cancer, mandating clinical breast examinations is of limited benefit and may deter some young women from seeking care. Data on this service would therefore no longer be collected if the Title X care guideline changed on the basis of scientific input. An important question is whether closing these data collection gaps would add to the burden on clinics. As more Title X clinic sites shift to elec- tronic health records, they will be able to perform longitudinal data analysis when provided with appropriate software and training. FPCA reports that with software and training, eight beta sites found the process relatively easy to implement and valued the additional data (Testimony by Dorothy Mann, May 19, 2008). The power of this approach is that it encourages each clinic to assess whether it is meeting national benchmarks set by OFP in keeping with HEDIS, CDC, and other federal agencies, thereby closing an important quality improvement feedback loop at the clinic level. Further, it becomes possible to analyze these data nationally and at the clinic level in terms of the demographics of the clients served. The rates at which low-income clients meet the benchmarks could be a particular focus of the within- and across-clinic analysis of outcome measures. Finding 5-8. The three core measures identified by OFP through the PART process are insufficient for successfully implementing and managing the Title X program or assessing outcomes. Outcomes An important product of OMB’s evaluation of OFP was that OFP identified the above three outcomes for evaluating the impact of Title X on population health. The committee recognizes both the difficulty and importance of defining and measuring these outcomes. For this assess- ment, the committee placed the greatest emphasis on the goal of increasing family planning services to low-income clients to decrease the number of unintended pregnancies since this is the predominant reason for visiting a family planning clinic. As discussed above, the committee suggests that OFP revisit the outcomes selected for evaluating the program’s impact on population health.

COLLECTION OF DATA 163 Decreasing Unintended Pregnancies While OFP can document the extent to which family planning services are provided to low-income clients, an important but much more difficult task is estimating the decrease in the number of unintended pregnancies associated with those services. As noted above, OFP estimates the decrease in the number of unintended pregnancies using a methodology originally developed in 1977 (Jaffe and Cutright, 1977). This methodology has evolved over time as better behavioral and utilization data at both the national and clinic levels have become available. The model estimates the number of unintended pregnancies by examining the current use of contraceptive methods by women visiting Title X clinics, the failure rates associated with those methods, and estimates of changes in contraception practices if Title X clinics were to close. A number of different estimates of the increase in u ­ nintended pregnancies are produced (five in the most recent paper by Frost et al., 2008b), depending on the assumptions made about how contraceptive behavior would change (including “would give up prescription contracep- tion altogether” and “using contraception in the same way as comparable women who do not attend Title X clinics”). OPA bases its estimates on the average of the four most realistic scenarios. While the committee agrees that this approach is a reasonable one for estimating the number of ­unintended pregnancies, it recommends that OFP provide data on the particular assump- tions used to make these estimates. In addition, techniques should be devel- oped to increase the robustness of these estimates and their sensitivity to the quality of the services provided. Three specific improvements in methodol- ogy and data collection would enhance OFP’s understanding of the effective- ness and cost-effectiveness of Title X services: • Estimates should rely on program-specific data on contraceptive methods dispensed and be sensitive to the types and quantity of methods used in Title X clinics rather than on the distribution found in the NSFG. Use of data on the provision of contraceptives by Title X programs would allow the analyses to reflect changes in methods dispensed, quantities dispensed, and method continuation. The committee anticipates that more sensitive evaluation methods would encourage quality improvement, as well as increases in clients served. This approach has been used in evaluations of the California Medicaid Waiver Program (Foster et al., 2004, 2006). • There is room for improvement in modeling the absence of the Title X program. Research on contraceptive use in the absence   he Frost et al. paper actually models the impact of publicly funding family planning T clinic services, not Title X clinics specifically. However, it is transferable to an examination of Title X clinics only.

164 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM of public funding could be improved, for example, by examining method use prior to program inception among new Title X clients or by asking a sample of Title X clients what they would do if subsidized services were unavailable. This information could be included in sensitivity analyses. • Cost savings should be adjusted to differentiate between costs that were entirely prevented through use of contraceptives provided by Title X clinics and those that were merely delayed. Although sig- nificant research would be needed to yield reliable estimates, failure to make this adjustment results in overestimation of costs averted (Amaral et al., 2007). Finally, given the importance of the problem, multiple models and approaches for estimating the number of unintended pregnancies averted through Title X would be useful to explore and compare. Strategic Use of OPA/OFP Research Funding to Demonstrate Outcomes Findings from Title X–funded research are disseminated in a number of ways. Summaries of the research are posted on the OPA website; research grant recipients make formal presentations at the biennial Title X national grantee meetings and other national conferences/meetings; and researchers publish their results in peer-reviewed journals (see Appendix I for a sample of peer-reviewed publications resulting from OPA-funded research). It is noteworthy that in the almost 40 years that Title X’s research pro- gram has been in place, too little remains known about how best to promote and encourage contraceptive use among both men and women. In a recent review, Kirby (2008) found that there is a significant lack of research and evaluation on interventions to promote the use of family planning methods, a lack that severely limits the nation’s ability to help couples both plan for pregnancy and prevent unintended pregnancy. Given the nation’s high rate of unintended pregnancy, this knowledge gap is particularly troubling. An important resource for OFP is the Title X research program’s ability to target relevant issues regarding both outcomes and service delivery. OPA and OFP should carefully evaluate the currently funded research program to ensure that it frames the mission and use of targeted research effectively. The goal is to fund research on concerns, issues, and interventions whose results will have value for subsequent decisions and dissemination among the Title X network. For example, OFP may want to consider targeting research to evaluate program initiatives it has piloted to determine whether wider dissemination of those initiatives is merited. Part of the Title X research effort should involve collaborating with other federal agencies to determine unmet need among the Title X target

COLLECTION OF DATA 165 population. Addressing this question will require additional data sets on nonusers of Title X clinics, describing family planning needs and indicators for target populations in the region. OFP should apply some of its research funds to using data available through other existing national databases more effectively. It should study the value of the current items available from the NSFG and evaluate the merits of adding other items if needed. Other sys- tematic national population surveys should be examined for the potential to augment the FPAR outcome evaluation. The more directly items collected by these additional national surveys can be tied to the priority outcomes for Title X, the greater the opportunity for comparisons will be. The committee also suggests that OFP consider regional evaluations whereby data from a variety of sources can be compared to identify the extent to which Title X is contributing to key outcomes and whether new initiatives appear to be helpful. Projects that take advantage of claims data from Medicaid and other third-party payers in some counties and regions offer unique opportunities for comparison with data collected on Title X clients. The 2006 FPAR data suggest additional analyses and probes to help evaluate the program’s impact on preventing unintended pregnancy. Accord- ing to these data, 85 percent of women attending Title X clinics were using a birth control method. The percentages of use by method were as follows: • Oral contraceptives, 39 percent • Condoms, 16 percent • Injectable, 12 percent • Patch, 4 percent • IUD, 2 percent • Sterilization, 2 percent • Vaginal ring, 2 percent • Abstinence, 1 percent Each of these methods differs in its profile of discontinuation and failure rates. For example, oral contraceptives have a probability of failure rate of 8.7 percent of users at 1 year, while the every 3 months injectable (Depo-Provera®) has a failure rate of 6.7 percent of users at 1 year. These failure rates are highest among those under age 30, especially low-income teenagers. Given that long-acting methods such as the IUD or the implant are more effective and cost-effective, it would be useful to track not only the availability of these methods at each family planning clinic visit, but also their prescription rate at the visit. Lastly, the 2006 FPAR reports that 15 percent of women using Title X services were not using birth control, were pregnant, or wanted to be. The FPAR provides very little information about this group. To help evaluate

166 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM OFP’s progress in helping women achieve their family planning goals, it would be useful to understand more about whether the majority of these women were planning to become pregnant and what role Title X played in helping them with the spacing of subsequent pregnancies. conclusions and recommendations The committee’s comparison of the data needed to monitor and evalu- ate the Title X program against the data actually collected supports the following conclusions: The program does not collect all the data needed to fully m ­ onitor the program and evaluate its impact. A comprehensive framework for approaching program evalua- tion could ensure that all major aspects of the program are ­evaluated and the needs of clients are being met. Gathering these data will require innovative approaches―and new funding―to minimize the burden on providers. The following recommendations are intended to help OFP strengthen its ability to meet its goals through improved data collection. These recom- mendations are based on recent literature reviews and reports on quality in the provision of family planning services (Sonenstein, 2006; Becker et al., 2007), papers commissioned by the committee (Gregory, 2009; The Lewin Group, 2009 [see Appendixes K and J, respectively]), previous reports com- missioned by OFP (Sonenstein et al., 2004; RTI International, 2005a,b), the committee’s site visits (see Appendix F), and testimony provided to the com- mittee during public workshops. The challenge is to move an evaluation agenda and process forward without imposing an undue burden on clinic providers, clients, and regional administration. The committee therefore recommends that a formal planning process be undertaken by OFP. Recommendation 5-1: Fund and use a comprehensive framework to evaluate the Title X program. OFP should develop, fund, and use a comprehensive framework to evaluate the Title X program. The use of such a framework would allow OFP to evaluate the pro- gram on the full continuum from clinic performance and quality, to clinic management, to program outcomes. It would also help in identifying the types of data needed for evaluation purposes. The development and use of a comprehensive framework to evaluate Title X would make it possible to explore the interactions among various

COLLECTION OF DATA 167 contributing factors and outcomes, as well as the completeness of different measures for each set of factors. It is important to providing funding for data collection, analysis, and use rather than diluting service dollars. Recommendation 5-2: Examine the data elements of the ­ Family Planning Annual Report (FPAR). When revising the Program Guidelines (see Recommendation 4-7), OFP should review and clarify data elements contained in the FPAR and where possible and useful, eliminate those that are unnecessary, particularly if additional elements are needed. Recommendation 5-3: Collect additional data. To help fill gaps in the Title X program’s data collection systems, OFP should collect additional data in the areas of client needs, structure, process, and outcomes for use in evaluating the program’s progress and its effec- tiveness in achieving its goals. Specifically, OFP should: • Collect additional data on client characteristics. Additional data sources, such as the Comprehensive Program Reviews (CPRs), should be used to supplement the FPAR data on client characteristics—for example, to obtain data on clients’ knowl- edge about available contraceptive methods and pregnancy intentions. • Collect data on system characteristics. Additional data are needed on such system characteristics as the availability of interpreters to meet the needs of clients with limited English proficiency. • Collect data on the process of care. — These data should include patients’ perceptions of care. With expert consultation, selected CPR site visits could be structured to sample a limited number of clients for the purpose of obtaining generalizable results. — With expert advice, OFP should examine the three core outcome measures identified for the PART process in rela- tion to evidence-based guidelines and national health pri- orities. After determining the most appropriate measures, OFP should develop related performance metrics for clinic service to establish quality improvement standards. • Conduct research to assess program outcomes. OFP should expand research aimed at evaluating program outcomes, such as the impact of the program on pregnancy planning and inten- tion, decreased infertility, outreach to those in need of services, and the prevention of unintended pregnancy.

168 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM Recommendation 5-4: Examine evaluation tools for outreach and education. To assist ongoing quality improvement and effective expansion of community outreach and education, OFP should work with grantees to develop and refine evaluation tools for out- reach and education that can be applied easily by delegates. It is important to acknowledge that in many ways, the outreach and education responsibilities (and opportunities) encompassed by Title X often receive less attention than other aspects of the program because of the pressing needs of the clinical care system and the lack of resources. How- ever, the expertise in reproductive health that exists in so many communities across the country needs to be utilized to fuller advantage. The increasing leadership role for OFP and the Title X system of services that the commit- tee recommends (see Chapter 3) extends to outreach and education respon- sibilities (and opportunities). In addition to clinic services, Title X grantees and delegates devote considerable effort to responding to the needs of their communities through outreach and education programs. As discussed in Chapter 4, there is a need to examine the evaluation of these important activities. The current tools used to evaluate the program tend to document quantity and client satisfaction, both of which are important. The question is whether more can be done to extend and evaluate outreach and educa- tion efforts. Can more be done to develop easily implemented strategies and tools for promoting and evaluating the quality of these efforts and their impact on such outcomes as knowledge, attitudes, and intentions related to preventing unintended pregnancy and associated preventive health issues, such as prevention of STDs? Recommendation 5-5: Obtain scientific input on evaluation efforts. OFP should expand its use of scientific expertise to strengthen its evaluation strategies and improve its evaluation research program, and consider expanding its use of national databases to evaluate program impacts. A group of scientific experts should be established for the program, representing knowledge of public health practices and principles; the rel- evant clinical specialties, including primary care, obstetrics and gynecology, and adolescent health; health education, behavioral science, and health services research; epidemiology; and ethnography. This group could serve multiple functions (see also Recommendation 4-7 in Chapter 4) by review- ing standards of care annually, assessing relevant FPAR and CPR measures and samples, and helping to identify an OFP research agenda. That agenda could (1) explore how the Title X program can reduce unintended pregnan- cies more effectively; (2) be linked directly to improving the nation’s family

COLLECTION OF DATA 169 planning service system; and (3) ensure coordination with other federal research efforts that address common issues of quality assurance, clinical guidelines, and related matters. This group could have systematic input into the quality improvement indicators collected and strategies for increasing clinic feedback and information exchange. Lastly, this group could assist OFP in addressing the recommendations offered in this report. Expert and clinic site consultation would also be helpful in addressing the key question of how OFP should better use the data it currently col- lects. As discussed, OFP relies on different but complementary sources of data, as well as data collection strategies, as part of its national evaluation system. This multiplicity is vital as any one source of data may be unable to provide the level of information necessary—each having its strengths and limitations. The ways in which OFP synthesizes and uses existing data for program planning, including process data and service delivery improvement research funded by OPA, are not fully transparent. As discussed in Chapter 3, grantees expressed concern regarding their perception that new service priorities often appear to be announced without explanation of how they were established or how they relate to an overall strategic plan. Furthermore, there appears to be a gap between requiring new priorities and adapting the existing data collection system to capture and document the program’s success in responding to these priorities. If the data collection system is not incorporating new program priorities, the efforts of grantees to respond to those priorities may not be fully captured. The FPAR was last modified in 2005 (to include new data elements, such as user health insurance coverage, English proficiency, contraceptive use by males, summary Pap [abnormal] and confidential HIV [positive] test results, and disease-specific information on STD screening). While it is unrealistic to modify the FPAR frequently, it may be realistic to require a more limited set of data elements for programs engaged in implementing new priorities to determine whether those priorities are being implemented fully and having the desired impact. Soliciting both scientific and clinic-based input on these questions would be useful. Recommendation 5-6: Communicate evaluation findings. To ensure transparency and broad-based dissemination of information and ultimately to improve care (see Recommendation 4‑8), OFP should enhance ongoing feedback and communication with grantees, del- egates, clinics, and others about important evaluation findings and how they can help improve care and track progress toward the achievement of program goals. During the committee’s site visits with Title X clinics, as well in testi- mony at public workshops, several providers expressed a desire for more

170 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM feedback and information on the FPAR. The committee recommends that OFP consider a variety of ways to meet this need (see Recommenda- tion 4-8). Feedback on clinic performance and data trends via webpages and web broadcasts could offer additional information to clinics. Equally important, it could help clinic staff understand the significance of their work on collecting FPAR data. CONCLUDING THOUGHTS The committee has identified a variety of ways in which the Title X program could be improved. These include focusing on its core mission; undertaking a strategic planning process with a longer time horizon; imple- menting patient-focused, scientifically based clinical practices; and enhanc- ing evaluation and communication. There is also a need to coordinate requirements of relevant federal agencies with Title X. The committee acknowledges that the current systems used by OFP to collect program data have significant strengths. Recognizing that more resources will be required, the committee believes there are opportunities to utilize data that are already available, as well as to improve the collection of those data in ways that are sensitive to the needs of both grantees and clients while also contributing to quality improvement efforts. Although there is room for improvement, it is important to note that the Title X program has successfully served low-income women and men and adolescents. Despite increasingly limited funds, the dedication of fed- eral agency staff, grantees, delegates, and clinic staff to the goals and clients of the Title X program has made it possible to deliver essential services that have helped individuals, families, communities, and the nation.

Next: References »
A Review of the HHS Family Planning Program: Mission, Management, and Measurement of Results Get This Book
×
Buy Paperback | $68.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

A Review of the HHS Family Planning Program provides a broad evaluation of the Title X family planning program since its establishment in 1970. The program successfully provides family planning services to its target audience of low-income individuals, but there is room for improvement. While the program's core goals are apparent, a secondary set of changing priorities has emerged without a clear, evidence-based strategic process. Also, funding for the program has increased in actual dollars, but has not kept pace with inflation or increased costs. Several aspects of the program's structure could be improved to increase the ability of Title X to meet the needs of its target population. At the same time, the extent to which the program meets those needs cannot be assessed without a greater capacity for long-term data collection.

A Review of the HHS Family Planning Program recommends several specific steps to enhance the management and improve the quality of the program, as well as to demonstrate its direct contribution to important end results, such as reducing rates of unintended pregnancy, cervical cancer, and infertility. The book will guide the Office of Family Planning toward improving the effectiveness of the program. Other parties who will find the research and recommendations valuable include programs receiving Title X funding from the Office of Family Planning, policy makers, researchers, and professional organizations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!