5
Collection of Data to Measure Program Outcomes

Under the auspices of the Office of Population Affairs (OPA), the Office of Family Planning (OFP) uses a variety of measures to provide programmatic information that is both timely and responsive to a wide range of stakeholders concerned with the program’s scope, quality, and reach. This chapter addresses the ability of these measures to assess the program’s goals, processes, and outcomes. It begins by summarizing the Office of Management and Budget’s (OMB) Performance Assessment Rating Tool (PART) evaluation process for Title X. It then presents a series of evaluation questions and an evaluation framework used by the committee to assess the sources and types of data collected by OFP. Next is a review of current sources of data for program assessment. The committee’s evaluation framework is then used to assess OFP’s measures and goals. This is followed by an assessment of the data collection infrastructure for the Title X program. The chapter ends with conclusions and recommendations for improving the Title X evaluation system as a whole, drawing on recommendations from earlier groups as well as this committee.

PART PROCESS FOR EVALUATING TITLE X

There have been several evaluations of specific aspects of the Title X program, including both government reviews and evaluations conducted by nongovernmental organizations at the request of OPA (see Appendix H for a summary of findings and recommendations from these evaluations). The present study was prompted by OMB’s evaluation of Title X under the PART process (OMB, 2005) (the PART is presented in Appendix E). The



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 145
5 Collection of Data to Measure Program Outcomes Under the auspices of the Office of Population Affairs (OPA), the Office of Family Planning (OFP) uses a variety of measures to provide program- matic information that is both timely and responsive to a wide range of stakeholders concerned with the program’s scope, quality, and reach. This chapter addresses the ability of these measures to assess the program’s goals, processes, and outcomes. It begins by summarizing the Office of Management and Budget’s (OMB) Performance Assessment Rating Tool (PART) evaluation process for Title X. It then presents a series of evaluation questions and an evaluation framework used by the committee to assess the sources and types of data collected by OFP. Next is a review of current sources of data for program assessment. The committee’s evaluation frame- work is then used to assess OFP’s measures and goals. This is followed by an assessment of the data collection infrastructure for the Title X program. The chapter ends with conclusions and recommendations for improving the Title X evaluation system as a whole, drawing on recommendations from earlier groups as well as this committee. PART PROCESS FOR EVALUATING TITLE X There have been several evaluations of specific aspects of the Title X program, including both government reviews and evaluations conducted by nongovernmental organizations at the request of OPA (see Appendix H for a summary of findings and recommendations from these evaluations). The present study was prompted by OMB’s evaluation of Title X under the PART process (OMB, 2005) (the PART is presented in Appendix E). The 5

OCR for page 145
6 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM program received a rating of Moderately Effective (rating categories include Results Not Demonstrated, Ineffective, Adequate, Moderately Effective, and Effective). The highest scores were achieved in the areas of Program Purpose and Design and Program Management, with lower scores in the areas of Strategic Planning and Program Results/Accountability. The com- mittee concurs that strategic planning is an area for improvement (see the discussion and recommendations in Chapter 3). As required by the OMB evaluation process, OFP defined three long- term annual performance measures, described in Chapter 3: (1) increasing the number of unintended pregnancies averted by providing Title X family planning services, with priority for services to low-income individuals; (2) reducing infertility among women attending family planning clinics by identifying chlamydia infection through screening of females aged 15–24; and (3) reducing invasive cervical cancer among women attending family planning clinics by providing Pap tests according to nationally recognized standards of care. In addition to defining these three long-term measures, OFP had earlier developed, implemented, and established targets for all annual and long-term performance goals, including its efficiency measure, which is to maintain the actual cost per client below the medical care inflation rate. These measures, along with quality-of-care indicators, are assessed in this chapter. COMMITTEE’S EVALUATION QUESTIONS The committee examined two key issues. The first is what data collec- tion infrastructure is currently in place for the Title X program. The second is how the data collection effort and its infrastructure can be improved. The following questions, which emerged from the committee’s charge, the PART process, and previous program evaluations, framed the committee’s assessment of the measures around these two issues: • What types of data are being used by OFP for monitoring and assessing the Title X program? • Does the existing portfolio of data collection approaches ade- quately capture Title X’s activities? • What modifications should be made to the data collection system and planning? COMMITTEE’S EVALUATION FRAMEWORK To answer the above questions, the committee developed an evaluation framework that focuses on Title X’s primary mission—to provide individuals the information and means to exercise personal choice in determining the

OCR for page 145
7 COLLECTION OF DATA CLIENT NEED STRUCTURE PROCESS OUTCOMES Enabling Client Predisposing Factors Outcomes Health Behaviors Services Received Factors Client System Process of Care: Service Use Modeled Characteristics Characteristics Client– Provider Performance Improved Encounter Indicators for Title Clinical X Priority Areas Outcomes FIGURE 5-1 Conceptual framework for Title X evaluation. number and spacing of their children (see Figure 5-1). The committee’s conceptual framework structures this final chapter. Discussion of how the framework can be used to evaluate the program is detailed below. The committee believes such a framework can help establish a more integrated and comprehensive evaluation approach for Title X by linking the program’s assessment to its stated goals and priorities. Consistent with Title X’s commitment to ensuring adequate quality of care, the commit- tee’s evaluation framework draws on well-tested models for evaluation of the quality and utilization of health services—Donabedian’s Quality Model (Donabedian, 1968, 1980) and Andersen’s Health Care Utiliza- tion Model (Andersen and Davidson, 2007). The use of this framework offers an opportunity to place the discussion of quality of care within the broader context of national and various state health policies and Title X financing as enabling factors for the Title X program, patient care, and education and outreach. The goal of the framework is to help OFP maintain a cohesive and pro- active evaluation program focused on quality improvement and the ability to document outcomes. The framework provides a logic model emphasizing measures linking (1) the clients to be served; (2) the resources, facilities, and personnel required to serve them (structure); (3) the services actually provided (process); and (4) the results of those services (outcomes). Having such a model allows planners to determine returns on various investments or interventions. The committee acknowledges the challenging task of identifying, at a minimum, meaningful and prioritized short- and longer- term outcomes that are valid indicators and can readily be measured in a clinic setting. As with any data collection, it is necessary to recognize the resource and staff costs associated with complete and timely data collec- tion at the local level. Moreover, longer-term outcomes may be difficult to capture adequately without significant investments in client tracking and data collection.

OCR for page 145
 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM The committee’s framework encompasses: • Client need, or predisposing factors (column 1)—These factors include characteristics of clients that motivate them to use services and that Title X should consider in defining its profile of services. • Structure, or enabling factors (column 2)—These system character- istics enable patient access to health care services and facilitate the delivery of quality services important to client outcomes. • Process, or health behaviors/services received (column 3)—These process factors pertain to the interaction between clients and pro- viders, as well as the services received and how they align with established service use performance indicators for the Title X priority areas. • Outcomes (column 4)—These factors refer to improvements in clinical outcomes that occur as a result of clients’ interactions with the system of services and outreach. Depending on the evaluation question, measures relevant to the Central Office, Regional Offices, grantees, delegates, clients, populations in need, and barriers to care may be included in the data collection. For example, Title X’s Central Office may be interested in answering questions pertaining to overall trends in the numbers and profiles of clients served across the country. At the delegate level, there may be interest in questions that pertain to comparisons among similar types of delegates or to contrasts among dif- ferent types of delegates in various geographic areas. CURRENT SOURCES OF DATA FOR PROGRAM ASSESSMENT OFP currently uses a variety of data sources for monitoring and assess- ing the Title X program (see Chapter 4). The Family Planning Annual Report (FPAR) is the primary source of annual, uniform reporting by all Title X grantees (see Appendix G for the FPAR data elements). Informa- tion from the FPAR is important to OPA for several reasons. First, FPAR data are used to monitor compliance with statutory requirements, regula- tions, and operational guidance set forth in the Program Guidelines, which include giving priority in the provision of services to low-income persons (Section 1006(c) of Title X of the Public Health Service Act, 42 USC § 300) and ensuring that Title X grantees and their subcontractors provide a broad range of family planning methods and services (Section 1001(a) of Title X of the Public Health Service Act, 42 USC § 300). Second, OPA uses FPAR data to comply with accountability and federal performance requirements for recipients of Title X family planning funds as required by the 1993 Government Performance and Results Act (GPRA).

OCR for page 145
 COLLECTION OF DATA Current GPRA performance goals for the Title X program include as pri- orities the provision of family planning services to low-income individuals, access to and utilization of cervical and breast cancer screening, and access to on-site HIV testing at Title X–funded clinics. Finally, the program relies on FPAR data to monitor performance, respond to inquiries about the program from policy makers and Congress, and support program planning. The FPAR allows OPA to assemble compa- rable and relevant program data to answer questions about the character- istics of the populations served by Title X clinics, utilization of the services offered, the composition of revenues, and program impact. FPAR data are the basis for objective grant reviews, program evaluation, and assessment of program technical needs (RTI International, 2008). Each year, FPAR data are analyzed to produce the Family Planning Annual Report National Summary. This report provides analyses of the FPAR measures by demographics; social and economic profile; method use, by gender; and cervical, breast, and sexually transmitted disease (STD) screening. It also includes information on staffing, family planning encounters, and revenue. The Research Triangle Institute (RTI) conducted analyses of the FPAR measures to produce the 2006 National Summary (RTI International, 2008). Although the committee anticipated that its site visits and the testimony of grantees and delegates would elicit reports of the burden imposed on clinic staff by meeting the FPAR requirements, a more complex picture emerged. Many interviewees reported that complying with FPAR require- ments was not burdensome, especially after they switched to electronic health records. Others, however, indicated it was difficult to meet these requirements; indeed, the committee was told that the FPAR requirements deterred some clinics from seeking Title X funding. Finding 5-1. Sites vary in their capacity to meet the FPAR data collection requirements. Another source of information that OFP uses to evaluate the Title X program is the Comprehensive Program Review (CPR), conducted every 3 years by OFP’s Regional Offices. This review is intended to ensure that grantees are complying with Title X policy and program requirements, as well as OMB regulations. The Program Review Tool (PRT) used in these CPRs contains key questions on administration, financial management, clinical services, and outreach/information (see Appendix J, Table J-2, for data collected by the PRT). Grantee-monitoring site visits are then conducted by the Regional Offices to check on compliance and to follow up on issues identified previously in the CPR, grant application, and/or needs assessment. The CPR activities suggest that there is considerable

OCR for page 145
50 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM system capacity for and commitment to evaluation. The question the com- mittee raises is whether this capacity could be used more strategically to strengthen the evaluation and performance of the program. OFP might further structure this evaluation visit, with input from evaluation experts, RPCs, grantees, and delegates. This group would define core goals and provide tools/approaches that would be used during site visits. Finding 5-2. The Comprehensive Program Review represents an underutilized opportunity for systematic assessment across the Title X program. Another source of data is the National Survey of Family Growth (NSFG), which Title X partially supports on an annual basis. This large- scale survey asks a nationally representative group of women a series of questions related to reproductive behavior and health. Key variables most relevant to Title X include (1) whether family planning medical or con- traceptive services were received, (2) contraceptive(s) received, (3) type of family planning medical services received, (4) type of setting where services were received (e.g., hospital, health maintenance organization facility, com- munity clinic), (5) location (i.e., name, address) of the facility, (6) whether the location was clients’ regular source of care, (7) how the service was paid for (e.g., copayment, Medicaid, insurance, free), and (8) whether services were paid for by clients on a sliding-scale basis. Data on the Title X program that may be useful for evaluation purposes are also available from several other sources. These include, for example, contracts that are funded by HHS 1 percent evaluation funds, the Service Delivery Improvement grant program, family planning research cooperative agreements, and demonstration projects aimed at addressing key program initiatives (see Appendix I). In 2008, OPA spent approximately $9 million on research and evalua- tion activities. The goal is to identify emerging needs in family planning, as well as to develop and evaluate service innovations that respond to identi- fied needs, with the implied intent of disseminating findings to grantees. Findings from OPA-funded research are also important for monitoring service delivery needs and the provision of care by clinics funded under the program. Finding 5-3. The National Survey of Family Growth and other research grant programs funded by OPA hold potential for more targeted research to enhance evaluation agendas for Title X. Moni- toring to ensure wider dissemination of OPA-supported research findings to the provider community and use of these findings is key to ongoing efforts to improve service delivery.

OCR for page 145
5 COLLECTION OF DATA APPLYING THE COMMITTEE’S FRAMEWORK TO ASSESS OFP’S MEASURES AND GOALS The committee used the evaluation framework presented in Figure 5-1 to evaluate whether the measures currently used by OFP adequately assess the program’s progress and effectiveness in meeting its goals. The primary data sources reviewed were the FPAR and to a lesser degree the CPR/PRT. Table 5-1 shows the data currently being collected that are relevant to each of the framework’s domain columns. Also listed are indicators and approaches the committee recommends be added (presented in italics). The evaluation framework was also used to guide the committee’s recommendations for improving the evaluation of Title X patient care and counseling, as well as to help in identifying emerging needs for outreach and education. OFP’s collection of data elements is much stronger for some columns of the framework than others. On the one hand, OFP already collects data on key characteristics of clients served, several critical system characteristics, and services performed. On the other hand, OFP does not systematically collect data on key process and outcome variables. The next section reviews the strengths and gaps under each of the frame- work’s domains—client need, structure, process, and outcomes. Recommen- dations for filling the identified gaps are presented at the end of the chapter. ADEQUACY OF THE DATA COLLECTION INFRASTRUCTURE Client Need Consistent with column 1 of the committee’s evaluation framework, a key goal of the Title X program is to provide services to low-income, uninsured individuals; ethnically/racially and linguistically diverse women and men; and adolescents. Individuals with certain predisposing factors clearly are at risk of being unable to obtain needed reproductive health ser- vices. They include those without the economic means to pay for services, low-income individuals whose insurance plans do not cover contraceptive services, those who are linguistically and geographically isolated, those with limited knowledge regarding available contraceptive methods, and those who may have limited support in planning for the number and spacing of their children. Thus it is critical that the FPAR clearly identify the extent to which the program is reaching its target populations. Strengths The FPAR successfully captures information on the key characteristics of Title X clients, and the data collected suggest that the program is reaching

OCR for page 145
52 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM TABLE 5-1 Committee’s Evaluation Framework Applied to the Title X Program CLIENT NEED STRUCTURE PROCESS OUTCOMES Enabling Client Predisposing Factors Outcomes Health Behaviors Services Received Factors Service Use Modeled Process of Care: Performance Improved Client System Client–Provider : Process of Care IService Use Title X Modeled Client System ndicators for Clinical Characteristics Characteristics Client– Provider Performance Improved Characteristics Characteristics Encounter Priority Areas Outcomes Encounter Indicators for Title Clinical DERIVED DERIVED DERIVED FROM DERIVED FROMs X Priority Area FPAR NEW Outcomes FROM FPAR FROM FPAR FPAR PROPOSED 1. Enumeration of MEASURE TO 1. Income 1. Title X 1. Mandated care is services provided, BE 2. Insurance funding/other state of the art including screenings, COLLECTED 3. Age revenue for testing, and BY CPR/PRT 4. Gender each clinic DERIVED FROM contraceptives 5. Ethnicity 2. Title X CPR/PRT 2. No. of users receiving 1. Low-income (Latino) program testing and other women 6. Users with requirements Administrative data: services; no. of achieve their limited English 3. Ratio of positive test results family proficiency staffing to 1. Range of client planning goals 7. Race patient services offered by EXAMPLES OF (Gregory, 8. Income— encounters qualified staff SERVICE USE 200 [see percent of a. No. of 2. Procedural outline PERFORMANCE Appendix K]) poverty full-time to offer client INDICATORS guidelines equivalents services and DERIVED DEFINED BY FAMILY 9. Insurance (FTEs) who document them in FROM PART PLANNING covers family are medical the client’s medical AND NSFG COUNCILS OF planning versus other record POPULATION AMERICA clinical 3. Written plan for DATA service client education NEW . Increased family providers 4. Report that 2. Decreased PROPOSED planning services to b. Nonclinical counseling, history, number of MEASURE TO low-income clients to service and exam services unintended BE decrease number of providers comply with Title X pregnancies, COLLECTED unintended 4. Limited requirements particularly BY CPR/PRT pregnancies information re 5. Quality assurance among a. 0% of interpreters program ongoing low-income . Client contracepting male during visits individuals knowledge, and female clients (not intendedness, who return to clinic necessarily visit agenda continue any staff) method for 0– DERIVED 5. No. of months unless FROM CPR/PRT delegates seeking a supported by pregnancy 1. No. of clients Title X b. 0% of female 6. Service clients seeking planning sites contraception do supported by not report a Title X positive pregnancy test within 5 months of receiving contraception

OCR for page 145
5 COLLECTION OF DATA TABLE 5-1 Continued Service Use Modeled Process of Care: Performance Improved Client System Client–Provider Indicators for Title X Clinical Characteristics Characteristics Encounter Priority Areas Outcomes DERIVED 2. Increased screening 3. Reduced NEW PROPOSED FROM CPR/PRT infertility MEASURES TO BE of females aged 5– among COLLECTED BY 2 for chlamydia Administrative women by CPR/PRT infection data: identifying a. 75% of female chlamydia . Evaluate monthly clients under 25 1. No. of clinic infections the range of receive at least sites through contraceptive one test for 2. Compliance screening of products available, chlamydia within with females aged including emergency  months administrative 15–24 contraception, to b. 00% of requirements 4. Reduced assess budget all female clients for Title X site invasive impact with a positive structure, cervical 2. Wait time for test for chlamydia having written cancer among scheduling visit by are retested at the goals and an women by reason for visit first visit that evaluation providing Pap . Continuity of care takes place 0 plan, facilities, tests at the same site if days or longer staffing, policy needed after treatment; for language 5% of those assistance, etc. Patient-based measures who are retested 3. Personnel and test negative clinic . Care is patient- . Increased services to management centered and reduce invasive systems respectful cervical cancer (such 4. Client care 5. Clear information is as HPV protocols offered (bilingual immunization and 5. Training and counseling offered Pap tests, to be technical for those with defined by an expert assistance limited English committee); 6. Financial proficiency) benchmarks to be management 6. Patients feel determined using system welcomed by evidence-based 7. Systems to reception and guidelines involve the clinical staff during . Increased screening community all calls and visits for HIV/AIDS (to 7. Services are be added if perceived as recommended by an confidential expert panel); benchmarks to be determined using evidence-based guidelines NOTE: Italicized text represents indicators and approaches the committee recommends be added to OFP’s data collection system.

OCR for page 145
5 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM its target populations. Important demographic data provided by the FPAR include each client’s income, insurance coverage, gender, race/ethnicity, and age and the number of users with limited English proficiency. In addition, the PRT used by the Regional Office or its designated representative col- lects relevant data as part of the CPR conducted every 3 years. These data include the total number of clients served, as well as the number served by each Title X delegate. In January 2005, Title X revised the FPAR to include new data elements, such as user health insurance coverage status, English proficiency, and contraceptive use by males (RTI International, 2006). These data elements help answer a variety of questions pertaining to the profile of the clients using Title X services. Gaps The FPAR does not provide client-level data on knowledge and preg- nancy intendedness. For example, if a clinic wanted to improve its education and support activities, it would need to collect data on clients’ knowledge of available contraceptive methods and their pregnancy intendedness. However, the clinics are not required to collect this information. OPA should explore the feasibility of gathering these data at the time of the CPR. It might be possible to use the CPR and site visit cycle to sample clients’ knowledge and intentions related to their visit agendas at the time of their clinic encounters if the representatives were provided the tools, through expert advice, neces- sary to obtain more information about knowledge, pregnancy intention, and satisfaction with provider interaction. These results then could be reviewed by the site visit team at the time of the CPR. It would be useful if the FPAR could collect more group-specific data in order to detect the effect of small changes in the performance indica- tors based on race and ethnicity, age and gender, level of income, and level of education. For example, this type of data could include unduplicated number of family planning users by age and gender; ethnicity and race; income level; limited English proficiency; primary contraceptive method; and number of gonorrhea, syphilis, HIV, and chlamydia tests. If there were a decline in the percentage of African American users, it would be useful to determine by group-specific data whether the decline was among teenagers, young adults, or older users who might have high STD rates and might not prefer birth control pills as their primary method. This information could help target outreach to be more specific and provide better program direc- tion. It would also be possible to identify any racial disparity in who, for example, obtains a Pap test or has an STD, or high infertility rates based on age, income, or educational level. It would also be useful to know the proportion of unmet need within the Title X target population, both overall and in different regions of the

OCR for page 145
55 COLLECTION OF DATA country. This information, along with pregnancy intendedness information, could help inform Title X outreach planning for different regions. However, the collection of these data would require a new data collection instrument. Any expansion of the CPR to accomplish these goals would need to be sci- entifically valid, conducted in a way that preserves client confidentiality, and required for all grantees. To date, OFP has relied primarily on the NSFG to gather population-level data, and it might be possible for the NSFG to collect these types of data. Other data currently collected under cooperative agreements with OPA specifically examine the population in need of pub- licly funded family planning and how much of that need is met by family planning clinics, whether funded through Title X or not, at the state and county levels. The use of these data should also be explored. Finding 5-4. OFP collects the data needed to affirm that it serves the target populations for the Title X program. Data needed to affirm client knowledge and pregnancy intendedness; resources available to support clients’ childbearing decision making; and the proportion of unmet need within the Title X target popula- tions, both overall and in different regions of the country, are not being collected. However, these data could be gathered by sampling clients served or expanding the NSFG instead of requiring that this information be obtained on every client served. Structure A client’s ability to access quality services and the processes used to offer services are influenced by public policy; funding; staff training and availability; and facility factors such as location, hours of operation, com- fort, and privacy. Strengths FPAR. To ensure that there is adequate and appropriate staffing, the FPAR documents the number of full-time equivalents, categorized as medical versus other clinical service providers and as nonclinical service providers. Data on the number of family planning encounters are also required. Together, these data provide the needed ratio of staffing to patient encounters. The FPAR also collects data on the sources of revenue for each Title X clinic so that federal grants, payment for services, and other sources of revenue are itemized. These data provide an opportunity to (1) calculate and compare the staff load within and across clinics, and (2) construct a cross-sectional and longitudinal profile of revenue sources and track how required service changes or expansions relate to gaps

OCR for page 145
60 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM would not be intended as an outcome measure but more as a formative evaluation indicator to improve the quality of service delivery. As noted earlier, OFP identified three performance measures through the PART process: (1) increasing family planning services to low-income clients to decrease the number of unintended pregnancies, (2) increasing screening of females aged 15–24 for chlamydia infection to reduce infer- tility, and (3) increasing Pap tests to reduce invasive cervical cancer. It is important to acknowledge the difficulty of connecting process indicators with outcomes. Nevertheless, the service performance indicators associated with these outcomes are valuable antecedents as quality improvement tools in themselves. To address quality improvement and management issues, OFP could establish benchmarks or standards for each of its priority outcome mea- sures. Table 5-1 provides examples of service use performance indicators for the Title X priorities using benchmarks from the Performance Measurement System (PMS) developed by the Family Planning Councils of America, Inc. (FPCA) and partially funded by OFP. OFP could closely examine the FPCA PMS as well as other potential benchmarks and related outcome measures with the assistance of an external panel of experts. Relevant Healthcare Employer Data and Information Set (HEDIS) and Centers for Disease Con- trol and Prevention (CDC) standards and goals could be used to reevaluate the benchmarks against which service quality can be measured. By using longitudinal within-clinic return visit data, clinics (especially those with electronic systems) could begin providing a broader picture of service per- formance quality directly linked to OFP goals. As an example, the goal of reducing infertility through chlamydia screening offers an opportunity for specific antecedent benchmarks against which service provision can be evaluated. According to the 2006 FPAR, 56 percent of all Title X female clients were aged 15–24; of those female clients 24 years and younger, only 51 percent were tested for chlamydia (RTI International, 2008), although HEDIS recommendations require that 100 percent of sexually active women in this age group be screened. To achieve the goal of reducing infertility among women, incremental targets for reaching national standards for rates of chlamydia testing and screen- ing among women aged 15–24 could be established. Rates for follow-up with treatment and for remaining negative at a follow-up visit also need to be tracked against benchmarks based on national standards (recognizing that whether one is negative on follow-up depends on both treatment and intervening exposure). Since some older women and men may be tested for chlamydia, OFP could consider how these groups fit into the Title X pri- orities. OFP might consider altering this goal to focus on the reduction of STDs among the groups most likely to be at risk, since the primary goal is to achieve reductions in infertility. Data could also be added to determine

OCR for page 145
6 COLLECTION OF DATA what percentage of users are treated for STDs. As STDs are a reportable disease, local and state health departments have the results of positive STD tests and treatment. Family planning agencies could consider linking through information technology to local and state health departments for this information. As part of this process, expert consultation will be needed to determine whether the performance measures themselves should be modified. OFP’s goal of reducing cervical cancer through Pap tests deserves further review by an expert panel in light of HEDIS and CDC recommendations (see also Chapter 3). Guidelines for Pap tests were revised in 2002 and now state that a Pap test is not indicated until a woman has been sexually active for 3 years or reaches age 21, whichever occurs first. Guidelines for manage- ment of abnormal Pap tests were revised in 2007. Follow-up of abnormal Pap tests for adolescents in particular changed significantly. Unless there is a very abnormal test (i.e., high-grade lesion or worse), the follow-up should be at 1 year. Both of these changes have led to a decrease in Pap tests required, yet a gap continues to exist between program requirements and recommended practice. OFP could use an expert committee of clinicians and scientists to examine its primary approach to reducing cervical cancer. A goal of early immunization with the human papillomavirus vaccine for Title X female clients up to age 26 and age-appropriate Pap screening for other female clients should be considered. This could be accomplished in conjunction with the Vaccines for Children program. With the input of an expert panel, benchmarks for the immunizations and follow-up could be established, similar to those for chlamydia testing, and parallel national goals could be incorporated. With respect to the possible addition of an outcome related to HIV/AIDS, in 2006 CDC revised its guidelines to recommend that all individuals aged 13–64 in health care settings undergo HIV screening. HIV-positive status has crucial implications for pregnancy planning, as well as early intervention for the disease itself. According to the 2006 FPAR, Title X providers performed 1.3 HIV tests for every 10 family planning users. This is a measurable item that should have greater importance in the Title X program. OFP could therefore consider adding a goal regarding HIV testing and include this goal in its discussions of measures and bench- marks for service performance quality improvement. OFP could evaluate quality of care using national guidelines and bench- marks. Particular attention should be paid to how guidelines vary with the age of the client. For example, while OFP currently requires that adoles- cents receive breast examinations, the U.S. Preventive Services Task Force has concluded that the evidence is insufficient to recommend for or against routine clinical breast exams alone to screen for breast cancer, especially among the young (USPSTF, 2003). According to the National Breast and

OCR for page 145
62 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM Cervical Cancer Early Detection Program, 85 percent of breast cancers are identified in women over age 50. Mammograms detect 90 percent of breast cancers; the sensitivity of a clinical breast exam is 40–69 percent. Because the majority of Title X users are in an age group at low risk of breast cancer, mandating clinical breast examinations is of limited benefit and may deter some young women from seeking care. Data on this service would therefore no longer be collected if the Title X care guideline changed on the basis of scientific input. An important question is whether closing these data collection gaps would add to the burden on clinics. As more Title X clinic sites shift to elec- tronic health records, they will be able to perform longitudinal data analysis when provided with appropriate software and training. FPCA reports that with software and training, eight beta sites found the process relatively easy to implement and valued the additional data (Testimony by Dorothy Mann, May 19, 2008). The power of this approach is that it encourages each clinic to assess whether it is meeting national benchmarks set by OFP in keeping with HEDIS, CDC, and other federal agencies, thereby closing an important quality improvement feedback loop at the clinic level. Further, it becomes possible to analyze these data nationally and at the clinic level in terms of the demographics of the clients served. The rates at which low-income clients meet the benchmarks could be a particular focus of the within- and across-clinic analysis of outcome measures. Finding 5-8. The three core measures identified by OFP through the PART process are insufficient for successfully implementing and managing the Title X program or assessing outcomes. Outcomes An important product of OMB’s evaluation of OFP was that OFP identified the above three outcomes for evaluating the impact of Title X on population health. The committee recognizes both the difficulty and importance of defining and measuring these outcomes. For this assess- ment, the committee placed the greatest emphasis on the goal of increasing family planning services to low-income clients to decrease the number of unintended pregnancies since this is the predominant reason for visiting a family planning clinic. As discussed above, the committee suggests that OFP revisit the outcomes selected for evaluating the program’s impact on population health.

OCR for page 145
6 COLLECTION OF DATA Decreasing Unintended Pregnancies While OFP can document the extent to which family planning services are provided to low-income clients, an important but much more difficult task is estimating the decrease in the number of unintended pregnancies associated with those services. As noted above, OFP estimates the decrease in the number of unintended pregnancies using a methodology originally developed in 1977 (Jaffe and Cutright, 1977). This methodology has evolved over time as better behavioral and utilization data at both the national and clinic levels have become available. The model estimates the number of unintended pregnancies by examining the current use of contraceptive methods by women visiting Title X clinics, the failure rates associated with those methods, and estimates of changes in contraception practices if Title X clinics were to close. A number of different estimates of the increase in unintended pregnancies are produced (five in the most recent paper by Frost et al., 2008b), depending on the assumptions made about how contraceptive behavior would change (including “would give up prescription contracep- tion altogether” and “using contraception in the same way as comparable women who do not attend Title X clinics”).1 OPA bases its estimates on the average of the four most realistic scenarios. While the committee agrees that this approach is a reasonable one for estimating the number of unintended pregnancies, it recommends that OFP provide data on the particular assump- tions used to make these estimates. In addition, techniques should be devel- oped to increase the robustness of these estimates and their sensitivity to the quality of the services provided. Three specific improvements in methodol- ogy and data collection would enhance OFP’s understanding of the effective- ness and cost-effectiveness of Title X services: • Estimates should rely on program-specific data on contraceptive methods dispensed and be sensitive to the types and quantity of methods used in Title X clinics rather than on the distribution found in the NSFG. Use of data on the provision of contraceptives by Title X programs would allow the analyses to reflect changes in methods dispensed, quantities dispensed, and method continuation. The committee anticipates that more sensitive evaluation methods would encourage quality improvement, as well as increases in clients served. This approach has been used in evaluations of the California Medicaid Waiver Program (Foster et al., 2004, 2006). • There is room for improvement in modeling the absence of the Title X program. Research on contraceptive use in the absence 1 The Frost et al. paper actually models the impact of publicly funding family planning clinic services, not Title X clinics specifically. However, it is transferable to an examination of Title X clinics only.

OCR for page 145
6 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM of public funding could be improved, for example, by examining method use prior to program inception among new Title X clients or by asking a sample of Title X clients what they would do if subsidized services were unavailable. This information could be included in sensitivity analyses. • Cost savings should be adjusted to differentiate between costs that were entirely prevented through use of contraceptives provided by Title X clinics and those that were merely delayed. Although sig- nificant research would be needed to yield reliable estimates, failure to make this adjustment results in overestimation of costs averted (Amaral et al., 2007). Finally, given the importance of the problem, multiple models and approaches for estimating the number of unintended pregnancies averted through Title X would be useful to explore and compare. Strategic Use of OPA/OFP Research Funding to Demonstrate Outcomes Findings from Title X–funded research are disseminated in a number of ways. Summaries of the research are posted on the OPA website; research grant recipients make formal presentations at the biennial Title X national grantee meetings and other national conferences/meetings; and researchers publish their results in peer-reviewed journals (see Appendix I for a sample of peer-reviewed publications resulting from OPA-funded research). It is noteworthy that in the almost 40 years that Title X’s research pro- gram has been in place, too little remains known about how best to promote and encourage contraceptive use among both men and women. In a recent review, Kirby (2008) found that there is a significant lack of research and evaluation on interventions to promote the use of family planning methods, a lack that severely limits the nation’s ability to help couples both plan for pregnancy and prevent unintended pregnancy. Given the nation’s high rate of unintended pregnancy, this knowledge gap is particularly troubling. An important resource for OFP is the Title X research program’s ability to target relevant issues regarding both outcomes and service delivery. OPA and OFP should carefully evaluate the currently funded research program to ensure that it frames the mission and use of targeted research effectively. The goal is to fund research on concerns, issues, and interventions whose results will have value for subsequent decisions and dissemination among the Title X network. For example, OFP may want to consider targeting research to evaluate program initiatives it has piloted to determine whether wider dissemination of those initiatives is merited. Part of the Title X research effort should involve collaborating with other federal agencies to determine unmet need among the Title X target

OCR for page 145
65 COLLECTION OF DATA population. Addressing this question will require additional data sets on nonusers of Title X clinics, describing family planning needs and indicators for target populations in the region. OFP should apply some of its research funds to using data available through other existing national databases more effectively. It should study the value of the current items available from the NSFG and evaluate the merits of adding other items if needed. Other sys- tematic national population surveys should be examined for the potential to augment the FPAR outcome evaluation. The more directly items collected by these additional national surveys can be tied to the priority outcomes for Title X, the greater the opportunity for comparisons will be. The committee also suggests that OFP consider regional evaluations whereby data from a variety of sources can be compared to identify the extent to which Title X is contributing to key outcomes and whether new initiatives appear to be helpful. Projects that take advantage of claims data from Medicaid and other third-party payers in some counties and regions offer unique opportunities for comparison with data collected on Title X clients. The 2006 FPAR data suggest additional analyses and probes to help evaluate the program’s impact on preventing unintended pregnancy. Accord- ing to these data, 85 percent of women attending Title X clinics were using a birth control method. The percentages of use by method were as follows: • Oral contraceptives, 39 percent • Condoms, 16 percent • Injectable, 12 percent • Patch, 4 percent • IUD, 2 percent • Sterilization, 2 percent • Vaginal ring, 2 percent • Abstinence, 1 percent Each of these methods differs in its profile of discontinuation and failure rates. For example, oral contraceptives have a probability of failure rate of 8.7 percent of users at 1 year, while the every 3 months injectable (Depo-Provera®) has a failure rate of 6.7 percent of users at 1 year. These failure rates are highest among those under age 30, especially low-income teenagers. Given that long-acting methods such as the IUD or the implant are more effective and cost-effective, it would be useful to track not only the availability of these methods at each family planning clinic visit, but also their prescription rate at the visit. Lastly, the 2006 FPAR reports that 15 percent of women using Title X services were not using birth control, were pregnant, or wanted to be. The FPAR provides very little information about this group. To help evaluate

OCR for page 145
66 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM OFP’s progress in helping women achieve their family planning goals, it would be useful to understand more about whether the majority of these women were planning to become pregnant and what role Title X played in helping them with the spacing of subsequent pregnancies. CONCLUSIONS AND RECOMMENDATIONS The committee’s comparison of the data needed to monitor and evalu- ate the Title X program against the data actually collected supports the following conclusions: The program does not collect all the data needed to fully monitor the program and evaluate its impact. A comprehensive framework for approaching program evalua- tion could ensure that all major aspects of the program are evaluated and the needs of clients are being met. Gathering these data will require innovative approaches―and new funding―to minimize the burden on providers. The following recommendations are intended to help OFP strengthen its ability to meet its goals through improved data collection. These recom- mendations are based on recent literature reviews and reports on quality in the provision of family planning services (Sonenstein, 2006; Becker et al., 2007), papers commissioned by the committee (Gregory, 2009; The Lewin Group, 2009 [see Appendixes K and J, respectively]), previous reports com- missioned by OFP (Sonenstein et al., 2004; RTI International, 2005a,b), the committee’s site visits (see Appendix F), and testimony provided to the com- mittee during public workshops. The challenge is to move an evaluation agenda and process forward without imposing an undue burden on clinic providers, clients, and regional administration. The committee therefore recommends that a formal planning process be undertaken by OFP. Recommendation 5-1: Fund and use a comprehensive framework to evaluate the Title X program. OFP should develop, fund, and use a comprehensive framework to evaluate the Title X program. The use of such a framework would allow OFP to evaluate the pro- gram on the full continuum from clinic performance and quality, to clinic management, to program outcomes. It would also help in identifying the types of data needed for evaluation purposes. The development and use of a comprehensive framework to evaluate Title X would make it possible to explore the interactions among various

OCR for page 145
67 COLLECTION OF DATA contributing factors and outcomes, as well as the completeness of different measures for each set of factors. It is important to providing funding for data collection, analysis, and use rather than diluting service dollars. Recommendation 5-2: Examine the data elements of the Family Planning Annual Report (FPAR). When revising the Program Guidelines (see Recommendation -7), OFP should review and clarify data elements contained in the FPAR and where possible and useful, eliminate those that are unnecessary, particularly if additional elements are needed. Recommendation 5-3: Collect additional data. To help fill gaps in the Title X program’s data collection systems, OFP should collect additional data in the areas of client needs, structure, process, and outcomes for use in evaluating the program’s progress and its effec- tiveness in achieving its goals. Specifically, OFP should: Collect additional data on client characteristics. Additional • data sources, such as the Comprehensive Program Reviews (CPRs), should be used to supplement the FPAR data on client characteristics—for example, to obtain data on clients’ knowl- edge about available contraceptive methods and pregnancy intentions. Collect data on system characteristics. Additional data are • needed on such system characteristics as the availability of interpreters to meet the needs of clients with limited English proficiency. Collect data on the process of care. • — These data should include patients’ perceptions of care. With expert consultation, selected CPR site visits could be structured to sample a limited number of clients for the purpose of obtaining generalizable results. — With expert advice, OFP should examine the three core outcome measures identified for the PART process in rela- tion to evidence-based guidelines and national health pri- orities. After determining the most appropriate measures, OFP should develop related performance metrics for clinic service to establish quality improvement standards. Conduct research to assess program outcomes. OFP should • expand research aimed at evaluating program outcomes, such as the impact of the program on pregnancy planning and inten- tion, decreased infertility, outreach to those in need of services, and the prevention of unintended pregnancy.

OCR for page 145
6 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM Recommendation 5-4: Examine evaluation tools for outreach and education. To assist ongoing quality improvement and effective expansion of community outreach and education, OFP should work with grantees to develop and refine evaluation tools for out- reach and education that can be applied easily by delegates. It is important to acknowledge that in many ways, the outreach and education responsibilities (and opportunities) encompassed by Title X often receive less attention than other aspects of the program because of the pressing needs of the clinical care system and the lack of resources. How- ever, the expertise in reproductive health that exists in so many communities across the country needs to be utilized to fuller advantage. The increasing leadership role for OFP and the Title X system of services that the commit- tee recommends (see Chapter 3) extends to outreach and education respon- sibilities (and opportunities). In addition to clinic services, Title X grantees and delegates devote considerable effort to responding to the needs of their communities through outreach and education programs. As discussed in Chapter 4, there is a need to examine the evaluation of these important activities. The current tools used to evaluate the program tend to document quantity and client satisfaction, both of which are important. The question is whether more can be done to extend and evaluate outreach and educa- tion efforts. Can more be done to develop easily implemented strategies and tools for promoting and evaluating the quality of these efforts and their impact on such outcomes as knowledge, attitudes, and intentions related to preventing unintended pregnancy and associated preventive health issues, such as prevention of STDs? Recommendation 5-5: Obtain scientific input on evaluation efforts. OFP should expand its use of scientific expertise to strengthen its evaluation strategies and improve its evaluation research program, and consider expanding its use of national databases to evaluate program impacts. A group of scientific experts should be established for the program, representing knowledge of public health practices and principles; the rel- evant clinical specialties, including primary care, obstetrics and gynecology, and adolescent health; health education, behavioral science, and health services research; epidemiology; and ethnography. This group could serve multiple functions (see also Recommendation 4-7 in Chapter 4) by review- ing standards of care annually, assessing relevant FPAR and CPR measures and samples, and helping to identify an OFP research agenda. That agenda could (1) explore how the Title X program can reduce unintended pregnan- cies more effectively; (2) be linked directly to improving the nation’s family

OCR for page 145
6 COLLECTION OF DATA planning service system; and (3) ensure coordination with other federal research efforts that address common issues of quality assurance, clinical guidelines, and related matters. This group could have systematic input into the quality improvement indicators collected and strategies for increasing clinic feedback and information exchange. Lastly, this group could assist OFP in addressing the recommendations offered in this report. Expert and clinic site consultation would also be helpful in addressing the key question of how OFP should better use the data it currently col- lects. As discussed, OFP relies on different but complementary sources of data, as well as data collection strategies, as part of its national evaluation system. This multiplicity is vital as any one source of data may be unable to provide the level of information necessary—each having its strengths and limitations. The ways in which OFP synthesizes and uses existing data for program planning, including process data and service delivery improvement research funded by OPA, are not fully transparent. As discussed in Chapter 3, grantees expressed concern regarding their perception that new service priorities often appear to be announced without explanation of how they were established or how they relate to an overall strategic plan. Furthermore, there appears to be a gap between requiring new priorities and adapting the existing data collection system to capture and document the program’s success in responding to these priorities. If the data collection system is not incorporating new program priorities, the efforts of grantees to respond to those priorities may not be fully captured. The FPAR was last modified in 2005 (to include new data elements, such as user health insurance coverage, English proficiency, contraceptive use by males, summary Pap [abnormal] and confidential HIV [positive] test results, and disease-specific information on STD screening). While it is unrealistic to modify the FPAR frequently, it may be realistic to require a more limited set of data elements for programs engaged in implementing new priorities to determine whether those priorities are being implemented fully and having the desired impact. Soliciting both scientific and clinic-based input on these questions would be useful. Recommendation 5-6: Communicate evaluation findings. To ensure transparency and broad-based dissemination of information and ultimately to improve care (see Recommendation -), OFP should enhance ongoing feedback and communication with grantees, del- egates, clinics, and others about important evaluation findings and how they can help improve care and track progress toward the achievement of program goals. During the committee’s site visits with Title X clinics, as well in testi- mony at public workshops, several providers expressed a desire for more

OCR for page 145
70 A REVIEW OF THE HHS FAMILY PLANNING PROGRAM feedback and information on the FPAR. The committee recommends that OFP consider a variety of ways to meet this need (see Recommenda- tion 4-8). Feedback on clinic performance and data trends via webpages and web broadcasts could offer additional information to clinics. Equally important, it could help clinic staff understand the significance of their work on collecting FPAR data. CONCLUDING THOUGHTS The committee has identified a variety of ways in which the Title X program could be improved. These include focusing on its core mission; undertaking a strategic planning process with a longer time horizon; imple- menting patient-focused, scientifically based clinical practices; and enhanc- ing evaluation and communication. There is also a need to coordinate requirements of relevant federal agencies with Title X. The committee acknowledges that the current systems used by OFP to collect program data have significant strengths. Recognizing that more resources will be required, the committee believes there are opportunities to utilize data that are already available, as well as to improve the collection of those data in ways that are sensitive to the needs of both grantees and clients while also contributing to quality improvement efforts. Although there is room for improvement, it is important to note that the Title X program has successfully served low-income women and men and adolescents. Despite increasingly limited funds, the dedication of fed- eral agency staff, grantees, delegates, and clinic staff to the goals and clients of the Title X program has made it possible to deliver essential services that have helped individuals, families, communities, and the nation.