APPENDIX A
Methods of the Study

In the spring of 2005, the Institute of Medicine (IOM) was asked by the Health Resources and Services Administration (HRSA) to perform an assessment of the impact of the HRSA Traumatic Brain Injury (TBI) Program focusing on (1) how state systems are working or failing to work in support of individuals with TBI; and (2) how HRSA could improve the program to best serve individuals with TBI and their families. The study focuses on whether the TBI Program has led to an expansion in state systems infrastructure as a precondition for better serving persons with TBI and their families. It is not a technical evaluation of the HRSA TBI Program’s impact on either the delivery of TBI-related services or on person-level outcomes—such an analysis is not feasible given currently available data (as noted in chapters 2 and 3 of this report).

The committee used a qualitative study method to assess the TBI Program’s impact. Qualitative methods are often used in health services research to investigate developing institutions and systems as well as to assess the impact of government programs (Caudle, 1994; Sofaer, 1999; Newcomer and Scheirer, 2001; World Bank Group, 2005). Qualitative data were gathered from a variety of sources and were analyzed for key themes and recurring issues. Primary sources of data included semi-structured interviews with TBI stakeholders in seven states and representatives of selected national organizations (a requirement of the IOM/HRSA agreement), research literature and TBI program materials, and relevant survey data.

The committee’s approach to the selection of the states and stakeholder



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program APPENDIX A Methods of the Study In the spring of 2005, the Institute of Medicine (IOM) was asked by the Health Resources and Services Administration (HRSA) to perform an assessment of the impact of the HRSA Traumatic Brain Injury (TBI) Program focusing on (1) how state systems are working or failing to work in support of individuals with TBI; and (2) how HRSA could improve the program to best serve individuals with TBI and their families. The study focuses on whether the TBI Program has led to an expansion in state systems infrastructure as a precondition for better serving persons with TBI and their families. It is not a technical evaluation of the HRSA TBI Program’s impact on either the delivery of TBI-related services or on person-level outcomes—such an analysis is not feasible given currently available data (as noted in chapters 2 and 3 of this report). The committee used a qualitative study method to assess the TBI Program’s impact. Qualitative methods are often used in health services research to investigate developing institutions and systems as well as to assess the impact of government programs (Caudle, 1994; Sofaer, 1999; Newcomer and Scheirer, 2001; World Bank Group, 2005). Qualitative data were gathered from a variety of sources and were analyzed for key themes and recurring issues. Primary sources of data included semi-structured interviews with TBI stakeholders in seven states and representatives of selected national organizations (a requirement of the IOM/HRSA agreement), research literature and TBI program materials, and relevant survey data. The committee’s approach to the selection of the states and stakeholder

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program interviews in those states is described in this appendix, along with other methods used in the study, including the literature review, committee meetings, and a public workshop. DATA LIMITATIONS The data used in this study had important limitations. Comprehensive, standardized analytic information on the Federal TBI Program is not available. The National Association of State Head Injury Administrators (NASHIA) Guide to State Government Brain Injury Policies, Funding and Services is the only source of comprehensive information on the characteristics and activities of state TBI programs (NASHIA, 2005). The guide, although informative, draws primarily from self-reported data from state TBI programs. The data are neither standardized nor audited. States do not use a standard diagnostic definition of TBI nor is there a standard terminology for describing TBI services. One state, for example, may define “personal care” as a single service, while another state uses the same term to describe a range of assisted living or residential supports. Eligibility for state TBI services also varies; for example, some programs target adults only, while other programs also serve children. As described below, the study used information gathered in interviews with TBI stakeholders in seven states and representatives of selected national organizations. The state-based interview informants were not disinterested participants; they were either state officials of the respective state TBI programs or other individuals identified by the state agencies. These individuals could be motivated by an interest in perpetuating federal funding. LITERATURE REVIEW Staff searched the primary scientific literature via Medline and collected TBI program data and relevant materials from HRSA, other federal agencies, and state TBI programs; the TBI Technical Assistance Center (TBI TAC), NASHIA, Brain Injury Associations, federally mandated protection and advocacy (P&A) systems in the states, and others. COMMITTEE MEETINGS AND PUBLIC WORKSHOP The committee met eight times by telephone conference and held two in-person meetings to formulate its approach to the study, review the data collected, and develop the report. The first in-person meeting included a 1-day public workshop to gain the perspectives and assessments of experts and key stakeholders. Box A-1 presents the workshop agenda, which in-

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program cluded presentations from researchers, state TBI program directors, representatives of federal agencies and nonprofit organizations, and individuals with family members with TBI. IN-DEPTH LOOK AT TBI INITIATIVES IN SEVEN STATES The committee agreed to study seven state TBI programs to collect qualitative (interview) data and additional data, where it existed, on states’ experiences with the HRSA TBI Program. Over the course of several meetings, as discussed below, the committee formulated a conceptual framework, developed selection criteria and picked the seven states, developed an interview guide for use in the seven states, and identified the respondent pool. It accomplished these activities with the full-time support of a staff consultant, Holly Korda, Ph.D., Principal, Health Systems Research Associates, Chevy Chase, Maryland, whose final report is presented in Appendix E. Conceptual Framework The basic premise of the HRSA TBI Program is that with the modest investment of federal funds, states will build necessary infrastructure, create channels of communication between relevant agencies, and integrate the disparate systems that serve the TBI population—while also finding the means to sustain these efforts. Figure A-1 illustrates the linkages between the HRSA TBI Program and the myriad state, federal, local, public, and private organizations that relate to persons with TBI and their families. Although the intended long-term outcome of the HRSA TBI Program is improved health and quality of life for persons with TBI and their families, the committee recognized, as had the federal Office of Management and Budget’s Program Assessment Rating Tool study before it in 2004, that it was impossible to evaluate the HRSA program on the basis of clinical outcomes data, which were nonstandardized, inadequate, and/or nonexistent. For that reason, the committee focused instead on demonstrable organizational and strategic change associated with the program, such as infrastructure development, education and training of relevant personnel, planning and implementation of new services, and the development of outreach services to persons with TBI and their families. Selection of the Seven States for an In-Depth Look A series of tables detailing the characteristics and self-reported accomplishments of the 50 state TBI programs and federally mandated P&A systems in the states appears in Appendix C. The data in these tables were

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program BOX A-1 Agenda for IOM Workshop on Traumatic Brain Injury July 18, 2005 8:30 Welcome and Introduction—Rosemary Stevens, Chair, IOM Committee on Traumatic Brain Injury 8:40 The Epidemiology of TBI—Jean Langlois, Senior Epidemiologist, National Center for Injury Prevention and Control, Centers for Disease Control and Prevention 9:15 Background and History of the HRSA TBI Program Legislative History and Challenges for the States—Susan Vaughn, Director of Public Policy, National Association of State Head Injury Administrators (NASHIA) History of the HRSA TBI Program—Jane Martin-Heppel, Director, HRSA TBI Program 10:00 Break 10:15 Video 10:30 TBI Resource Facilitation: Clinical, Psycho-Social, Services Challenges Many Services, Many Types of Care—Susan Connors, CEO and President, Brain Injury Association (BIA) of America TBI Needs and Challenges among Native Americans—Alta Bruce, President, Indigenous People’s BIA and Injury Prevention Specialist, Indian Health Service 11:15 TBI Model Systems Program—Ruth Brannon, Associate Director, Division of Research, National Institute on Disability and Rehabilitation Research drawn from materials provided to the committee by state and federal TBI programs, the TBI Technical Assistance Center (TBI TAC), and NASHIA. The committee found the national data to be very helpful in indicating activities as reported for grant purposes by different states, but it concluded that the data were not sufficiently standardized to form a study population from which statistically representative states could be drawn. The committee noted, in addition, that each state’s experience with the HRSA program was contingent on the prior history and organizational context of TBI services in that state. After reviewing this information, the committee concluded that the state programs were too diverse, and the study’s resources too limited, to

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program 11:45 Lunch 12:30 HRSA TBI Technical Assistance Center (TBI TAC) Role of TBI TAC and State Technical Assistance—Ken Currier, Acting Executive Director, NASHIA, and Director, TBI TAC Demonstration of TBI Collaborative Space, Benchmarks, Other Tools—Rebecca Zeltinger, TBITAC Director of Knowledge Exchange 1:15 TBI State Grantees and Stakeholders Speak on the Program History of the Disability Legal Advocacy Network—Curt Decker, Executive Director, National Association of Protection and Advocacy Consumer Perspectives—Jan Brown, Member, TBI Technical Assistance Center Steering Committee and Carolyn Rocchio, Family member, Florida Brain Injury Association 2:00 Break 2:15 Leveraging HRSA Funds to Improve Access to TBI Services and Systems National Overview—Bill Ditto, Director, New Jersey Division of Disability Services Empowering People with TBI and Legislative Change—Bil Schmidt, New Mexico TBI Program Director and Beverly James, Participant, NM “Empowerment” Program Using State Data for TBI Systems Change—Manfred Tatzmann, Director, Michigan TBI Program Moving from HRSA Program Funding to Sustainability—Augusta Cash, Alabama TBI Program Director 4:15 Discussion: What Have We Learned? Where Do We Go from Here?—Rosemary Stevens 4:45 Adjourn identify a group of seven states that would represent the full range of TBI experiences. Because the number of states was limited by budget considerations, the committee’s criteria for selection focused on identifying seven states with diverse state experiences that would signal major successes and failures, as well as common themes. Several criteria were used to select the seven states for in-depth study—for example, participation in the HRSA TBI Program, state TBI funding, locus of the lead state agency for TBI, geographical diversity, etc. (For the rationale for each selection criterion, see Table A-1). Ultimately, the committee asked seven states—Alabama, California, Colorado, Georgia, New Jersey, Ohio, and Washington State—to partici-

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program FIGURE A-1 Linkages between the Federal TBI Program and other entities.

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program TABLE A-1 The Committee’s Criteria for Selecting Seven States for an In-Depth Look Criterion Description Rationale Participation in the Federal TBI Program Type of TBI Program Grant received by the state from the Health Resources and Services Administration (HRSA)—TBI Planning Grant, TBI Implementation Grant, and/or TBI Post-Demonstration Grant)—and duration of the grant received. In 2005, 2 states had a TBI Planning Grant; 10, a TBI Implementation Grant; and 0, a Post-Demonstration Grant. Past history: 37 states had previous Planning Grants; 40, Implementation Grants; and 23, Post-Demonstration Grants. Some states appear to have moved swiftly and effectively after receiving TBI Program Grant funding from HRSA; other states had not moved beyond modest planning activities. State TBI funding TBI trust fund, Medicaid waiver, or dedicated state TBI funding. The relative importance of Federal TBI Program Grants to states depends on other state funding sources. States with highly developed TBI programs often have TBI trust funds, TBI-specific or other Medicaid waivers, or other dedicated state funding. Less developed TBI programs draw from general state funds. Locus of the lead state agency for TBI State agency with lead authority for TBI (appointed by the governor) The state agency with lead authority for TBI programs varies considerably and may affect the program’s effectiveness, especially as the need for integration of TBI services becomes increasingly complex. Other Region, urban/rural, cultural diversity, data systems, and participation in the TBI Model Systems Program.* The potential impact of a Federal TBI Program Grant to a state may be compromised or enhanced by any of these factors. *The National Institute on Disability and Rehabilitation Research (NIDRR) provides grants to 16 state TBI Model Systems, programs that develop and conduct research on TBI service delivery systems.

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program TABLE A-2 Dedicated State Funding for TBI in the Seven States State Dedicated State Funding (in $ millions) Trust Fund Medicaid Waiver General Funds Alabama $1.2   California $1.1 Colorado $2.5 $5.2   Georgia $2.3   New Jersey $3.4 $14.6   Ohio   $.346 Washington   SOURCE: TBI TAC, 2005. TABLE A-3 Participation in the Federal TBI Program by the Seven States State Federal TBI Program Grant Received TBI Planning Grant Implementation Grant Post-Demonstration Grant Protection and Advocacy for TBI (PATBI) Grant* Alabama — √ √ √ California √ — — √ Colorado √ √ √ √ Georgia √ √ √ √ New Jersey — √ √ √ Ohio — √ √ √ Washington √ √ — √ *All 50 states receive PATBI Grants. SOURCE: TBI TAC, 2005. pate in the interviews (and all agreed).1 These states were selected as the likely best representatives of the considerable state-to-state differences in their preparedness for applying for and using these grants. Tables A-2 to A-5 provide details on selected features of the TBI programs in the seven states. 1   See Appendix D for detailed profiles of TBI initiatives in each of the seven sample states, including the states’ HRSA grant histories; resources; services for people with TBI and their families; interorganizational collaboration and coordination; TBI-related data, monitoring, and evaluation; and successes and challenges of the HRSA grant experience.

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program TABLE A-4 Locus of the Lead State Agency for TBI in the Seven States State Lead Agency Alabama Department of Rehabilitative Services California Department of Mental Health Colorado Office of Behavioral Health & Housing, Department of Human Services Georgia Brain and Spinal Injury Trust Fund Commission New Jersey Division of Disability Services, Department of Human Services Ohio Rehabilitation Services Commission Washington Aging and Disability Services Administration, Department of Social and Health Services SOURCE: TBI TAC, 2005. Interview Guide Developed by the Committee Appendix B presents the interview guide developed by the committee for the interviews with TBI stakeholders in the seven states. The committee structured the guide so that it could be used for a 45-minute, in-person, or telephone interview. The interview questions were formulated TABLE A-5 Other Pertinent Characteristics of the Seven States State TBI Data Systems Percent Rural Population TBI Registry Trauma Registry CDC TBI Surveillancea TBI Model Systemb Alabama 44.6 √ √ √ √ California 5.6 — — √ √ Colorado 15.5 — √ √ √ Georgia 28.4 √ √ — — New Jersey 5.6 — √ √ √ Ohio 22.6 — √ — √ Washington 18.0 — √ — √ aAlabama, California, Colorado, and New Jersey have received Centers for Disease Control and Prevention (CDC) grants for TBI surveillance. Alabama and Colorado participate in other CDC TBI data projects, medical records studies, and a TBI follow-up registry (Colorado only). bThe National Institute on Disability and Rehabilitation Research (NIDRR) provides grants to 16 state TBI Model Systems, programs that develop and conduct research on TBI service delivery systems. SOURCE: TBI TAC, 2005; U.S. 2000 Census Data.

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program to learn respondents’ perceptions in eight areas: (1) the background and context for state TBI activities; (2) HRSA Grant Program participation; (3) TBI services and the impact of the grants; (4) TBI program monitoring and evaluation; (5) state TBI data sources and uses; (6) experience with the TBI TAC; (7) experience as a HRSA TBI Program grantee; and (8) expectations for the future. The committee structured the interview guide to be flexible and open-ended, recognizing that there was wide variation in TBI programs, interview participants, and strategies used to develop TBI infrastructure in different states. Selection of Interview Respondents For reasons of efficiency, the committee asked the TBI lead agencies in the states to help identify potential respondents representing TBI interests in the state, including the lead state agency for TBI, the independent P&A system, the Brain Injury Association, the state agency responsible for injury prevention, TBI trust funds, Medicaid waiver programs, and persons with TBI or family members. FIGURE A-2 Interview respondent pool by organizational affiliation.

OCR for page 105
Evaluating the HRSA Traumatic Brain Injury Program Telephone or in-person interviews were conducted with one or more representatives from the various state agencies and organizations. In two states, Georgia and Colorado, the interviews were conducted in person and held as part of 2-day site visits by consultant Holly Korda, committee chair Rosemary Stevens, and study director Jill Eden. In addition to conducting the interviews, the IOM group toured and met with officials of two state-of-the-art rehabilitation facilities: the Shepherd Center in Atlanta and Craig Hospital outside of Denver. Figure A-2 shows the final makeup of the respondent pool by organizational affiliation. Interviews were also conducted with the leaders of the following national organizations: HRSA TBI Program, NASHIA, TBI TAC,2 Brain Injury Association of America, and National Disability Rights Network.3 During the study period, the Brain Injury Association of America and NASHIA conducted an independent, online survey of stakeholders in all 50 states and the District of Columbia; those results too were made available to the committee. REFERENCES Caudle SL. 1994.Using qualitative approaches. Handbook of Practical Program Evaluation (eds. Wholey, Hatry, and Newcomer). San Francisco: Jossey-Bass. NASHIA (National Association of State Head Injury Administrators). 2005. Guide to State Government Brain Injury Policies, Funding and Services. [Online] Available: http://www.nashia.org [accessed 10/04/2005]. Newcomer KE, Scheirer, MA. 2001. Using evaluation to support performance management: A guide for federal executives. Arlington, VA: PricewaterhouseCoopers Endowment for the Business of Government; Jan. Sofaer S. 1999. Qualitative methods: What are they and why use them? Health Services Research. 34(5): 1101–1116. Traumatic Brain Injury Technical Assistance Center (TBI TAC). 2005. Pathway for Systems Change: Benchmarks. Bethesda, MD: TBI TAC. World Bank Group. 2005. Understanding Impact Evaluation [Web Page]. [Online[ [accessed 12/15/2005]. Available at: http://www.worldbank.org/poverty/impact/index.htm. 2   NASHIA and TBI TAC are led by the same individual. 3   Formerly the National Association of Protection and Advocacy Systems, Inc.