APPENDIX A
Methods of the Study
In the spring of 2005, the Institute of Medicine (IOM) was asked by the Health Resources and Services Administration (HRSA) to perform an assessment of the impact of the HRSA Traumatic Brain Injury (TBI) Program focusing on (1) how state systems are working or failing to work in support of individuals with TBI; and (2) how HRSA could improve the program to best serve individuals with TBI and their families. The study focuses on whether the TBI Program has led to an expansion in state systems infrastructure as a precondition for better serving persons with TBI and their families. It is not a technical evaluation of the HRSA TBI Program’s impact on either the delivery of TBI-related services or on person-level outcomes—such an analysis is not feasible given currently available data (as noted in chapters 2 and 3 of this report).
The committee used a qualitative study method to assess the TBI Program’s impact. Qualitative methods are often used in health services research to investigate developing institutions and systems as well as to assess the impact of government programs (Caudle, 1994; Sofaer, 1999; Newcomer and Scheirer, 2001; World Bank Group, 2005). Qualitative data were gathered from a variety of sources and were analyzed for key themes and recurring issues. Primary sources of data included semi-structured interviews with TBI stakeholders in seven states and representatives of selected national organizations (a requirement of the IOM/HRSA agreement), research literature and TBI program materials, and relevant survey data.
The committee’s approach to the selection of the states and stakeholder
interviews in those states is described in this appendix, along with other methods used in the study, including the literature review, committee meetings, and a public workshop.
DATA LIMITATIONS
The data used in this study had important limitations. Comprehensive, standardized analytic information on the Federal TBI Program is not available. The National Association of State Head Injury Administrators (NASHIA) Guide to State Government Brain Injury Policies, Funding and Services is the only source of comprehensive information on the characteristics and activities of state TBI programs (NASHIA, 2005). The guide, although informative, draws primarily from self-reported data from state TBI programs. The data are neither standardized nor audited. States do not use a standard diagnostic definition of TBI nor is there a standard terminology for describing TBI services. One state, for example, may define “personal care” as a single service, while another state uses the same term to describe a range of assisted living or residential supports. Eligibility for state TBI services also varies; for example, some programs target adults only, while other programs also serve children.
As described below, the study used information gathered in interviews with TBI stakeholders in seven states and representatives of selected national organizations. The state-based interview informants were not disinterested participants; they were either state officials of the respective state TBI programs or other individuals identified by the state agencies. These individuals could be motivated by an interest in perpetuating federal funding.
LITERATURE REVIEW
Staff searched the primary scientific literature via Medline and collected TBI program data and relevant materials from HRSA, other federal agencies, and state TBI programs; the TBI Technical Assistance Center (TBI TAC), NASHIA, Brain Injury Associations, federally mandated protection and advocacy (P&A) systems in the states, and others.
COMMITTEE MEETINGS AND PUBLIC WORKSHOP
The committee met eight times by telephone conference and held two in-person meetings to formulate its approach to the study, review the data collected, and develop the report. The first in-person meeting included a 1-day public workshop to gain the perspectives and assessments of experts and key stakeholders. Box A-1 presents the workshop agenda, which in-
cluded presentations from researchers, state TBI program directors, representatives of federal agencies and nonprofit organizations, and individuals with family members with TBI.
IN-DEPTH LOOK AT TBI INITIATIVES IN SEVEN STATES
The committee agreed to study seven state TBI programs to collect qualitative (interview) data and additional data, where it existed, on states’ experiences with the HRSA TBI Program. Over the course of several meetings, as discussed below, the committee formulated a conceptual framework, developed selection criteria and picked the seven states, developed an interview guide for use in the seven states, and identified the respondent pool. It accomplished these activities with the full-time support of a staff consultant, Holly Korda, Ph.D., Principal, Health Systems Research Associates, Chevy Chase, Maryland, whose final report is presented in Appendix E.
Conceptual Framework
The basic premise of the HRSA TBI Program is that with the modest investment of federal funds, states will build necessary infrastructure, create channels of communication between relevant agencies, and integrate the disparate systems that serve the TBI population—while also finding the means to sustain these efforts. Figure A-1 illustrates the linkages between the HRSA TBI Program and the myriad state, federal, local, public, and private organizations that relate to persons with TBI and their families.
Although the intended long-term outcome of the HRSA TBI Program is improved health and quality of life for persons with TBI and their families, the committee recognized, as had the federal Office of Management and Budget’s Program Assessment Rating Tool study before it in 2004, that it was impossible to evaluate the HRSA program on the basis of clinical outcomes data, which were nonstandardized, inadequate, and/or nonexistent. For that reason, the committee focused instead on demonstrable organizational and strategic change associated with the program, such as infrastructure development, education and training of relevant personnel, planning and implementation of new services, and the development of outreach services to persons with TBI and their families.
Selection of the Seven States for an In-Depth Look
A series of tables detailing the characteristics and self-reported accomplishments of the 50 state TBI programs and federally mandated P&A systems in the states appears in Appendix C. The data in these tables were
BOX A-1
|
drawn from materials provided to the committee by state and federal TBI programs, the TBI Technical Assistance Center (TBI TAC), and NASHIA.
The committee found the national data to be very helpful in indicating activities as reported for grant purposes by different states, but it concluded that the data were not sufficiently standardized to form a study population from which statistically representative states could be drawn. The committee noted, in addition, that each state’s experience with the HRSA program was contingent on the prior history and organizational context of TBI services in that state.
After reviewing this information, the committee concluded that the state programs were too diverse, and the study’s resources too limited, to
|
identify a group of seven states that would represent the full range of TBI experiences. Because the number of states was limited by budget considerations, the committee’s criteria for selection focused on identifying seven states with diverse state experiences that would signal major successes and failures, as well as common themes.
Several criteria were used to select the seven states for in-depth study—for example, participation in the HRSA TBI Program, state TBI funding, locus of the lead state agency for TBI, geographical diversity, etc. (For the rationale for each selection criterion, see Table A-1).
Ultimately, the committee asked seven states—Alabama, California, Colorado, Georgia, New Jersey, Ohio, and Washington State—to partici-
TABLE A-1 The Committee’s Criteria for Selecting Seven States for an In-Depth Look
Criterion |
Description |
Rationale |
Participation in the Federal TBI Program |
Type of TBI Program Grant received by the state from the Health Resources and Services Administration (HRSA)—TBI Planning Grant, TBI Implementation Grant, and/or TBI Post-Demonstration Grant)—and duration of the grant received. |
In 2005, 2 states had a TBI Planning Grant; 10, a TBI Implementation Grant; and 0, a Post-Demonstration Grant. Past history: 37 states had previous Planning Grants; 40, Implementation Grants; and 23, Post-Demonstration Grants. Some states appear to have moved swiftly and effectively after receiving TBI Program Grant funding from HRSA; other states had not moved beyond modest planning activities. |
State TBI funding |
TBI trust fund, Medicaid waiver, or dedicated state TBI funding. |
The relative importance of Federal TBI Program Grants to states depends on other state funding sources. States with highly developed TBI programs often have TBI trust funds, TBI-specific or other Medicaid waivers, or other dedicated state funding. Less developed TBI programs draw from general state funds. |
Locus of the lead state agency for TBI |
State agency with lead authority for TBI (appointed by the governor) |
The state agency with lead authority for TBI programs varies considerably and may affect the program’s effectiveness, especially as the need for integration of TBI services becomes increasingly complex. |
Other |
Region, urban/rural, cultural diversity, data systems, and participation in the TBI Model Systems Program.* |
The potential impact of a Federal TBI Program Grant to a state may be compromised or enhanced by any of these factors. |
*The National Institute on Disability and Rehabilitation Research (NIDRR) provides grants to 16 state TBI Model Systems, programs that develop and conduct research on TBI service delivery systems. |
TABLE A-2 Dedicated State Funding for TBI in the Seven States
State |
Dedicated State Funding (in $ millions) |
||
Trust Fund |
Medicaid Waiver |
General Funds |
|
Alabama |
$1.2 |
|
|
California |
$1.1 |
||
Colorado |
$2.5 |
$5.2 |
|
Georgia |
$2.3 |
|
|
New Jersey |
$3.4 |
$14.6 |
|
Ohio |
|
$.346 |
|
Washington |
|
||
SOURCE: TBI TAC, 2005. |
TABLE A-3 Participation in the Federal TBI Program by the Seven States
State |
Federal TBI Program Grant Received |
|||
TBI Planning Grant |
Implementation Grant |
Post-Demonstration Grant |
Protection and Advocacy for TBI (PATBI) Grant* |
|
Alabama |
— |
√ |
√ |
√ |
California |
√ |
— |
— |
√ |
Colorado |
√ |
√ |
√ |
√ |
Georgia |
√ |
√ |
√ |
√ |
New Jersey |
— |
√ |
√ |
√ |
Ohio |
— |
√ |
√ |
√ |
Washington |
√ |
√ |
— |
√ |
*All 50 states receive PATBI Grants. SOURCE: TBI TAC, 2005. |
pate in the interviews (and all agreed).1 These states were selected as the likely best representatives of the considerable state-to-state differences in their preparedness for applying for and using these grants. Tables A-2 to A-5 provide details on selected features of the TBI programs in the seven states.
1 |
See Appendix D for detailed profiles of TBI initiatives in each of the seven sample states, including the states’ HRSA grant histories; resources; services for people with TBI and their families; interorganizational collaboration and coordination; TBI-related data, monitoring, and evaluation; and successes and challenges of the HRSA grant experience. |
TABLE A-4 Locus of the Lead State Agency for TBI in the Seven States
State |
Lead Agency |
Alabama |
Department of Rehabilitative Services |
California |
Department of Mental Health |
Colorado |
Office of Behavioral Health & Housing, Department of Human Services |
Georgia |
Brain and Spinal Injury Trust Fund Commission |
New Jersey |
Division of Disability Services, Department of Human Services |
Ohio |
Rehabilitation Services Commission |
Washington |
Aging and Disability Services Administration, Department of Social and Health Services |
SOURCE: TBI TAC, 2005. |
Interview Guide Developed by the Committee
Appendix B presents the interview guide developed by the committee for the interviews with TBI stakeholders in the seven states. The committee structured the guide so that it could be used for a 45-minute, in-person, or telephone interview. The interview questions were formulated
TABLE A-5 Other Pertinent Characteristics of the Seven States
State |
TBI Data Systems |
||||
Percent Rural Population |
TBI Registry |
Trauma Registry |
CDC TBI Surveillancea |
TBI Model Systemb |
|
Alabama |
44.6 |
√ |
√ |
√ |
√ |
California |
5.6 |
— |
— |
√ |
√ |
Colorado |
15.5 |
— |
√ |
√ |
√ |
Georgia |
28.4 |
√ |
√ |
— |
— |
New Jersey |
5.6 |
— |
√ |
√ |
√ |
Ohio |
22.6 |
— |
√ |
— |
√ |
Washington |
18.0 |
— |
√ |
— |
√ |
aAlabama, California, Colorado, and New Jersey have received Centers for Disease Control and Prevention (CDC) grants for TBI surveillance. Alabama and Colorado participate in other CDC TBI data projects, medical records studies, and a TBI follow-up registry (Colorado only). bThe National Institute on Disability and Rehabilitation Research (NIDRR) provides grants to 16 state TBI Model Systems, programs that develop and conduct research on TBI service delivery systems. SOURCE: TBI TAC, 2005; U.S. 2000 Census Data. |
to learn respondents’ perceptions in eight areas: (1) the background and context for state TBI activities; (2) HRSA Grant Program participation; (3) TBI services and the impact of the grants; (4) TBI program monitoring and evaluation; (5) state TBI data sources and uses; (6) experience with the TBI TAC; (7) experience as a HRSA TBI Program grantee; and (8) expectations for the future. The committee structured the interview guide to be flexible and open-ended, recognizing that there was wide variation in TBI programs, interview participants, and strategies used to develop TBI infrastructure in different states.
Selection of Interview Respondents
For reasons of efficiency, the committee asked the TBI lead agencies in the states to help identify potential respondents representing TBI interests in the state, including the lead state agency for TBI, the independent P&A system, the Brain Injury Association, the state agency responsible for injury prevention, TBI trust funds, Medicaid waiver programs, and persons with TBI or family members.
Telephone or in-person interviews were conducted with one or more representatives from the various state agencies and organizations. In two states, Georgia and Colorado, the interviews were conducted in person and held as part of 2-day site visits by consultant Holly Korda, committee chair Rosemary Stevens, and study director Jill Eden. In addition to conducting the interviews, the IOM group toured and met with officials of two state-of-the-art rehabilitation facilities: the Shepherd Center in Atlanta and Craig Hospital outside of Denver. Figure A-2 shows the final makeup of the respondent pool by organizational affiliation.
Interviews were also conducted with the leaders of the following national organizations: HRSA TBI Program, NASHIA, TBI TAC,2 Brain Injury Association of America, and National Disability Rights Network.3 During the study period, the Brain Injury Association of America and NASHIA conducted an independent, online survey of stakeholders in all 50 states and the District of Columbia; those results too were made available to the committee.
REFERENCES
Caudle SL. 1994.Using qualitative approaches. Handbook of Practical Program Evaluation (eds. Wholey, Hatry, and Newcomer). San Francisco: Jossey-Bass.
NASHIA (National Association of State Head Injury Administrators). 2005. Guide to State Government Brain Injury Policies, Funding and Services. [Online] Available: http://www.nashia.org [accessed 10/04/2005].
Newcomer KE, Scheirer, MA. 2001. Using evaluation to support performance management: A guide for federal executives. Arlington, VA: PricewaterhouseCoopers Endowment for the Business of Government; Jan.
Sofaer S. 1999. Qualitative methods: What are they and why use them? Health Services Research. 34(5): 1101–1116.
Traumatic Brain Injury Technical Assistance Center (TBI TAC). 2005. Pathway for Systems Change: Benchmarks. Bethesda, MD: TBI TAC.
World Bank Group. 2005. Understanding Impact Evaluation [Web Page]. [Online[ [accessed 12/15/2005]. Available at: http://www.worldbank.org/poverty/impact/index.htm.