National Academies Press: OpenBook

Evaluating the HRSA Traumatic Brain Injury Program (2006)

Chapter: Appendix A: Methods of the Study

« Previous: 4 Assessment of the HRSA TBI Program
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

APPENDIX A
Methods of the Study

In the spring of 2005, the Institute of Medicine (IOM) was asked by the Health Resources and Services Administration (HRSA) to perform an assessment of the impact of the HRSA Traumatic Brain Injury (TBI) Program focusing on (1) how state systems are working or failing to work in support of individuals with TBI; and (2) how HRSA could improve the program to best serve individuals with TBI and their families. The study focuses on whether the TBI Program has led to an expansion in state systems infrastructure as a precondition for better serving persons with TBI and their families. It is not a technical evaluation of the HRSA TBI Program’s impact on either the delivery of TBI-related services or on person-level outcomes—such an analysis is not feasible given currently available data (as noted in chapters 2 and 3 of this report).

The committee used a qualitative study method to assess the TBI Program’s impact. Qualitative methods are often used in health services research to investigate developing institutions and systems as well as to assess the impact of government programs (Caudle, 1994; Sofaer, 1999; Newcomer and Scheirer, 2001; World Bank Group, 2005). Qualitative data were gathered from a variety of sources and were analyzed for key themes and recurring issues. Primary sources of data included semi-structured interviews with TBI stakeholders in seven states and representatives of selected national organizations (a requirement of the IOM/HRSA agreement), research literature and TBI program materials, and relevant survey data.

The committee’s approach to the selection of the states and stakeholder

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

interviews in those states is described in this appendix, along with other methods used in the study, including the literature review, committee meetings, and a public workshop.

DATA LIMITATIONS

The data used in this study had important limitations. Comprehensive, standardized analytic information on the Federal TBI Program is not available. The National Association of State Head Injury Administrators (NASHIA) Guide to State Government Brain Injury Policies, Funding and Services is the only source of comprehensive information on the characteristics and activities of state TBI programs (NASHIA, 2005). The guide, although informative, draws primarily from self-reported data from state TBI programs. The data are neither standardized nor audited. States do not use a standard diagnostic definition of TBI nor is there a standard terminology for describing TBI services. One state, for example, may define “personal care” as a single service, while another state uses the same term to describe a range of assisted living or residential supports. Eligibility for state TBI services also varies; for example, some programs target adults only, while other programs also serve children.

As described below, the study used information gathered in interviews with TBI stakeholders in seven states and representatives of selected national organizations. The state-based interview informants were not disinterested participants; they were either state officials of the respective state TBI programs or other individuals identified by the state agencies. These individuals could be motivated by an interest in perpetuating federal funding.

LITERATURE REVIEW

Staff searched the primary scientific literature via Medline and collected TBI program data and relevant materials from HRSA, other federal agencies, and state TBI programs; the TBI Technical Assistance Center (TBI TAC), NASHIA, Brain Injury Associations, federally mandated protection and advocacy (P&A) systems in the states, and others.

COMMITTEE MEETINGS AND PUBLIC WORKSHOP

The committee met eight times by telephone conference and held two in-person meetings to formulate its approach to the study, review the data collected, and develop the report. The first in-person meeting included a 1-day public workshop to gain the perspectives and assessments of experts and key stakeholders. Box A-1 presents the workshop agenda, which in-

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

cluded presentations from researchers, state TBI program directors, representatives of federal agencies and nonprofit organizations, and individuals with family members with TBI.

IN-DEPTH LOOK AT TBI INITIATIVES IN SEVEN STATES

The committee agreed to study seven state TBI programs to collect qualitative (interview) data and additional data, where it existed, on states’ experiences with the HRSA TBI Program. Over the course of several meetings, as discussed below, the committee formulated a conceptual framework, developed selection criteria and picked the seven states, developed an interview guide for use in the seven states, and identified the respondent pool. It accomplished these activities with the full-time support of a staff consultant, Holly Korda, Ph.D., Principal, Health Systems Research Associates, Chevy Chase, Maryland, whose final report is presented in Appendix E.

Conceptual Framework

The basic premise of the HRSA TBI Program is that with the modest investment of federal funds, states will build necessary infrastructure, create channels of communication between relevant agencies, and integrate the disparate systems that serve the TBI population—while also finding the means to sustain these efforts. Figure A-1 illustrates the linkages between the HRSA TBI Program and the myriad state, federal, local, public, and private organizations that relate to persons with TBI and their families.

Although the intended long-term outcome of the HRSA TBI Program is improved health and quality of life for persons with TBI and their families, the committee recognized, as had the federal Office of Management and Budget’s Program Assessment Rating Tool study before it in 2004, that it was impossible to evaluate the HRSA program on the basis of clinical outcomes data, which were nonstandardized, inadequate, and/or nonexistent. For that reason, the committee focused instead on demonstrable organizational and strategic change associated with the program, such as infrastructure development, education and training of relevant personnel, planning and implementation of new services, and the development of outreach services to persons with TBI and their families.

Selection of the Seven States for an In-Depth Look

A series of tables detailing the characteristics and self-reported accomplishments of the 50 state TBI programs and federally mandated P&A systems in the states appears in Appendix C. The data in these tables were

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

BOX A-1
Agenda for IOM Workshop on Traumatic Brain Injury July 18, 2005

8:30

Welcome and IntroductionRosemary Stevens, Chair, IOM Committee on Traumatic Brain Injury

8:40

The Epidemiology of TBIJean Langlois, Senior Epidemiologist, National Center for Injury Prevention and Control, Centers for Disease Control and Prevention

9:15

Background and History of the HRSA TBI Program

  • Legislative History and Challenges for the States—Susan Vaughn, Director of Public Policy, National Association of State Head Injury Administrators (NASHIA)

  • History of the HRSA TBI Program—Jane Martin-Heppel, Director, HRSA TBI Program

10:00

Break

10:15

Video

10:30

TBI Resource Facilitation: Clinical, Psycho-Social, Services Challenges

  • Many Services, Many Types of Care—Susan Connors, CEO and President, Brain Injury Association (BIA) of America

  • TBI Needs and Challenges among Native Americans—Alta Bruce, President, Indigenous People’s BIA and Injury Prevention Specialist, Indian Health Service

11:15

TBI Model Systems ProgramRuth Brannon, Associate Director, Division of Research, National Institute on Disability and Rehabilitation Research

drawn from materials provided to the committee by state and federal TBI programs, the TBI Technical Assistance Center (TBI TAC), and NASHIA.

The committee found the national data to be very helpful in indicating activities as reported for grant purposes by different states, but it concluded that the data were not sufficiently standardized to form a study population from which statistically representative states could be drawn. The committee noted, in addition, that each state’s experience with the HRSA program was contingent on the prior history and organizational context of TBI services in that state.

After reviewing this information, the committee concluded that the state programs were too diverse, and the study’s resources too limited, to

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

11:45

Lunch

12:30

HRSA TBI Technical Assistance Center (TBI TAC)

  • Role of TBI TAC and State Technical Assistance—Ken Currier, Acting Executive Director, NASHIA, and Director, TBI TAC

  • Demonstration of TBI Collaborative Space, Benchmarks, Other Tools—Rebecca Zeltinger, TBITAC Director of Knowledge Exchange

1:15

TBI State Grantees and Stakeholders Speak on the Program

  • History of the Disability Legal Advocacy Network—Curt Decker, Executive Director, National Association of Protection and Advocacy

  • Consumer Perspectives—Jan Brown, Member, TBI Technical Assistance Center Steering Committee and Carolyn Rocchio, Family member, Florida Brain Injury Association

2:00

Break

2:15

Leveraging HRSA Funds to Improve Access to TBI Services and Systems

  • National Overview—Bill Ditto, Director, New Jersey Division of Disability Services

  • Empowering People with TBI and Legislative Change—Bil Schmidt, New Mexico TBI Program Director and Beverly James, Participant, NM “Empowerment” Program

  • Using State Data for TBI Systems Change—Manfred Tatzmann, Director, Michigan TBI Program

  • Moving from HRSA Program Funding to Sustainability—Augusta Cash, Alabama TBI Program Director

4:15

Discussion: What Have We Learned? Where Do We Go from Here?Rosemary Stevens

4:45

Adjourn

identify a group of seven states that would represent the full range of TBI experiences. Because the number of states was limited by budget considerations, the committee’s criteria for selection focused on identifying seven states with diverse state experiences that would signal major successes and failures, as well as common themes.

Several criteria were used to select the seven states for in-depth study—for example, participation in the HRSA TBI Program, state TBI funding, locus of the lead state agency for TBI, geographical diversity, etc. (For the rationale for each selection criterion, see Table A-1).

Ultimately, the committee asked seven states—Alabama, California, Colorado, Georgia, New Jersey, Ohio, and Washington State—to partici-

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

FIGURE A-1 Linkages between the Federal TBI Program and other entities.

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

TABLE A-1 The Committee’s Criteria for Selecting Seven States for an In-Depth Look

Criterion

Description

Rationale

Participation in the Federal TBI Program

Type of TBI Program Grant received by the state from the Health Resources and Services Administration (HRSA)—TBI Planning Grant, TBI Implementation Grant, and/or TBI Post-Demonstration Grant)—and duration of the grant received.

In 2005, 2 states had a TBI Planning Grant; 10, a TBI Implementation Grant; and 0, a Post-Demonstration Grant. Past history: 37 states had previous Planning Grants; 40, Implementation Grants; and 23, Post-Demonstration Grants. Some states appear to have moved swiftly and effectively after receiving TBI Program Grant funding from HRSA; other states had not moved beyond modest planning activities.

State TBI funding

TBI trust fund, Medicaid waiver, or dedicated state TBI funding.

The relative importance of Federal TBI Program Grants to states depends on other state funding sources. States with highly developed TBI programs often have TBI trust funds, TBI-specific or other Medicaid waivers, or other dedicated state funding. Less developed TBI programs draw from general state funds.

Locus of the lead state agency for TBI

State agency with lead authority for TBI (appointed by the governor)

The state agency with lead authority for TBI programs varies considerably and may affect the program’s effectiveness, especially as the need for integration of TBI services becomes increasingly complex.

Other

Region, urban/rural, cultural diversity, data systems, and participation in the TBI Model Systems Program.*

The potential impact of a Federal TBI Program Grant to a state may be compromised or enhanced by any of these factors.

*The National Institute on Disability and Rehabilitation Research (NIDRR) provides grants to 16 state TBI Model Systems, programs that develop and conduct research on TBI service delivery systems.

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

TABLE A-2 Dedicated State Funding for TBI in the Seven States

State

Dedicated State Funding (in $ millions)

Trust Fund

Medicaid Waiver

General Funds

Alabama

$1.2

 

California

$1.1

Colorado

$2.5

$5.2

 

Georgia

$2.3

 

New Jersey

$3.4

$14.6

 

Ohio

 

$.346

Washington

 

SOURCE: TBI TAC, 2005.

TABLE A-3 Participation in the Federal TBI Program by the Seven States

State

Federal TBI Program Grant Received

TBI Planning Grant

Implementation Grant

Post-Demonstration Grant

Protection and Advocacy for TBI (PATBI) Grant*

Alabama

California

Colorado

Georgia

New Jersey

Ohio

Washington

*All 50 states receive PATBI Grants.

SOURCE: TBI TAC, 2005.

pate in the interviews (and all agreed).1 These states were selected as the likely best representatives of the considerable state-to-state differences in their preparedness for applying for and using these grants. Tables A-2 to A-5 provide details on selected features of the TBI programs in the seven states.

1  

See Appendix D for detailed profiles of TBI initiatives in each of the seven sample states, including the states’ HRSA grant histories; resources; services for people with TBI and their families; interorganizational collaboration and coordination; TBI-related data, monitoring, and evaluation; and successes and challenges of the HRSA grant experience.

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

TABLE A-4 Locus of the Lead State Agency for TBI in the Seven States

State

Lead Agency

Alabama

Department of Rehabilitative Services

California

Department of Mental Health

Colorado

Office of Behavioral Health & Housing, Department of Human Services

Georgia

Brain and Spinal Injury Trust Fund Commission

New Jersey

Division of Disability Services, Department of Human Services

Ohio

Rehabilitation Services Commission

Washington

Aging and Disability Services Administration, Department of Social and Health Services

SOURCE: TBI TAC, 2005.

Interview Guide Developed by the Committee

Appendix B presents the interview guide developed by the committee for the interviews with TBI stakeholders in the seven states. The committee structured the guide so that it could be used for a 45-minute, in-person, or telephone interview. The interview questions were formulated

TABLE A-5 Other Pertinent Characteristics of the Seven States

State

TBI Data Systems

Percent Rural Population

TBI Registry

Trauma Registry

CDC TBI Surveillancea

TBI Model Systemb

Alabama

44.6

California

5.6

Colorado

15.5

Georgia

28.4

New Jersey

5.6

Ohio

22.6

Washington

18.0

aAlabama, California, Colorado, and New Jersey have received Centers for Disease Control and Prevention (CDC) grants for TBI surveillance. Alabama and Colorado participate in other CDC TBI data projects, medical records studies, and a TBI follow-up registry (Colorado only).

bThe National Institute on Disability and Rehabilitation Research (NIDRR) provides grants to 16 state TBI Model Systems, programs that develop and conduct research on TBI service delivery systems.

SOURCE: TBI TAC, 2005; U.S. 2000 Census Data.

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

to learn respondents’ perceptions in eight areas: (1) the background and context for state TBI activities; (2) HRSA Grant Program participation; (3) TBI services and the impact of the grants; (4) TBI program monitoring and evaluation; (5) state TBI data sources and uses; (6) experience with the TBI TAC; (7) experience as a HRSA TBI Program grantee; and (8) expectations for the future. The committee structured the interview guide to be flexible and open-ended, recognizing that there was wide variation in TBI programs, interview participants, and strategies used to develop TBI infrastructure in different states.

Selection of Interview Respondents

For reasons of efficiency, the committee asked the TBI lead agencies in the states to help identify potential respondents representing TBI interests in the state, including the lead state agency for TBI, the independent P&A system, the Brain Injury Association, the state agency responsible for injury prevention, TBI trust funds, Medicaid waiver programs, and persons with TBI or family members.

FIGURE A-2 Interview respondent pool by organizational affiliation.

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×

Telephone or in-person interviews were conducted with one or more representatives from the various state agencies and organizations. In two states, Georgia and Colorado, the interviews were conducted in person and held as part of 2-day site visits by consultant Holly Korda, committee chair Rosemary Stevens, and study director Jill Eden. In addition to conducting the interviews, the IOM group toured and met with officials of two state-of-the-art rehabilitation facilities: the Shepherd Center in Atlanta and Craig Hospital outside of Denver. Figure A-2 shows the final makeup of the respondent pool by organizational affiliation.

Interviews were also conducted with the leaders of the following national organizations: HRSA TBI Program, NASHIA, TBI TAC,2 Brain Injury Association of America, and National Disability Rights Network.3 During the study period, the Brain Injury Association of America and NASHIA conducted an independent, online survey of stakeholders in all 50 states and the District of Columbia; those results too were made available to the committee.

REFERENCES

Caudle SL. 1994.Using qualitative approaches. Handbook of Practical Program Evaluation (eds. Wholey, Hatry, and Newcomer). San Francisco: Jossey-Bass.


NASHIA (National Association of State Head Injury Administrators). 2005. Guide to State Government Brain Injury Policies, Funding and Services. [Online] Available: http://www.nashia.org [accessed 10/04/2005].

Newcomer KE, Scheirer, MA. 2001. Using evaluation to support performance management: A guide for federal executives. Arlington, VA: PricewaterhouseCoopers Endowment for the Business of Government; Jan.


Sofaer S. 1999. Qualitative methods: What are they and why use them? Health Services Research. 34(5): 1101–1116.


Traumatic Brain Injury Technical Assistance Center (TBI TAC). 2005. Pathway for Systems Change: Benchmarks. Bethesda, MD: TBI TAC.


World Bank Group. 2005. Understanding Impact Evaluation [Web Page]. [Online[ [accessed 12/15/2005]. Available at: http://www.worldbank.org/poverty/impact/index.htm.

2  

NASHIA and TBI TAC are led by the same individual.

3  

Formerly the National Association of Protection and Advocacy Systems, Inc.

Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 105
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 106
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 107
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 108
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 109
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 110
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 111
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 112
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 113
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 114
Suggested Citation:"Appendix A: Methods of the Study." Institute of Medicine. 2006. Evaluating the HRSA Traumatic Brain Injury Program. Washington, DC: The National Academies Press. doi: 10.17226/11600.
×
Page 115
Next: Appendix B: Interview Guide Developed by The IOM Committee on Traumatic Brain Injury With the Assistance of Holly Korda, Ph.D. »
Evaluating the HRSA Traumatic Brain Injury Program Get This Book
×
Buy Paperback | $95.00 Buy Ebook | $74.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

A traumatic brain injury (TBI)—a brain injury caused by a sudden jolt, blow, or penetrating head trauma that disrupts the function of the brain—can happen to anyone. The effects of a TBI vary from person to person, depending on the force dynamics of injury and the patient's anatomy and physiology. People with TBI-related disabilities and their family members and caregivers need comprehensive, coordinated, person-centered systems of care that attend to their changing needs long after their acute injury has been treated medically. At least 5.3 million Americans are estimated to have a TBI-related disability.

The Health Resources and Services Administration's (HRSA) TBI Program, initially authorized by the Traumatic Brain Injury Act of 1996 (P.L. 104-106) and reauthorized by the Children's Health Act of 2000 is a modest federal program with broad ambitions: a $9 million grants program aimed at motivating states to create systems improvement on behalf of persons with TBI with disabilities and their families. In 2004, the federal Office of Management and Budget (OMB) questioned the effectiveness of the HRSA TBI Program, noting that there had been no regular independent evaluations of the program's effects on TBI patients and their families.

To address these concerns, HRSA contracted with the Institute of Medicine (IOM) in the spring of 2005 to conduct a study: (1) to assess the impact of the HRSA Program on how state systems are working or failing to work in support of individuals with TBI; and (2) to advise HRSA on how it could improve the program to best serve individuals with TBI and their families. The IOM appointed an 11-member Committee on Traumatic Brain Injury to perform the study. This report presents the IOM Committee on Traumatic Brain Injury's assessment of the HRSA TBI Program's impact and recommendations for improving the program.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!