A New Partnership between Systems Engineering and Health Care
THE PARADOX OF AMERICAN HEALTH CARE
The United States leads the world in medical science and technology, defining the cutting edge in most fields of clinical research, training, and practice. U.S.-based manufacturers of drugs and medical devices and equipment are considered the most innovative and competitive in the world (AdvaMed, 2004; NSB, 2004). U.S. leadership has been achieved largely by focusing public and private resources on research in the life sciences and physical sciences and on the engineering of devices, instruments, and equipment for treating individual patients. The U.S. market for health care services has supported this focus by rewarding innovation in medical procedures and interventions and the drugs, devices, and equipment linked to them with relatively little regard for cost. Thus, the U.S. health care system provides high quality, highly specialized care for some individuals, but at a very high cost.
At the same time, the U.S. health care enterprise has devoted relatively little technical talent, material resources, or intellectual effort to improving or optimizing the operations of health care systems (especially higher level systems, such as hospitals, health systems, health networks, etc.) or to measuring performance in terms of quality and productivity. The costs to the American economy and the health of Americans of this collective inattention have been enormous. The $1.6 trillion U.S. health care enterprise now faces crises in safety, quality, cost, and access that seriously threaten the health and welfare of many Americans (IOM, 2000, 2001, 2004a,b,c).
To plan a response to these challenges and missed opportunities, the National Institutes of Health, National Science Foundation, and Robert Wood Johnson Foundation asked the National Academy of Engineering and Institute of Medicine (IOM) of the National Academies to conduct a study to identify: (1) engineering applications and tools with the potential to improve health care delivery in the short, medium, and long terms; (2) factors that would facilitate, or inhibit, the deployment of these applications and tools; and (3) priorities for research in engineering and other areas that would contribute to expeditious improvements in the health care delivery system. The sponsors further directed that the study “evaluate current needs and opportunities in the… areas [of]: existing engineering applications that have been proven to improve health care delivery but are not widely deployed; emerging technologies and tools that would help overcome barriers to the delivery of high-quality care; [and] envisioned engineering applications and technologies that could be used to redesign care processes at various levels of the delivery system.”
This report presents a case for a vigorous new partnership between engineering and health care to redress system imbalances. The report outlines a strategy for using information/communications technology and systems-engineering tools to address the crises in health care and improve the quality and productivity of the health care system. In this chapter, the historical origins and structural underpinnings of the interconnected health care crises are described. This is followed by an outline of IOM’s vision of a twenty-first century health care system that meets six quality performance goals: safety, effectiveness, timeliness, patient-centeredness, efficiency, and equity (IOM, 2001). The chapter concludes with a framework for a new partnership between engineering and health care based on systems engineering and advances in information/communications technology with the potential to improve health care and realize IOM’s vision of a truly patient-centered health care delivery system.
INTERCONNECTED CRISES IN U.S. HEALTH CARE
Today, “broken” health care processes and system failures result in the deaths of more than 98,000 Americans and injuries to more than 1 million patients every year (IOM, 2000;
Starfield, 2000). The gap between the rapidly advancing medical knowledge base and its application to patient care can best be described as a chasm. Little more than half of the patients in the United States receive known “best practice” treatment for their illnesses, and less than half of large physician practices provide recommended care processes (e.g., as recommended in disease registries and guidelines) for patients with chronic diseases (Casalino et al., 2003; McGlynn et al., 2003). Many patients are aware that the quality of care they receive is not what it could, or should, be. According to one survey, 75 percent of patients describe the health care system as fragmented and fractured; a “nightmare” to navigate; and plagued by duplications of effort, lack of communication, conflicting advice regarding treatment, and tenuous links to the evolving medical evidence base (Picker Institute, 2000).
The poor quality of care has enormous costs. Health care costs have been rising at double-digit rates since the late 1990s—roughly three times the rate of inflation—claiming a growing share of individual incomes, inflicting economic hardships on many, and making access to care increasingly difficult. Lawrence (see paper in this volume) estimates that $.30 to $.40 of every dollar spent on health care, more than half a trillion dollars per year, is spent on costs associated with “overuse, underuse, misuse, duplication, system failures, unnecessary repetition, poor communication, and inefficiency.”
In addition, the number of people without health insurance has risen to more than 43 million, more than one-sixth of the U.S. population under the age of 65 (IOM, 2004a). Because the uninsured receive little preventive care, they tend to require a disproportionate share of costly chronic and acute care. In addition, the growing number of uninsured increases the disease burden on the uninsured population and imposes a heavy cost burden on providers and payers.
In response to the escalating cost of health care, government and industry—the third-party payers for most people—have shifted a growing share of the cost burden back to care providers and patients by reducing health care benefits, requiring that providers and patients pay a greater share of rising health insurance premiums, increasing co-payments, increasing deductibles, and, in some cases, dropping employee health coverage altogether (Regopoulos and Trude, 2004).
Hospitals and ambulatory care facilities are being forced to do more work with fewer people to keep revenues ahead of rising costs. Unable or unwilling to invest in tools and the complementary capabilities that might increase their productivity, many care-provider organizations have instead cut support staff and increased the workload on existing professional staff. This has undermined morale, causing many nurses to cut back to part-time employment or leave the profession altogether. In addition, these policies have seriously undermined the recruitment of new people to the field. The shortage of nurses alone has been shown to have adverse consequences for safety, quality, and access to health care (IOM, 2004b).
Many physicians have responded by seeing more patients per hour and focusing on activities with high rates of reimbursement and paying less attention to activities related to prevention. Some have even dropped out of the main payment system altogether and demanded retainers from patients who can afford personalized care—a practice known as boutique or concierge medicine.
PROXIMATE CAUSES OF HEALTH CARE CRISES
There are multiple, complex causes of the interrelated crises in health care delivery, but most of them can be traced to the confluence of six factors:
rapid advances in medical science and technology and the increasing complexity of health care during the past half century
the “cottage-industry” structure and acute-care orientation of the health care delivery system
a patient population that predominantly needs chronic care, rather than acute care
the structure of the U.S. market for health care services, which has encouraged and supported innovation in medical procedures, drugs, devices, and equipment, but has been indifferent to, if not discouraged, innovation directed at improving the quality and productivity of care delivery
persistent underinvestment by the health care delivery sector in information/communications technology
the inability or unwillingness of the health care delivery sector to take advantage of engineering-based systems-design, -analysis, and -management tools that have transformed other sectors of the American economy
Science, Technology, Specialization, and Complexity
Advances in medical science and technology since World War II have been a major reason for the growing complexity of American health care, the growing number and increased specialization of people involved in health care delivery, rising expectations about what can be done to treat illnesses, and the enormous increase in scientific and technological information health care providers must manage. To appreciate the impact of advances in medical science, consider the following changes. In the last 30 years, the number of randomized control trials (RCTs) published annually in the U.S. medical literature increased 100-fold, from 100 RCTs per year in the late 1960s to nearly 10,000 RCTs per year by the late 1990s (Chassin, 1998). In the last half-century, the number of categories of health care professionals in the United States increased from 10 to more than 220, roughly a 20-fold increase. Over the same period of time, the number of specialties in medicine increased from fewer than 10 to more than 100 (see paper by Lawrence in this volume).
The increase in specialization in medicine has reinforced the cottage-industry structure of U.S. health care, helping to create a delivery system characterized by disconnected silos of function and specialization. Of the approximately 700,000 clinicians in the United States, who represent more than 100 clinical specialties, more than 80 percent practice medicine in groups of 10 or fewer (see paper by Lawrence in this volume). Less than 24 percent of all physicians directly involved in patient care have practices based in one or more of the 5,800 public or privately owned hospitals, and fewer than 40 percent of hospital-based physicians (roughly 9 percent of all clinicians nationwide) are employed as full-time staff by hospitals (AHA, 2004; Pasko and Smart, 2004). In other words, the vast majority of hospitals, which provide the infrastructure, management systems, and supporting human and material resources for the health care professionals who deliver care to patients, rely heavily on clinicians who function as “independent agents.”
This highly fragmented, highly specialized, independent-practitioner-driven, hospital-centered system of health care delivery has not kept pace with rapid advances in medical knowledge or adapted well to the growing need for chronic care. For decades, McGlynn and many others have documented extensive variations among practitioners (locally, regionally, and nationally) in the treatment of patients with given conditions (McGlynn et al., 2003; Wennberg et al., 1989, 2002). Clearly, a strong attachment to the autonomy of individual clinicians and a deeply held belief that the ultimate responsibility of each clinician is to the individual patient—and that each patient is unique—have actually impeded the diffusion of standard care protocols based on the latest medical evidence (Reinersten, 1996).
Although many clinicians now acknowledge the value of “evidence-based medicine” (the notion that there is a fundamentally correct way to diagnose and treat patients with a given condition) and recognize that they cannot keep up with advances, let alone deliver evidence-based care on their own, the persistent “guild” structure of the health care profession and the hierarchical nature of interaction continue to interfere with the diffusion of evidence-based medicine and the team-oriented care it requires. Indeed, most health care professionals still have little or no training in, or timely access to, the tools and infrastructure necessary to the practice of evidence-based medicine.
The Chronic-Care Imperative
Overall, Americans are living longer, thanks to advances in sanitation and water-treatment systems, emergency care, antibiotics and other medications (e.g., insulin and anti-hypertensive drugs), and other factors. At the same time, chronic conditions in the United States, as in other developed countries, are widespread. About 50 percent of the U.S. population—125 million people—have at least one chronic condition, and about 60 million of these suffer from more than one (Partnership for Solutions, 2002). In addition, a disproportionate amount of health care dollars (more than 75 percent) is spent on patients with chronic conditions (Partnership for Solutions, 2002).
Chronic-care patients require integrated, longitudinal care, that is, coordinated, uninterrupted care, which depends on connectivity among distributed care providers (including family members, physicians, nurses, pharmacists, and others) and the coordination and integration of many functions and specialized areas of knowledge over time. In fact, despite this tremendous need, connectivity, integrated care, and coordination are inadequate at all stages of the treatment of illnesses, from preventive care to acute and chronic care to rehabilitation to long-term care to end-of-life care. Most physicians are not trained to work effectively as members of care-provider teams, and the health care sector as a whole has failed to invest its resources in information infrastructure, information and systems-management tools, and supporting educational, research, and organizational capital that could begin to offset the deep-seated structural and cultural obstacles to coordinated, integrated, continuous patient care.
Structure of the U.S. Market for Health Care Services
The peculiar structure of the U.S. market for health care services and products has also been a significant factor in the current crises. The true cost of health care services is borne not by the patient, or customer, but by third-party payers—employers, private insurers, and the federal government (through Medicare/Medicaid). Insulated from the cost of care, the insured majority of Americans has increasingly come to consider health care as an entitlement. At the same time, the extremely successful U.S. biomedical research establishment has contributed to rising public expectations about the power of medical science and technology to cure diseases and treat illnesses.
In this environment, public and private insurers have been under constant pressure to cover new devices and therapies as they become available, regardless of cost. In the absence of measures of the relative quality or productivity performance of different care providers, insurers have controlled costs by limiting the services they reimburse, offering no incentives for, and, in some cases, actively discouraging, innovation and the application of technologies that could improve the quality and increase the efficiency of care delivery processes and systems.
Information Technology Deficit
For decades, health care has made much less use of information technology than other sectors of the U.S. economy. As of the late 1990s, health services ranked thirty-eighth among 53 major non-farm industries tracked by the U.S.
Department of Commerce in terms of information technology investment per worker. The health services industry spent less than one-tenth the amount invested by banks and nine other manufacturing and services industries (DOC, 1999). Even today, health care has barely begun to take advantage of the information/communications technology systems that have radically reshaped and revolutionized the performance of most major manufacturing and services industries in the United States. In transportation, financial services, communications, and manufacturing industries, modern information/communications systems have enabled and hastened the development of new high-quality products and services and the management of increasingly dispersed and complex production systems. Along with rapid increases in productivity, many of those industries also operate more efficiently with geographically dispersed operations. Although the health care industry has begun to close the information/communications technology gap in the financial and administrative dimensions of its business, core clinical operations are still information technology starved.
Given that the fundamental currency of health care is information, the information/communications technology deficit is ironic. Health care can be thought of as a continual series of information-processing experiments. From the initial collection of data (the patient’s history, physical exam, and diagnostic tests), a hypothesis (diagnosis) is formed and then validated by further data collection. Feedback (the success of the treatment) is a test of the efficacy of the earlier data collection and hypothesis procedures. Information technologies would greatly facilitate every step of these information-processing experiments.
The reasons for the clinical information technology deficit are difficult to untangle. One major contributor is the cottage-industry structure of American health care, which includes many thousands of small businesses (individual clinical practices and small clinics) that cannot rationalize substantial investments in information/communications systems. Moreover, the payment/reimbursement structure for health care services does not reward clinicians for using information/communication technologies in clinical operations.
Another contributor to the clinical information technology gap is limited understanding by clinicians of the potential uses, impacts, and benefits of advanced information/communications technologies for the delivery of care. Clinical information systems in health care delivery can create new relationships that facilitate the exchange of information among sources with different perspectives and develop patient-centered processes of integrated, coordinated care. Designing systems for patient-centered care will require not only investments in information technology hardware and software, but also corresponding investments in related fields, such as human/computer interactions, computer-supported cooperative work, and cognitive engineering (Cook et al., 1998; Woods, 2000). As information/ communications technology is used to expand patient-centered care, dependence on software intensive systems will also increase, which, in turn, will entail new investments in measures to ensure software reliability (NRC, 2004).
Limited Use of Systems Engineering
Given the complexity of health care delivery, which involves the coordination and management of large numbers of highly specialized, distributed personnel, multiple streams of information, and material and financial resources across multiple care settings, it is astounding that health care has not made better use of the design, analysis, and control tools of systems engineering. The experiences of other major manufacturing and services industries, which have relied heavily on systems-engineering concepts and tools to understand, control/manage, and optimize the performance of complex production/distribution systems to meet quality, cost, safety, and other objectives, can provide valuable lessons for health care.
General Motors, Wal-Mart, and Boeing, just to mention a few, could not operate their far-flung organizations in today’s competitive environment without the benefit of comprehensive information/communications systems and the extensive use of engineering tools for the design, analysis, and control of complex production/distribution systems. Deliveries from suppliers are controlled automatically; complex design operations share data instantaneously, resulting in the flawless production of parts and products on different continents; and factory outputs are becoming increasingly responsive to customer demand. Analogous operations can be found throughout the health care system. Thus, it is reasonable to suggest that the use of information/communications technologies and systems tools could lead to higher productivity, better quality care, and improved patient satisfaction.
One must be careful, however, about oversimplifying the parallels between health care and manufacturing and other services industries. Because of the complexities of disease processes, variations in human physiology, and the difficulties in restoring health, simple cut-and-copy approaches to improving health care processes will not suffice. Meeting the challenges of providing health care will require innovative uses of systems-engineering principles and techniques.
THE ROLE OF ENGINEERING IN THE TRANSFORMATION OF HEALTH CARE
In 2001, IOM documented the connections among crises in American health care, set forth a compelling vision for a transformed, twenty-first century, patient-centered health care system, and appealed to engineering for help. IOM identified six interrelated dimensions of quality for the health care system that must be improved. A transformed system must be safe, effective, patient-centered, timely, efficient, and equitable (IOM, 2001):
Safe—avoiding injuries to patients from the care that is intended to help them.
Effective—providing services based on scientific knowledge to all who could benefit and refraining from providing services to those not likely to benefit (avoiding underuse and overuse, respectively).
Patient-centered—providing care that is respectful of and responsive to individual patient preferences, needs, and values and ensuring that patient values guide all clinical decisions.
Timely—reducing waiting times and sometimes harmful delays for those who receive and those who give care.
Efficient—avoiding waste, including waste of equipment, supplies, ideas, and energy.
Equitable—providing care that does not vary in quality because of personal characteristics, such as gender, ethnicity, geographic location, and socioeconomic status.
IOM identified “patient-centeredness” as the unifying and guiding principle for redesigning and improving the health care system to achieve these performance goals.
This patient-centered vision for the twenty-first century health care system not only provides a compelling case for increasing investment in information/communications technology and improving collaboration between medicine and engineering in health care delivery, but also offers a clear functional road map for transformation of the existing system. The IOM report underscores the importance of information/communications technology for meeting multidimensional performance challenges and identified proven, fundamental engineering concepts, such as designing for safety, mass customization, continuous flow, and production planning, that could be brought to bear immediately to redesign and improve care processes.
Currents of Progress in a Stagnant Sea
Since IOM’s clarion call for action, there have been many isolated, localized examples of the selective use of information/communications technologies, systems-engineering tools, and organizational innovations to address one or more of the health care crises. (See, for example, Brandeau et al., 2004, and papers in this volume by Bohmer, Breslow, Coffey, Gustafson, Halamka, Hendrich, Lawrence, Sahney, Stead, Uzsoy, and Zachariah). Although a few institutions have made some progress toward meeting some of IOM’s six quality aims, evidence indicates that the health care delivery system as a whole has not (IOM, 2004d).
Most health care providers continue to underinvest in the technologies, tools, people, and organizational changes necessary to manage and improve clinical care in any of the six dimensions of quality. Overall, crises of quality, cost, and access have become more intense, and scant progress has been made in improving safety, bringing advances in medical science to bear more rapidly on care delivery (effectiveness and timeliness), addressing inequities, and increasing efficiency. Not surprisingly, then, little headway has been made toward patient-centeredness, as many patients can attest (Picker Institute, 2000; see also Safran in this volume).
Given these persistent problems and scattered, isolated attempts to address them, the committee believes it is time to take up the challenge presented in the IOM report to establish a vigorous new partnership between engineering and health care to help bring a systems perspective to health care and hasten the transition to a patient-centered, twenty-first century health care system.
THE ENGINEERING/HEALTH CARE PARTNERSHIP
Engineering and health care have had a long and productive history of collaboration in the development of medical technologies (devices, equipment, pharmaceuticals) and in support of medical research (instrumentation, computational tools, etc.) (IOM, 1995; NAE, 2003). The ongoing revolutions in bioengineering and genomics and the promise of quantum advances in diagnostic tools and therapies testify to the continued vitality of the partnership. Nevertheless, engineering has remained on the periphery of efforts to assess, manage, and redress the shortcomings of the health care delivery system. Information/communications technology, the product of engineering, has been widely used to improve the administrative and financial aspects of the health care industry, but has had relatively little impact on the core business of health care—clinical operations. In short, the principles, tools, and research from engineering disciplines associated with the analysis, design, and control of complex systems (systems engineering, industrial engineering, operations research, human-factors engineering, financial engineering/risk analysis, materials/microelectromechanical systems engineering, etc.)—disciplines that have helped improve, and sometimes transform, many manufacturing and other services industries—are largely unknown in the clinical operations of health care delivery.
The recent history of multiple, interrelated crises of quality, access, and cost in the health care system testifies to the inherent complexity of the health care system and a desperate need for systems-engineering tools and information/communications technology. This complexity reflects the tensions and trade-offs between IOM’s six quality aims for the transformation of health care and the goals, priorities, and perspectives on quality of the many stakeholders in the system—patients, physicians, nurses, administrators, insurers, regulators, and others. Trade-offs among major objectives are not unique to health care. For example, a manufacturer (e.g., an automaker) must make trade-offs between product features that may reduce maintenance costs for the customer but increase manufacturing costs and thus the initial cost of the product. There are many,
many examples of trade-offs in other economic sectors and, in fact, in all complex systems and operations.
Because of the extensive experience of systems engineers in dealing with trade-offs in manufacturing and other technology-intensive service industries, they are adept with the tools, methods, and knowledge base to grasp the deep functions and dynamics of complex systems, provide insights into interactions between subsystems and processes, and understand and manage the tensions and trade-offs among competing system-performance goals and competing priorities of stakeholders in the health care system. Engineering tools and technologies can be used to measure and optimize system performance to meet performance goals, such as safety, effectiveness, patient-centeredness, and timeliness, and, at the same time, anticipate, measure, and manage the effects of these interventions on other performance goals, such as equity, efficiency, and productivity.
Although systems engineering seems a natural partner for addressing the challenges of the health care delivery system, practitioners of the two disciplines are still largely ignorant of each other’s methods, metrics, values, and mind-sets. Most clinicians, as well as most health care administrators, have had little exposure to the research and problem-solving methodologies of engineering; thus, they do not readily grasp how their applications might lead to improvements. By the same token, few engineers are knowledgeable of the complex sociotechnical fabric of health care processes and systems. Thus, they cannot communicate with health care providers in terms and concepts that take account of their values and perceptions. They do not have a common vocabulary for defining problems.
STRUCTURE OF THE REPORT
In the following four chapters of this consensus report (Part 1), the committee attempts to bridge the knowledge/ awareness divide separating health care professionals from their potential partners in the fields of systems engineering and related disciplines. Two overlapping sets of engineering applications are identified—systems-engineering tools and information/communications technologies—that could potentially transform the American health care system. The committee believes that by taking advantage of existing opportunities and pursuing longer range research, short-term and long-term improvements can be made.
In Chapter 2, the committee elaborates on a four-level model of the structure and dynamic of the health care system, the rough division of labor and interdependencies among major elements of the system, and the levers for change throughout the system. An outline of the core elements of a systems approach to the health care delivery system is provided to give both health care professionals and engineers a systems perspective on the major challenges and opportunities facing the health care system and its constituent parts.
In Chapters 3 and 4, two major, interrelated opportunities are described for transforming the system: (1) the use of systems-engineering tools; and (2) the application of information/communications technologies. In Chapter 3, the focus is on (1) the identification of tools that have been demonstrated to be useful in managing large, complex systems that could lead to short-term improvements; and (2) the identification of research opportunities for improving existing tools and making them more user-friendly to achieve long-term improvements and create new, more powerful tools. In Chapter 4, the committee describes opportunities for accelerating the development and widespread diffusion of modern information/communications systems for health care delivery that are integrated with core system tools and technologies and capable of improving connectivity, continuity of care, and responsiveness in the overall health care system. In Chapter 5, the committee proposes an institutional strategy for developing a vigorous partnership between the engineering, management, and health care fields that could lead to the realization of the IOM vision of a high-performance, patient-centered twenty-first century health care system.
Part 2 of the report includes 38 edited, individually authored papers that were presented at three fact-finding workshops. The papers, many of which are cited in Part 1, address various dimensions of the quality/productivity challenges at all levels of the health care system, describe specific applications of systems-engineering tools and information/communications technologies to advance the quality and patient-centeredness of health care delivery, and describe various barriers and incentives to change.
AdvaMed. 2004. The Medical Technology Industry at a Glance 2004. Available online at: http://www.advamed.org/newsroom/chartbook.pdf.
AHA (American Hospital Association). 2004. Hospital Statistics 2004. Chicago, Ill.: AHA.
Brandeau, M.L., F. Sainfort, and W.P. Pierskalla, eds. 2004. Operations Research and Health Care: A Handbook of Methods and Applications. International Series in Operations Research and Management Science Vol. 70. Boston, Mass.: Kluwer Academic Publishers.
Casalino, L., R.R. Gillies, S.M. Shortell, J.A.Schmittdiel, T. Bodenheimer, J.C. Robinson, T. Rundall, N. Oswald, H. Schauffler, and M.C. Wang. 2003. External incentives, information technology, and organized processes to improve health care quality for patients with chronic diseases. Journal of the American Medical Association 289(4): 434–441.
Chassin, M.R. 1998. Is health care ready for six sigma quality? Milbank Quarterly 76(4): 575–591.
Cook, R.I., D.D. Woods, and C. Miller. 1998. A Tale of Two Stories: Contrasting Views on Patient Safety. Chicago, Ill.: National Patient Safety Foundation. Available online at: http://www.npsf.org/exec/report.html.
DOC (U.S. Department of Commerce). 1999. The Emerging Digital Economy II: Appendices. Washington, D.C.: DOC.
IOM (Institute of Medicine). 1995. Sources of Medical Technology: Universities and Industry, edited by N. Rosenberg, A.C. Gelijns, and H. Dawkins. Washington, D.C.: National Academy Press.
IOM. 2000. To Err Is Human: Building a Safer Health System, edited by L.T. Kohn, J.M. Corrigan, and M.S. Donaldson. Washington, D.C.: National Academy Press.
IOM. 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, D.C.: National Academy Press.
IOM. 2004a. Insuring America’s Health: Principles and Recommendations. Washington, D.C.: National Academies Press.
IOM. 2004b. Keeping Patients Safe: Transforming the Work Environment of Nurses. Washington, D.C.: National Academies Press.
IOM. 2004c. Patient Safety: Achieving a New Standard of Care. Washington, D.C.: National Academies Press.
IOM. 2004d. 1st Annual Crossing the Quality Chasm Summit: A Focus on Communities. Washington, D.C.: National Academies Press.
McGlynn, E.A., S.M. Asch, J. Adams, J. Keesey, J. Hicks, A. DeCristofaro, and E.A. Kerr. 2003. The quality of health care delivered to adults in the United States. New England Journal of Medicine 348(26): 2635–2645.
NAE (National Academy of Engineering). 2003. The Impact of Academic Research on Industrial Performance. Washington, D.C.: National Academies Press.
NRC (National Research Council). 2004. Summary of a Workshop on Software Certification and Dependability. Washington, D.C.: National Academies Press.
NSB (National Science Board). 2004. Science and Engineering Indicators 2004 (2 volumes). Arlington, Va.: National Science Foundation. Available online at: http://www.nsf.gov/sbe/srs/seind04/start.htm.
Partnership for Solutions. 2002. Chronic Conditions: Making the Case for Ongoing Care. Prepared for the Robert Wood Johnson Foundation. Baltimore, Md.: Johns Hopkins University Press.
Pasko, T., and D.R. Smart. 2004. Physician Characteristics and Distribution in the US., 2004 ed. Chicago, Ill.: American Medical Association.
Picker Institute. 2000. Eye on Patients. A Report by the Picker Institute for the American Hospital Association. Boston, Mass.: Picker Institute.
Regopoulos, L.E., and S. Trude. 2004. Employers Shift Rising Health Care Costs to Workers: No Long-term Solution in Sight. Issue Brief No. 83. May 2004. Washington, D.C.: Center for Studying Health System Change. Available online at: http://www.hschange.org/CONTENT/677/ [accessed Sept. 9, 2004].
Reinersten, J.L. 1996. Health Care: Past, Present, and Future. 14th Annual Terry C. Shackelford, MD, Memorial Lecture. The Bulletin 40: 61–70.
Starfield, B. 2000. Is U.S. health really the best in the world? Journal of the American Medical Association 284(4): 483–485.
Wennberg, J.E., J.L. Freeman, and R.M. Shelton. 1989. Hospital use and mortality among Medicare beneficiaries in Boston and New Haven. New England Journal of Medicine 321(17): 1168–1173.
Wennberg, J.E., E.S. Fisher, and J.S. Skinner. 2002. Geography and the debate over Medicare reform. Health Affairs: W96–W114. Web Exclusive, February 13, 2002.
Woods, D.D. 2000. Behind Human Error: Human Factors Research to Improve Patient Safety. Washington, D.C.: American Psychological Association.