The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 37
How Science Works daVid GoodsTein David Goodstein, Ph.D., is Professor of Physics and Applied Physics, and the Frank J. Gilloon Distinguished Teaching and Service Professor, Emeritus, California Institute of Technology, Pasadena, California, where he also served for 20 years as Vice Provost. ConTenTs I. Introduction, 38 II. A Bit of History, 38 III. Theories of Science, 39 A. Francis Bacon’s Scientific Method, 39 B. Karl Popper’s Falsification Theory, 40 C. Thomas Kuhn’s Paradigm Shifts, 41 D. An Evolved Theory of Science, 43 IV. Becoming a Professional Scientist, 45 A. The Institutions, 45 B. The Reward System and Authority Structure, 46 V. Some Myths and Facts About Science, 47 VI. Comparing Science and the Law, 51 A. Language, 51 B. Objectives, 52 VII. A Scientist’s View of Daubert, 52 37

OCR for page 37
Reference Manual on Scientific Evidence I. Introduction Recent Supreme Court decisions have put judges in the position of having to decide what is scientific and what is not.1 Some judges may not be entirely comfortable making such decisions, despite the guidance supplied by the Court and illuminated by learned commentators.2 The purpose of this chapter is not to resolve the practical difficulties that judges will encounter in reaching those decisions; it is to demystify somewhat the business of science and to help judges understand the Daubert decision, at least as it appears to a scientist. In the hope of accomplishing these tasks, I take a mildly irreverent look at some formidable subjects. I hope the reader will accept this chapter in that spirit. II. A Bit of History Modern science can reasonably be said to have come into being during the time of Queen Elizabeth I of England and William Shakespeare. Almost immediately, it came into conflict with the law. While Shakespeare was composing his sonnets and penning his plays in England, Galileo Galilei in Italy was inventing the idea that careful experiments in a laboratory could reveal universal truths about the way objects move through space. A bit later, after hearing about the newly invented telescope, he made one for himself, and with it he made discoveries in the heavens that astonished and thrilled all of Europe. Nonetheless, in 1633, Galileo was put on trial for his scientific teachings. The trial of Galileo is usually portrayed as a conflict between science and the Roman Catholic Church, but it was, after all, a trial, with judges and lawyers, and all the other trappings of a formal legal procedure. Another great scientist of the day, William Harvey, who discovered the circu- lation of blood, worked not only at the same time as Galileo, but also at the same place—the University of Padua, not far from Venice. If you visit the University of Padua today and tour the old campus at the heart of the city, you will be shown Galileo’s cattedra, the wooden pulpit from which he lectured (and curiously, one of his vertebrae in a display case just outside the rector’s office—maybe the rector needs to be reminded to have a little spine). You will also be shown the lecture 1. These Supreme Court decisions are discussed in Margaret A. Berger, The Admissibility of Expert Testimony, Sections II–III, IV.A, in this manual. For a discussion of the difficulty in distin- guishing between science and engineering, see Channing R. Robertson et al., Reference Guide on Engineering, in this manual. 2. Since publication of the first edition of this manual, a number of works have been developed to assist judges and attorneys in understanding a wide range of scientific evidence. See, e.g., 1 & 2 Modern Scientific Evidence: The Law and Science of Expert Testimony (David L. Faigman et al. eds., 1997); Expert Evidence: A Practitioner’s Guide to Law, Science, and the FJC Manual (Bert Black & Patrick W. Lee eds., 1997). 38

OCR for page 37
How Science Works theater in which Harvey dissected cadavers while eager students peered down- ward from tiers of overhanging balconies. Because dissecting cadavers was illegal in Harvey’s time, the floor of the theater was equipped with a mechanism that whisked the body out of sight when a lookout gave the word that the authorities were coming. Obviously, both science and the law have changed a great deal since the seventeenth century. Another important player who lived in the same era was not a scientist at all, but a lawyer who rose to be Lord Chancellor of England in the reign of Elizabeth’s successor, James I. His name was Sir Francis Bacon, and in his magnum opus, which he called Novum Organum, he put forth the first theory of the scientific method. In Bacon’s view, the scientist should be an impartial observer of nature, collecting observations with a mind cleansed of harmful preconceptions that might cause error to creep into the scientific record. Once enough such observations were gathered, patterns would emerge, giving rise to truths about nature. Bacon’s theory has been remarkably influential down through the centuries, even though in his own time there were those who knew better. “That’s exactly how a Lord Chancellor would do science,” William Harvey is said to have grumbled. III. Theories of Science Today, in contrast to the seventeenth century, few would deny the central impor- tance of science to our lives, but not many would be able to give a good account of what science is. To most, the word probably brings to mind not science itself, but the fruits of science, the pervasive complex of technology and discoveries that has transformed all of our lives. However, science also might equally be thought to include the vast body of knowledge we have accumulated about the natural world. There are still mysteries, and there always will be mysteries, but the fact is that, by and large, we understand how nature works. A. Francis Bacon’s Scientific Method But science is even more than that. Ask a scientist what science is, and the answer will almost surely be that it is a process—a way of examining the natural world and discovering important truths about it. In short, the essence of science is the scientific method.3 3. The Supreme Court, in Daubert v. Merrell Dow Pharmaceuticals, Inc., acknowledged the impor- tance of defining science in terms of its methods as follows: “‘Science is not an encyclopedic body of knowledge about the universe. Instead, it represents a process for proposing and refining theoretical explanations about the world that are subject to further testing and refinement’” (emphasis in original). 509 U.S. 579, 590 (1993) (quoting Brief for the American Association for the Advancement of Science and the National Academy of Sciences as Amici Curiae at 7–8). 39

OCR for page 37
Reference Manual on Scientific Evidence This stirring description suffers from an important shortcoming. We do not really know what the scientific method is.4 There have been many attempts at formulating a general theory of how science works, or at least how it should work, starting, as we have seen, with the theory of Sir Francis Bacon. But Bacon’s idea, that science proceeds through the collection of observations without prejudice, has been rejected by all serious thinkers. Everything about the way we do science— the language we use, the instruments we use, the methods we use—depends on clear presuppositions about how the world works. Modern science is full of things that cannot be observed at all, such as force fields and complex molecules. At the most fundamental level, it is impossible to observe nature without having some reason to choose what is and is not worth observing. Once that elementary choice is choice is made, Bacon has been left behind. B. Karl Popper’s Falsification Theory Over the past century, the ideas of the Vienna-born philosopher Sir Karl Popper have had a profound effect on theories of the scientific method.5 In contrast to Bacon, Popper believed that all science begins with a prejudice, or perhaps more politely, a theory or hypothesis. Nobody can say where the theory comes from. Formulating the theory is the creative part of science, and it can- not be analyzed within the realm of philosophy. However, once the theory is in hand, Popper tells us, it is the duty of the scientist to extract from it logical but unexpected predictions that, if they are shown by experiment not to be correct, will serve to render the theory invalid. Popper was deeply influenced by the fact that a theory can never be proved right by agreement with observation, but it can be proved wrong by disagreement with observation. Because of this asymmetry, science uniquely makes progress by proving that good ideas are wrong so that they can be replaced by even better ideas. Thus, Bacon’s impartial observer of nature is replaced by Popper’s skeptical theorist. The good Popperian scientist somehow comes up with a hypothesis that fits all or most of the known facts, then proceeds to attack that hypothesis at its weakest point by extracting from it predictions that can be shown to be false. This process is known as falsification.6 4. For a general discussion of theories of the scientific method, see Alan F. Chalmers, What Is This Thing Called Science? (1982). For a discussion of the ethical implications of the various theories, see James Woodward & David Goodstein, Conduct, Misconduct and the Structure of Science, 84 Am. Scientist 479 (1996). 5. See, e.g., Karl R. Popper, The Logic of Scientific Discovery (Karl R. Popper trans., 1959). 6. The Supreme Court in Daubert recognized Popper’s conceptualization of scientific knowl- edge by noting that “[o]rdinarily, a key question to be answered in determining whether a theory or technique is scientific knowledge that will assist the trier of fact will be whether it can be (and has been) tested.” 509 U.S. at 593. In support of this point, the Court cited as parenthetical passages from both Carl Gustav Hempel, Philosophy of Natural Science 49 (1966) (“‘[T]he statements constituting 40

OCR for page 37
How Science Works Popper’s ideas have been fruitful in weaning the philosophy of science away from the Baconian view and some other earlier theories, but they fall short in a number of ways in describing correctly how science works. The first of these is the observation that, although it may be impossible to prove a theory is true by observation or experiment, it is as almost equally impossible to prove one is false by these same methods. Almost without exception, in order to extract a falsifiable prediction from a theory, it is necessary to make additional assumptions beyond the theory itself. Then, when the prediction turns out to be false, it may well be one of the other assumptions, rather than the theory itself, that is false. To take a simple example, early in the twentieth century it was found that the orbits of the outermost planets did not quite obey the predictions of Newton’s laws of gravity and mechanics. Rather than take this to be a falsification of Newton’s laws, astron- omers concluded that the orbits were being perturbed by an additional unseen body out there. They were right. That is precisely how Pluto was discovered. The apparent asymmetry between falsification and verification that lies at the heart of Popper’s theory thus vanishes. But the difficulties with Popper’s view go even beyond that problem. It takes a great deal of hard work to come up with a new theory that is consistent with nearly everything that is known in any area of science. Popper’s notion that the scientist’s duty is then to attack that theory at its most vulnerable point is fundamentally inconsistent with human nature. It would be impossible to invest the enormous amount of time and energy necessary to develop a new theory in any part of modern science if the primary purpose of all that work was to show that the theory was wrong. This point is underlined by the fact that the behavior of the scientific com- munity is not consistent with Popper’s notion of how it should be. Credit in science is most often given for offering correct theories, not wrong ones, or for demonstrating the correctness of unexpected predictions, not for falsifying them. I know of no example of a Nobel Prize awarded to a scientist for falsifying his or her own theory. C. Thomas Kuhn’s Paradigm Shifts Another towering figure in the twentieth century theory of science is Thomas Kuhn.7 Kuhn was not a philosopher but a historian (more accurately, a physi- cist who retrained himself as a historian). It is Kuhn who popularized the word paradigm, which has today come to seem so inescapable. A paradigm, for Kuhn, is a kind of consensual worldview within which scien- tists work. It comprises an agreed-upon set of assumptions, methods, language, and a scientific explanation must be capable of empirical test’”) and Karl R. Popper, Conjectures and Refutations: The Growth of Scientific Knowledge 37 (5th ed. 1989) (“‘[T]he criterion of the scientific status of a theory is its falsifiability, or refutability, or testability’”). 7. Thomas S. Kuhn, The Structure of Scientific Revolutions (1962). 41

OCR for page 37
Reference Manual on Scientific Evidence everything else needed to do science. Within a given paradigm, scientists make steady, incremental progress, doing what Kuhn calls “normal science.” As time goes on, difficulties and contradictions arise that cannot be resolved, but the tendency among scientists is to resist acknowledging them. One way or another they are swept under the rug, rather than being allowed to threaten the central paradigm. However, at a certain point, enough of these difficulties accu- mulate to make the situation intolerable. At that point, a scientific revolution occurs, shattering the paradigm and replacing it with an entirely new one. This new paradigm, says Kuhn, is so radically different from the old that normal discourse between the practitioners of the two paradigms becomes impos- sible. They view the world in different ways and speak different languages. It is not even possible to tell which of the two paradigms is superior, because they address different sets of problems. They are incommensurate. Thus, science does not progress incrementally, as the science textbooks would have it, except during periods of normal science. Every once in a while, a scientific revolution brings about a paradigm shift, and science heads off in an entirely new direction. Kuhn’s view was formed largely on the basis of two important historical revolutions. One was the original scientific revolution that started with Nicolaus Copernicus and culminated with the new mechanics of Isaac Newton. The very word revolution, whether it refers to the scientific kind, the political kind, or any other kind, refers metaphorically to the revolutions in the heavens that Copernicus described in a book, De Revolutionibus Orbium Caelestium, published as he lay dying in 1543.8 Before Copernicus, the dominant paradigm was the worldview of ancient Greek philosophy, frozen in the fourth century B.C.E. ideas of Plato and Aristotle. After Newton, whose masterwork, Philosophiæ Naturalis Principia Mathematica, was published in 1687, every scientist was a Newtonian, and Aristotelianism was banished forever from the world stage. It is even possible that Sir Francis Bacon’s disinterested observer was a reaction to Aristotelian authority. Look to nature, not to the ancient texts, Bacon may have been saying. The second revolution that served as an example for Kuhn occurred early in the twentieth century. In a headlong series of events that lasted a mere 25 years, the Newtonian paradigm was overturned and replaced with the new physics, in the form of quantum mechanics and Einstein’s theories of special and general relativity. This second revolution, although it happened much faster, was no less profound than the first. The idea that science proceeds by periods of normal activity punctuated by shattering breakthroughs that make scientists rethink the whole problem is an appealing one, especially to the scientists themselves, who believe from personal experience that it really happens that way. Kuhn’s contribution is important. It offers us a useful context (a paradigm, one might say) for organizing the entire history of science. 8. I. Bernard Cohen, Revolution in Science (1985). 42

OCR for page 37
How Science Works Nonetheless, Kuhn’s theory does suffer from a number of shortcomings. One of them is that it contains no measure of how big the change must be in order to qualify as a revolution or paradigm shift. Most scientists will say that there is a paradigm shift in their laboratory every 6 months or so (or at least every time it becomes necessary to write another proposal for research support). That is not exactly what Kuhn had in mind. Another difficulty is that even when a paradigm shift is truly profound, the paradigms it separates are not necessarily incommensurate. The new sciences of quantum mechanics and relativity, for example, did indeed show that Newton’s laws of mechanics were not the most fundamental laws of nature. However, they did not show that they were wrong. Quite the contrary, they showed why Newton’s laws were right: Newton’s laws arose out of newly discovered laws that were even deeper and that covered a wider range of circumstances unimagined by Newton and his followers—that is, phenomena as small as atoms, or nearly as fast as the speed of light, or as dense as black holes. In our more familiar realms of experience, Newton’s laws go on working just as well as they always did. Thus, there is no quarrel and no ambiguity at all about which paradigm is “better.” The new laws of quantum mechanics and relativity subsume and enhance the older Newtonian worldview. D. An Evolved Theory of Science If neither Bacon nor Popper nor Kuhn gives us a perfect description of what science is or how it works, all three of them help us to gain a much deeper understanding of it. Scientists are not Baconian observers of nature, but all scientists become Baconians when it comes to describing their observations. With very few excep- tions, scientists are rigorously, even passionately, honest about reporting scientific results and how they were obtained. Scientific data are the coin of the realm in science, and they are always treated with reverence. Those rare instances in which scientists are found to have fabricated or altered their data in some way are always traumatic scandals of the first order.9 Scientists are also not Popperian falsifiers of their own theories, but they do not have to be. They do not work in isolation. If a scientist has a rival with a different theory of the same phenomenon, the rival will be more than happy to perform the Popperian duty of attacking the scientist’s theory at its weakest point. 9. Such instances are discussed in David Goodstein, Scientific Fraud, 60 Am. Scholar 505 (1991). For a summary of recent investigations into scientific fraud and lesser instances of scientific misconduct, see Office of Research Integrity, Department of Health and Human Services, Scientific Misconduct Investigations: 1993–1997, http://ori.dhhs.gov/PDF/scientific.pdf (last visited Nov. 21, 1999) (summarizing 150 scientific misconduct investigations closed by the Office of Research Integrity). 43

OCR for page 37
Reference Manual on Scientific Evidence Moreover, if falsification is no more definitive than verification, and scientists prefer in any case to be right rather than wrong, they nonetheless know how to hold verification to a very high standard. If a theory makes novel and unexpected predictions, and those predictions are verified by experiments that reveal new and useful or interesting phenomena, then the chances that the theory is correct are greatly enhanced. And, even if it is not correct, it has been fruitful in the sense that it has led to the discovery of previously unknown phenomena that might prove useful in themselves and that will have to be explained by the next theory that comes along. Finally, science does not, as Kuhn seemed to think, periodically self-destruct and need to start over again. It does, however, undergo startling changes of per- spective that lead to new and, invariably, better ways of understanding the world. Thus, although science does not proceed smoothly and incrementally, it is one of the few areas of human endeavor that is genuinely progressive. There is no doubt at all that the quality of twentieth century science is better than nineteenth century science, and we can be absolutely confident that the quality of science in the twenty-first century will be better still. One cannot say the same about, say, art or literature.10 To all of this, a few things must be added. The first is that science is, above all, an adversarial process. It is an arena in which ideas do battle, with observa- tions and data the tools of combat. The scientific debate is very different from what happens in a court of law, but just as in the law, it is crucial that every idea receive the most vigorous possible advocacy, just in case it might be right. Thus, the Popperian ideal of holding one’s hypothesis in a skeptical and tentative way is not merely inconsistent with reality; it would be harmful to science if it were pursued. As will be discussed shortly, not only ideas, but the scientists themselves, engage in endless competition according to rules that, although they are not writ- ten down, are nevertheless complex and binding. In the competition among ideas, the institution of peer review plays a central role. Scientific articles submitted for publication and proposals for funding often are sent to anonymous experts in the field, in other words, to peers of the author, for review. Peer review works superbly to separate valid science from nonsense, 10. The law, too, can claim to be progressive. The development of legal constructs, such as due process, equal protection, and individual privacy, reflects notable progress in the betterment of man- kind. See Laura Kalman, The Strange Career of Legal Liberalism 2–4 (1996) (recognizing the “faith” of legal liberalists in the use of law as an engine for progressive social change in favor of society’s disadvantaged). Such progress is measured by a less precise form of social judgment than the consensus that develops regarding scientific progress. See Steven Goldberg, The Reluctant Embrace: Law and Science in America, 75 Geo. L.J. 1341, 1346 (1987) (“Social judgments, however imprecise, can sometimes be reached on legal outcomes. If a court’s decision appears to lead to a sudden surge in the crime rate, it may be judged wrong. If it appears to lead to new opportunities for millions of citizens, it may be judged right. The law does gradually change to reflect this kind of social testing. But the process is slow, uncertain, and controversial; there is nothing in the legal community like the consensus in the scientific community on whether a particular result constitutes progress.”). 44

OCR for page 37
How Science Works or, in Kuhnian terms, to ensure that the current paradigm has been respected.11 It works less well as a means of choosing between competing valid ideas, in part because the peer doing the reviewing is often a competitor for the same resources (space in prestigious journals, funds from government agencies or private foun- dations) being sought by the authors. It works very poorly in catching cheating or fraud, because all scientists are socialized to believe that even their toughest competitor is rigorously honest in the reporting of scientific results, which makes it easy for a purposefully dishonest scientist to fool a referee. Despite all of this, peer review is one of the venerated pillars of the scientific edifice. IV. Becoming a Professional Scientist Science as a profession or career has become highly organized and structured.12 It is not, relatively speaking, a very remunerative profession—that would be inconsistent with the Baconian ideal—but it is intensely competitive, and material well-being does tend to follow in the wake of success (successful scientists, one might say, do get to bring home the Bacon). A. The Institutions These are the institutions of science: Research is done in the Ph.D.-granting universities and, to a lesser extent, in colleges that do not grant Ph.D.s. It is also done in national laboratories and in industrial laboratories. Before World War II, basic science was financed mostly by private foundations (Rockefeller, Carnegie), but since the war, the funding of science (except in industrial laboratories) has largely been taken over by agencies of the federal government, notably the National Science Foundation (an independent agency), the National Institutes of 11. The Supreme Court received differing views regarding the proper role of peer review. Compare Brief for Amici Curiae Daryl E. Chubin et al. at 10, Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579 (1993) (No. 92-102) (“peer review referees and editors limit their assessment of submit- ted articles to such matters as style, plausibility, and defensibility; they do not duplicate experiments from scratch or plow through reams of computer-generated data in order to guarantee accuracy or veracity or certainty”), with Brief for Amici Curiae New England Journal of Medicine, Journal of the American Medical Association, and Annals of Internal Medicine in Support of Respondent, Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579 (1993) (No. 92-102) (proposing that publication in a peer-reviewed journal be the primary criterion for admitting scientific evidence in the courtroom). See generally Daryl E. Chubin & Edward J. Hackett, Peerless Science: Peer Review and U.S. Science Policy (1990); Arnold S. Relman & Marcia Angell, How Good Is Peer Review? 321 New Eng. J. Med. 827–29 (1989). As a practicing scientist and frequent peer reviewer, I can testify that Chubin’s view is correct. 12. The analysis that follows is based on David Goodstein & James Woodward, Inside Science, 68 Am. Scholar 83 (1999). 45

OCR for page 37
Reference Manual on Scientific Evidence Health (part of the Department of Health and Human Services), and parts of the Department of Energy and the Department of Defense. Scientists who work at all these organizations—universities, colleges, national and industrial laboratories, and funding agencies—belong to scientific societies that are organized mostly by discipline. There are large societies, such as the American Physical Society and the American Chemical Society; societies for subdisciplines, such as optics and spectroscopy; and even organizations of societies, such as FASEB, the Federation of American Societies for Experimental Biology. Scientific societies are private organizations that elect their own officers, hold scientific meetings, publish journals, and finance their operations from the collection of dues and from the proceeds of their publishing and educational activities. The American Association for the Advancement of Science also holds meetings and publishes Science, a famous journal, but it is not restricted to any one discipline. The National Academy of Sciences holds meetings and publishes the Proceedings of the National Academy of Sciences, and, along with the National Acad- emy of Engineering, Institute of Medicine, and its operational arm, the National Research Council, advises various government agencies on matters pertaining to science, engineering, and health. In addition to the advisory activities, one of its most important activities is to elect its own members. These are the basic institutions of American science. It should not come as news that the universities and colleges engage in a fierce but curious competi- tion, in which no one knows who is keeping score, but everyone knows roughly what the score is. (In recent years, some national and international media outlets have found it worthwhile to appoint themselves scorekeepers in this competi- tion. Academic officials dismiss these journalistic judgments, except when their own institutions come out on top.) Departments in each discipline compete with one another, as do national and industrial laboratories and even funding agencies. Competition in science is at its most refined, however, at the level of individual careers. B. The Reward System and Authority Structure To regulate competition among scientists, there is a reward system and an authority structure. The fruits of the reward system are fame, glory, and immortality. The purposes of the authority structure are power and influence. The reward system and the authority structure are closely related to one another, but scientists distinguish sharply between them. When they speak of a colleague who has become president of a famous university, they will say sadly, “It’s a pity—he was still capable of good work,” sounding like warriors lamenting the loss of a fallen comrade. The univer- sity president is a kingpin of the authority structure, but, with rare exceptions, he is a dropout from the reward system. Similar kinds of behavior can be observed in industrial and government laboratories, but a description of what goes on in universities will be enough to illustrate how the system works. 46

OCR for page 37
How Science Works A career in academic science begins at the first step on the reward system ladder, earning a Ph.D., followed in many areas by one or two stints as a post- doctoral fellow. The Ph.D. and postdoctoral positions had best be at universities (or at least departments) that are high up in that fierce but invisible competi- tion, because all subsequent steps are more likely than not to take the individual sideways or downward on the list. The next step is a crucial one: appointment to a tenure-track junior faculty position. About two-thirds of all postdoctoral fellows in biology in American universities believe that they are going to make this step, but in fact, only about a quarter of them succeed. This step and all sub- sequent steps require growing renown as a scientist beyond the individual’s own circle of acquaintances. Thus, it is essential by this time that the individual has accomplished something. The remaining steps up the reward system ladder are promotion to an academic tenured position and full professorship; various prizes, medals, and awards given out by the scientific societies; an endowed chair (the virtual equivalent of Galileo’s wooden cattedra); election to the National Academy of Sciences; particularly prestigious awards up to and including the Nobel Prize; and, finally, a reputation equivalent to immortality. Positions in the authority structure are generally rewards for having achieved a certain level in the reward system. For example, starting from the Ph.D. or junior faculty level, it is possible to step sideways temporarily or even permanently into a position as contract officer in a funding agency. Because contract officers influence the distribution of research funds, they have a role in deciding who will succeed in the climb up the reward system ladder. At successively higher levels one can become a journal editor; department chair; dean, provost, director of national research laboratory or president of a university; and even the head of a funding agency, a key player in determining national policy as it relates to science and technology. People in these positions have stepped out of the traditional reward system, but they have something to say about who succeeds within it. V. Some Myths and Facts About Science “In matters of science,” Galileo wrote, “the authority of thousands is not worth the humble reasoning of one single person.”13 Doing battle with the Aristotelian professors of his day, Galileo believed that kowtowing to authority was the enemy of reason. But, contrary to Galileo’s famous remark, the fact is that within the scientific community itself, authority is of fundamental importance. If a paper’s 13. I found this statement framed on the office wall of a colleague in Italy in the form, “In questioni di scienza L’autorità di mille non vale l’umile ragionare di un singolo.” However, I have not been able to find the famous remark in this form in Galileo’s writings. An equivalent statement in different words can be found in Galileo’s Il Saggiatore (1623). See Andrea Frova & Mariapiera Marenzona, Parola di Galileo 473 (1998). 47

OCR for page 37
Reference Manual on Scientific Evidence author is a famous scientist, the paper is probably worth reading. The triumph of reason over authority is just one of the many myths about science. Following is a brief list of some others: Myth: Scientists must have open minds, being ready to discard old ideas in favor of new ones. Fact: Because science is an adversarial process through which each idea deserves the most vigorous possible defense, it is useful for the suc- cessful progress of science that scientists tenaciously cling to their own ideas, even in the face of contrary evidence. Myth: The institution of peer review assures that all published papers are sound and dependable. Fact: Peer review generally will catch something that is completely out of step with majority thinking at the time, but it is practically useless for catching outright fraud, and it is not very good at dealing with truly novel ideas. Peer review mostly assures that all papers follow the cur- rent paradigm (see comments on Kuhn, above). It certainly does not ensure that the work has been fully vetted in terms of the data analysis and the proper application of research methods. Myth: Science must be an open book. For example, every new experiment must be described so completely that any other scientist can reproduce it. Fact: There is a very large component of skill in making cutting-edge experiments work. Often, the only way to import a new technique into a laboratory is to hire someone (usually a postdoctoral fellow) who has already made it work elsewhere. Nonetheless, scientists have a solemn responsibility to describe the methods they use as fully and accurately as possible. And, eventually, the skill will be acquired by enough people to make the new technique commonplace. Myth: When a new theory comes along, the scientist’s duty is to falsify it. Fact: When a new theory comes along, the scientist’s instinct is to verify it. When a theory is new, the effect of a decisive experiment that shows it to be wrong is that both the theory and the experiment are in most cases quickly forgotten. This result leads to no progress for anybody in the reward system. Only when a theory is well established and widely accepted does it pay off to prove that it is wrong. Myth: University-based research is pure and free of conflicts of interest. Fact: The Bayh-Dole Act of the early 1980s permits universities to patent the results of research supported by the federal government. Many uni- versities have become adept at obtaining such patents. In many cases 48

OCR for page 37
How Science Works this raises conflict-of-interest problems when the universities’ interest in pursuing knowledge comes into conflict with its need for revenue. This is an area that has generated considerable scrutiny. For instance, the recent Institute of Medicine report Conflict of Interest in Medical Research, Education, and Practice sheds light on the changing dimensions of conflicts of interest associated with growing interdisciplinary col- laborations between individuals, universities, and industry especially in life sciences and biomedical research.14 Myth: Real science is easily distinguished from pseudoscience. Fact: This is what philosophers call the problem of demarcation: One of Popper’s principal motives in proposing his standard of falsifiability was precisely to provide a means of demarcation between real science and impostors. For example, Einstein’s general theory of relativity (with which Popper was deeply impressed) made clear predictions that could certainly be falsified if they were not correct. In contrast, Freud’s theories of psychoanalysis (with which Popper was far less impressed) could never be proven wrong. Thus, to Popper, relativity was science but psychoanalysis was not. Real scientists do not behave as Popper says they should, and there is another problem with Popper’s criterion (or indeed any other criterion) for demarcation: Would-be scientists read books too. If it becomes widely accepted (and to some extent it has) that falsifiable predictions are the signature of real science, then pretenders to the throne of science will make falsifiable predictions too.15 There is no simple, mechanical criterion for distinguishing real science from some- thing that is not real science. That certainly does not mean, however, that the job cannot be done. As I discuss below, the Supreme Court, in the Daubert decision, has made a respectable stab at showing how to do it.16 14. Institute of Medicine, Conflict of Interest in Medical Research, Education, and Practice (Bernard Lo & Marilyn Field eds., 2009). 15. For a list of such pretenders, see Larry Laudan, Beyond Positivism and Relativism 219 (1996). 16. The Supreme Court in Daubert identified four nondefinitive factors that were thought to be illustrative of characteristics of scientific knowledge: testability or falsifiability, peer review, a known or potential error rate, and general acceptance within the scientific community. 509 U.S. at 590. Subse- quent cases have expanded on these factors. See, e.g., In re TMI Litig. Cases Consol. II, 911 F. Supp. 775, 787 (M.D. Pa. 1995) (which considered the following additional factors: the relationship of the technique to methods that have been established to be reliable, the qualifications of the expert witness testifying based on the methodology, the nonjudicial uses of the method, logical or internal consistency of the hypothesis, the consistency of the hypothesis with accepted theories, and the precision of the hypothesis or theory). See generally Bert Black et al., Science and the Law in the Wake of Daubert: A New Search for Scientific Knowledge, 72 Tex. L. Rev. 715, 783–84 (1994) (discussion of expanded list of factors). 49

OCR for page 37
Reference Manual on Scientific Evidence Myth: Scientific theories are just that: theories. All scientific theories are eventually proved wrong and are replaced by other theories. Fact: The things that science has taught us about how the world works are the most secure elements in all of human knowledge. Here I must distinguish between science at the frontiers of knowledge (where by definition we do not yet understand everything and where theories are indeed vulnerable) and textbook science that is known with great confidence. Matter is made of atoms, DNA transmits the blueprints of organisms from generation to generation, light is an electromagnetic wave—these things are not likely to be proved wrong. The theory of relativity and the theory of evolution are in the same class and are still called “theories” for historic reasons only.17 The GPS device in my car routinely uses the general theory of relativity to make calculations accurate enough to tell me exactly where I am and to take me to my destination with unerring precision. The phenomenon of natural selec- tion has been observed under numerous field conditions as well as in controlled laboratory experiments. In recent times, the courts have had much to say about the teach- ing of the theory of evolution in public schools.18 In one instance the school district decided that students should be taught the “gaps/ problems” in Darwin’s theory and given “Intelligent Design” as an alternative explanation. The courts (Judge Jones of the United States District Court for the Middle District of Pennsylvania) came down hard on the side of Darwin, ruling that “Intelligent Design” was thinly disguised religion that had no place in the science classroom. It should be said here that the incorrect notion that all theories must eventually be wrong is fundamental to the work of both Popper and Kuhn, and these theorists have been crucial in helping us under- stand how science works. Thus, their theories, like good scientific theories at the frontiers of knowledge, can be both useful and wrong. Myth: Scientists are people of uncompromising honesty and integrity. Fact: They would have to be if Bacon were right about how science works, but he was not. Most scientists are rigorously honest where honesty matters most to them: in the reporting of scientific procedures and data in peer-reviewed publications. In all else, they are ordinary mortals. 17. According to the National Academy of Sciences and Institute of Medicine’s 2008 report Science, Evolution, and Creationism, “the strength of a theory rests in part on providing scientists with the basis to explain observed phenomena and to predict what they are likely to find when exploring new phenomena and observations.” The report also helps differentiate a theory from a hypothesis, the latter being testable natural explanations that may offer tentative scientific insights. 18. Kitzmiller v. Dover Area School District, 400 F. Supp. 2d 707 (M.D. Pa. 2005). 50

OCR for page 37
How Science Works VI. Comparing Science and the Law Science and the law differ both in the language they use and the objectives they seek to accomplish. A. Language Oscar Wilde (and G.B. Shaw too) once remarked that the United States and England are two nations divided by a common language. Something similar can be said, with perhaps more truth (if less wit), of science and the law. There are any number of words commonly used in both disciplines, but with different meanings. For example, the word force, as it is used by lawyers, has connotations of vio- lence and the domination of one person’s will over another, when used in phrases such as “excessive use of force” and “forced entry.” In science, force is something that when applied to a body, causes its speed and direction of motion to change. Also, all forces arise from a few fundamental forces, most notably gravity and the electric force. The word carries no other baggage. In contrast, the word evidence is used much more loosely in science than in law. The law has precise rules of evidence that govern what is admissible and what is not. In science, the word merely seems to mean something less than “proof.” A certain number of the papers in any issue of a scientific journal will have titles that begin with “Evidence for (or against) . . .” What that means is, the authors were not able to prove their point, but are presenting their results anyway. The word theory is a particularly interesting example of a word that has dif- ferent meanings in each discipline. A legal theory is a proposal that fits the known facts and legal precedents and that favors the attorney’s client. What’s required of a theory in science is that it make new predictions that can be tested by new experiments or observations and falsified or verified (as discussed above). Even the word law has different meanings in the two disciplines. To a legal practitioner, a law is something that has been promulgated by some human authority, such as a legislature or parliament. In science, a law is a law of nature, something that humans can hope to discover and describe accurately, but that can never be changed by any human authority or intervention. My final example is, to me, the most interesting of all. It is the word error. In the law, and in common usage, error and mistake are more or less synonymous. A legal decision can be overturned if it is found to be contaminated by judicial error. In science, however, error and mistake have different meanings. Anyone can make a mistake, and scientists have no obligation to report theirs in the scientific literature. They just clean up the mess and go on to the next attempt. Error, on the other hand, is intrinsic to any measurement, and far from ignoring it or covering it up or even attempting to eliminate it, authors of every paper about a scientific experiment will include a careful analysis of the errors to put limits on 51

OCR for page 37
Reference Manual on Scientific Evidence the uncertainty in the measured result. To make mistakes is human, one might say, but error is intrinsic to our interaction with nature, and is therefore part of science. B. Objectives Beyond the meanings of certain key words, science and the law differ funda- mentally in their objectives. The objective of the law is justice; that of science is truth.19 These are among the highest goals to which humans can aspire, but they are not the same thing. Justice, of course, also seeks truth, but it requires that clear decisions be made in a reasonable and limited period of time. In the scientific search for truth there are no time limits and no point at which a final decision must be made. And yet, despite all these differences, science and the law share, at the deepest possible level, the same aspirations and many of the same methods. Both disci- plines seek, in structured debate and using empirical evidence, to arrive at rational conclusions that transcend the prejudices and self-interest of individuals. VII. A Scientist’s View of Daubert In the 1993 Daubert decision, the U.S. Supreme Court took it upon itself to resolve, once and for all, the knotty problem of the demarcation between science and pseudoscience. Better yet, it undertook to enable every federal judge to solve that problem in deciding the admissibility of each scientific expert witness in every case that arises. In light of all the uncertainties discussed in this chapter, it must be considered an ambitious thing to do.20 The presentation of scientific evidence in a court of law is a kind of shotgun marriage between the two disciplines. Both are obliged to some extent to yield 19. This point was made eloquently by D. Allen Bromley in Science and the Law, Address at the 1998 Annual Meeting of the American Bar Association (Aug. 2, 1998). 20. Chief Justice Rehnquist, responding to the majority opinion in Daubert, was the first to express his uneasiness with the task assigned to federal judges, as follows: “I defer to no one in my confidence in federal judges; but I am at a loss to know what is meant when it is said that the scientific status of a theory depends on its ‘falsifiability,’ and I suspect some of them will be, too.” 509 U.S. at 579 (Rehnquist, C.J., concurring in part and dissenting in part). His concern was then echoed by Judge Alex Kozinski when the case was reconsidered by the U.S. Court of Appeals for the Ninth Circuit following remand by the Supreme Court. 43 F.3d 1311, 1316 (9th Cir. 1995) (“Our responsibility, then, unless we badly misread the Supreme Court’s opinion, is to resolve disputes among respected, well-credentialed scientists about matters squarely within their expertise, in areas where there is no scientific consensus as to what is and what is not ‘good science,’ and occasionally to reject such expert testimony because it was not ‘derived by the scientific method.’ Mindful of our position in the hier- archy of the federal judiciary, we take a deep breath and proceed with this heady task.”). 52

OCR for page 37
How Science Works to the central imperatives of the other’s way of doing business, and it is likely that neither will be shown in its best light. The Daubert decision is an attempt (not the first, of course) to regulate that encounter. Judges are asked to decide the “evidential reliability” of the intended testimony, based not on the conclusions to be offered, but on the methods used to reach those conclusions. In particular, Daubert says, the methods should be judged by the following four criteria: 1. The theoretical underpinnings of the methods must yield testable predic- tions by means of which the theory could be falsified. 2. The methods should preferably be published in a peer-reviewed journal. 3. There should be a known rate of error that can be used in evaluating the results. 4. The methods should be generally accepted within the relevant scientific community. In reading these four illustrative criteria mentioned by the Court, one is struck immediately by the specter of Karl Popper looming above the robed justices. (It is no mere illusion. The dependence on Popper is explicit in the written deci- sion.) Popper alone is not enough, however, and the doctrine of falsification is supplemented by a bow to the institution of peer review, an acknowledgment of the scientific meaning of error, and a paradigm check (really, an inclusion of the earlier Frye standard).21 The Daubert case and two others (General Electric v. Joiner,22 and Kumho Tires v. Carmichael23) have led to increasing attention on the part of judges to scientific and technical issues and have led to the increased exclusion of expert testimony, but the Daubert criteria seem too general to resolve many of the difficult decisions the courts face when considering scientific evidence. Nonetheless, despite some inconsistency in rulings by various judges, the Daubert decision has given the courts new flexibility, and so far, it has stood the test of time. All in all, I would give the decision pretty high marks.24 The justices ventured into the treacherous crosscurrents of the philosophy of science—where even most scientists fear to tread—and emerged with at least their dignity intact. Falsifiability may not be a good way of doing science, but it is not the worst a posteriori way to judge science, and that is all that’s required here. At least they managed to avoid the Popperian trap of demanding that the scientists be skeptical of their own ideas. 21. In Frye v. United States, 293 F. 1013, 1014 (D.C. Cir. 1923), the court stated that expert opinion based on a scientific technique is inadmissible unless the technique is “generally accepted” as reliable in the relevant scientific community. 22. 522 U.S. 136 (1997). 23. 526 U.S. 137 (1999). 24. For a contrary view, see Gary Edmond & David Mercer, Recognizing Daubert: What Judges Should Know About Falsification, 5 Expert Evid. 29–42 (1996). 53

OCR for page 37
Reference Manual on Scientific Evidence The other considerations help lend substance and flexibility.25 The jury is still out (so to speak) on how well this decision will work in practice, but it is certainly an impressive attempt to serve justice, if not truth. Applying it in practice will never be easy, but then that is what this manual is about.26 25. See supra note 16. 26. For further reading, see John Ziman, PublicKnowledge: An Essay Concerning the Social Dimension of Science (Cambridge University Press 1968). 54