Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
PART II Objectives of Biology Education and Measurement of Achievement
7 Issues in Objectives and Evaluation JAMES T. ROBINSON GOALS, OBJECTIVES, AND OUTCOMES Biology is taken by most students during the high-school years. It is incumbent on us to re-examine why biology is important for most or all students and what we expect the benefits of biology education to be, both for the individual and for society at large. Several issues in the field of goals, objectives, and outcomes of biology education will need to be resolved as the Committee on High-School Biol- ogy Education addresses its tasks. "Scientific literacy" has been espoused as a social imperative for a society affected so importantly by science and technology. The American Academy of Arts and Sciences (1983) devoted an entire issue of its proceedings to elaborate the meanings of scientific literacy. That same year, in Educating Americans for the 21st Century (Na- tional Science Foundation, 1983), the National Science Board Commission on Precollege Education in Mathematics, Science and Technology "found that virtually every child can develop an understanding of mathematics, science and technology if appropriately and skillfully introduced at the elementary, middle and secondary levels." The commission recommended the following criteria for improving high-school science (National Science Foundation, 1983, p. 98~: James T. Robinson is a former executive director for curriculum and evaluation in the Boulder Valley (Colorado) School District. He served as a staff officer for the Biological Sciences Cur- riculum Study. 45
46 HIGH-SCHOOL BIOLOGY · Drastically reduce the number of topics covered in high-school science courses. - ogy? Direct attention toward the integration of the remaining facts, concepts, and principles within each discipline and with other sciences and such areas as mathematics, technology, and the social sciences. · Select ideas that can be developed honestly at a level comprehen- sible to high-school students. · Develop ideas out of experimental evidence that high-school stu- dents can gather or, at least, understand. · Tie ideas into other parts of the course, so that their use can be reinforced by practice. · Let all courses provide opportunities to develop the ability to read scientific materials. These criteria raise issues for biology education that pervade all areas of our concern at this conference. If outcomes are to be determiners of curricula and evaluation, then the other subjects of this conference are derivative from the goals, objectives, or outcomes to be formulated as a major function of the committee. Several questions are proposed for consideration here. Should high- school biology goals and objectives: · Be designed for all students, or should separate courses be devel- oped for students with different interests and goals? · Be formulated in the context of a science and contribute to public understanding of science or as a separate discipline independent of other sciences? · Include the application of knowledge and understandings or be limited to the acquisition of knowledge? · Include attitudes toward science and technology and developing interest in biology and other sciences? · Include ethical and societal issues of science, biology, and technol · Specify the development of problem-solving, critical thinking, and other `'higher-order" thinking skills? manner? Be measurable or assessable in some objective and C`practical" The literature is fairly consistent in an affirmation of positive positions on these questions, but in the classrooms in high schools these issues are not settled at all, in stated objectives, actual practice in instruction, or testing and evaluation. Also, coverage of subject matter dominates instruction (Stake and Easley, 1978~; it is questionable whether retaining the current breadth of coverage will permit students to attain the other outcomes specified above.
ISSUES IN OBJECTIVES AND EVALUATION 47 It is ironic that Educating An~er~cans for the 21st Century lists drastic reduction of content as a major need in high-school science and then, in the statement of outcomes, includes all the major areas currently included in high-school biology. For example, the National Science Board commission, in discussing science education and high-school biology, proposed that scientific education programs in K-12 should be designed to produce the following outcomes (National Science Foundation, 1983, p. 44~: . Ability to formulate questions about nature and seek answers from observation and interpretation of natural phenomena. · Capacities for problem-solving and critical thinking in all areas of learning. · Innovative and creative thinking skills. · Awareness of the nature and scope of a wide variety of science- and technology-related careers open to students of varied aptitudes and interests. · Basic academic knowledge necessary for advanced study by students who are likely to pursue science professionally. · Scientific and technical knowledge needed to fulfill civic responsi- bilities and improve students' own health, life, and ability to cope with an increasingly technical world. · Means to judge the worth of articles presenting scientific conclu- sions. The commission proposed that general biology in high schools should emphasize biology in a social and ecological context. Biology should enable students to attain the following outcomes (National Science Foundation, 1983, p. 98~: Understanding of biologically based personal or social problems and issues, such as health, nutrition, environmental management, and human adaptation. Ability to resolve problems and issues in a biosocial context involv- ing value or ethical consideration. Continued development of students' skills in making careful obser- vatic~ns collecting and analv~in~ data thinking lnnicaliv and critically, and ~-~ ~ D ~--~ ~--- - - -D ~ -my -a -----O --O--~--~ making quantitative and qualitative interpretations. · Ability to identify sources of reliable information in biology that they may tap long after formal education has ended. · Understanding of basic biological principles, such as genetics, nu- trition, evolution, reproduction of various life forms, structure-function relationships, disease, diversity, integration of life systems, life cycles, and energetics.
48 HIGH-SCHOOL BIOLOGY The problems associated with formulating goals, objectives, or out- comes are formidable. First, a national consensus on such a statement would be extremely difficult to attain; and second, evidence seems to sup- port the observation that classroom instruction Is determined more often by the textbook used by teachers than by statements of goals in curriculum guides (Stake and Easley, 1978, pp. 13:59-64~. The issues implied here have included the question of the target population for high-school biology, its range of content, its context (social, technological, scientific), and its attention to application of knowledge and to the inclusion of higher-order thinking skills. Sorting these Issues out Is essential and is related to all the other dimensions of high-school biology. EVALUATION STUDIES The preliminary report by the International Association for the Eval- uation of Educational Achievement (IEA, 1988) presents International comparisons of student achievement. A biology test of 30 items was given to ~velfth-graders In 17 countries. Table 1 shows the numbers of items in the various topics. The U.S. sample taking the biology test was drawn from 43 schools with a total of 659 students taking a second year of high-school biology. There are no U.S. data on first-year biology students, nor for conscience students. Validity of the biology test was measured by three indexes (IEA, TABLE 1 Biological Content Areas and Naumbers of Items Given to Twelfth-Grade Students in 17 Countries Biological Topic No. of Items Transport and cellular material Concept of gene Diversity of life Metabolism of the organism Regulation of the organism Behavior of the organism Reproduction and development, plants Reproduction and development, animals Human biology Natural environment Evolution Total 3 3 2 2 6 30 a rive items, undesignated, were cut from the test given to students in the United States.
ISSUES IN OBJECTIVES AND EVALUATION 49 1988, p. 93~: a curriculum-relevant index (0.76), a test-relevant index (1.00), and a curriculum-coverage index (1.00~. Interpreting the results of the IEA biology test cannot be straightfor- ward, because of several conditions. Five items were dropped from the test given to U. S. students, and "the scores (comparing countries) are presented in percentage frequencies but it must be noted that the United States with 25 items is being compared with other countries with 30 items. The reduced number of items in percentage form will result in a reduced range" (IEA, 1988, p. 46~. A second year of biology may be inferred to be an advanced course for able students, but in the district in which I recently worked, a second biology course is offered for students who do not want to take chemistry or physics, but wish to take more science. I do not know how prevalent this practice is. However, the biology scores are reported as scores of the "elite" (IEA, 1988, p. 73~. The mean achievement of students in the United States for the 1986 administration was 37.9%, with a K-20 reliability of 0.669, which indicates that the items are not very homogeneous in difficulty. The highest national score reported was for Singapore, with a mean of 66.8%. With the limita- tions of the test data, the United States had the distinction of having the lowest mean percentage score on the biology test. The next lowest mean percentage was attained by Italy, with a mean of 42.3%. ~ give you a flavor of the test, one item asked, "What initially determines whether a human baby is going to be a male or a female?" Response options and percentages of U. S. students selecting them were (IEA, 1988, p. 120~: The DNA in the sperm. B. The DNA in the egg. C. The RNA in the sperm. D. The RNA in the egg. E. The DNA and RNA in both sperm and egg. No response. 48.44% 6.00% 9.17% 2.72% 33.25% 0.42% I reviewed Modern Biology (Otto and ldwle, 1985) and Biological Sciences: An Ecolog~calApproach (BSCS, 1982) to find out how they treated the subject. In both books, although they treat the subjects differently, sex inheritance is explained through X and Y chromosomes, and the more extensive presentation of DNA is associated with the function of DNA and RNA in gene action. This linking of DNA and RNA in gene action could have led students to select response E. The preliminary report of the IEA study will be followed by more de- tailed analyses of the test data and other variables not currently processed. The main report will be published in 1989. The National Assessment of Education Progress (Blumberg et al.,
so HIGH-SCHOOL BIOLOGY 1986) piloted the development and testing of higher-order thinking skills in science and mathematics for potential use in future national assessments. Exercises included hands-on activities of students to solve problems. Three modes of administration were used: intact classes with paper-and-pencil tasks, but with materials as stimuli; station activities with students rotating from station to station, each station having apparatus and investigations; and full investigations administered to individual students with an observer using a checklist to record what students did as they performed an inves- tigation. Third-, seventh-, and eleventh-graders were tested in 12 school districts. In one example of a station problem, eleventh-graders were to examine a set of 11 vertebrae, put them into three groups, and explain the similarities of the bones in each group. Cat, rabbit, and dog vertebrae were used. Fifty-four percent of the students were able to place the thoracic, cervical, and lumbar vertebrae into their proper groups. Another 20% grouped all but the atlas vertebra appropriately. Sixty-seven percent of the students provided at least one distinguishing feature for each group of vertebrae (Blumberg et al., 1986, Part II). BIOLOGY TEACHERS Only one recent study was found regarding biology teachers' knowl- edge of biological concepts. This study was reported in Cleveland, Ohio, newspaper, The Plain Dealer (Epstein, 1987), and found that only 12% of biology teachers surveyed correctly defined the modern theory of evolution. This study was based on written responses to items about evolution from 404 Ohio high-school biology teachers, about one-third of the biology-teacher population. Michael Zimmerman, a biology professor at Oberlin College who conducted the study, also found that 37.7% of the teachers surveyed favored teaching creationism and three-fourths felt that creationism was a favorable explanation for the origin of life (Epstein, 1987~. From these two studies and from those reported by other panelists, I believe we can conclude that major reconsideration of the goals and objectives of high-school biology education and of methods of assessing student interests, achievement, and attitudes is important. EVALUATION IN lIIGH-SCHOOL BIOLOGY Schools and such courses as biology are continuously subjected to informal evaluation by their many publics: parents, students, administrators, teachers, scientists, business men and women, and national groups. These informal evaluations carry great weight about the quality of education in each community and in the country as a whole. Efforts to inform these many judgments by more objectives measures and indicators of student
ISSUES IN OBJECTIVES AND EVALUATION 51 achievement have been low-technology, low-budget items. My judgment here is based on comparison of expenditures for accurate instruments for measurement in astronomy, physics, biology, medicine, and space activities. As I looked over evaluation instruments for biology, I saw little change in the last 50 years. A few efforts, such as those of the Educational Testing Service (Dresser and Nelson, 1956) and the Biological Sciences Curriculum Study (Schwab, 1963; Klinckmann, 1970; Mayer, 1978), provided teachers with resources for improving multiple-choice test items in biology. These resources provided sample items for going beyond pure recall and enabling students to demonstrate their capabilities of interpreting experimental data, applying knowledge to novel situations, and interpreting graphed data. More recently, the National Research Council (Raizen and Jones, 1985; Murnane and Raizen, 1988) has broadened the discussion of evaluation to include indicators of quality in science and mathematics education. The major issues in evaluation revolve around purposes and related instruments. Do we want to sort students on test scores similarly to the way we can sort students on height or weight? If so, we have norm- referenced tests (most standardized tests) that are designed to do just that. Norm-referenced tests are constructed, and items selected, to provide a normal distribution with mean and median at the 50th percentile. Most standardized tests are renormed about every 10 years. The new tests may be more or less difficult than the previously normed tests, but the new norms have statistical characteristics similar to those of the old. Another characteristic of the commonly used standardized tests is that they are designed to measure general knowledge and are not directly related to what is taught in any particular classroom. Within the last 20 years, criterion-referenced tests have been devel- oped, especially as part of the "minimal-competence" movement. Criterion- and domain-referenced tests are directly interpretable in terms of a "stan- dard." One problem with these tests is determining what the standard should be, other than in arbitrary ways. A second problem is the desire to make inferences about student competence by generalizing beyond an ability to achieve similar scores on similar paper-and-pencil tests (Haertel, l9SS). This identifies a second issue: "Can a single instrument serve all the purposes desired?" Among the purposes are diagnosis and guiding instruction, rank-ordering students, judging instructional quality, judging curricular quality, forcing curriculum and instruction to move in a particular direction, predicting future performance of individuals, and formulating policies for schools, districts, or states. Another issue is measuring student performance in a way different from the "recognition knowledge" that is assessed in multiple-choice formats. A great deal of interest is developing in generating alternatives to both the
52 HIGH-SCHOOL BIOLOGY commonly used forms of testing. One such alternative is performance testing: assessments that call on the examinee to demonstrate specific skills and competences and to apply them to novel situations (Stiggins, 1987~. Performance assessments have "four basic components: a reason for assessment, a particular performance to be evaluated, exercises to elicit that performance, and systematic rating procedures" (Stiggins, 1987, p. 344. Laboratory work is considered to be an important and necessary means of enabling students to attain the essential goals of biology education, but assessment of any unique contributions of laboratory work is rare (Robin- son, 1979~. Laboratory practicals have been used, but Gallagher (1987) commented that, despite the prevalence of laboratory work in science, we know very little about its effects on high-school biology achievement in the United States. Indeed, both effective and comprehensive evaluation prac- tices and evaluative instruments are a critical need for the improvement of high-school biology. Tamir and co-workers (Tamir, 1974; 1hmir et al., 1982) developed and have placed in use a laboratory practical in the schools of Israel, but evidence of its use outside Israel is lacking. A science-test review panel convened by the National Research Council (Murnane and Raizen, 1988) carefully examined nine science tests. The panel consisted of 12 scientists and high-school science teachers. They made three recommendations to avoid the misuse of science-test results (Murnane and Raizen, 1988, p. 180~: . Results from tests constructed for one purpose . . . should not be used for a quite different purpose. · School or classroom average test scores should not be applied to individuals, and individual test scores should not be interpreted as a rating or ranking of the persons, but only of performance on a test that assesses specific skills. · Test results or tests of the kind reviewed should not be used as the major force driving curriculum and instruction. CONCLUDING REMARKS Goals, objectives, and outcomes and the evaluation procedures used to assess them are two critical aspects of any proposal for policy formulation for high-school biology. I did not mention accountability earlier, but the accountability movement has stimulated the development of evaluation processes and can pressure curriculum and instruction to be concerned with only the aspects of biology that can be easily measured. In many instances, especially with many standardized testing programs, the student is forgotten in the process. It would seem that a first criterion of evaluation programs would be that they have significance to the students themselves.
ISSUES IN OBJECTIVES AND EVALUATION 53 The technology of assessment needs to have infusions of creativity, research, and development. Surely, computers and associated technologies can provide for more useful, instructive, and informative evaluation ~n- formation. Devising more effective evaluation instruments and procedures requires that we be clear and specific about the purposes of biology ed- ucation and the outcomes that we can reasonably be expected to attain with the approximately 134 hours we have to help a very diverse group of adolescents attain the understanding we propose. REFERENCES American Academy of Arts and Sciences. 1983. Scientific literacy. Daedalus 112:Spring issue. Blumberg, F., M. Epstein, W. MacDonald, and I. Mullis. 1986. A Pilot Study of Higher- Order Thinking Skills Assessment Techniques in Science and Mathematics. Final Report. Part II. Princeton, N.J.: National Assessment of Educational Progress. BSCS (Biological Sciences Curnculum Study). 1982. Biological Science: An Ecological Approach. BSCS Green Version. 5th ed. Boston: Houghton Mifflin. Dressel, P. L., and C. H. Nelson. 1956. Questions and Problems in Science. Test Item Folio No. 1. Princeton, NJ.: Educational Testing Service, Cooperative Test Division. Epstein, K C. September 3, 1987. Many Ohio science teachers favor study of creationism. The Plain Dealer. Cleveland, Ohio. Gallagher, J. J. 1987. A summary of research in science education-1985. Sci. Educ. 71:271-455. Haertel, E. 1985. Construct validity and criterion-referenced testing. Rev. Educ. Res. 55:2346. IEA (International Association for the Evaluation of Education Achievement). 1988. Science Achievement in Seventeen Countries. A Preliminary Report. New York: Pergamon Press. Klinckmann, E. 1970. Biology Teachers' Handbook. 2nd ed. New York: John Wiley and Sons. Mayer, W. V. 1978. Biology Teachers' Handbook. 3rd ed. New York: John Wiley and Sons. Murnane, R. J., and S. A. Raizen. 1988. Improving Indicators of the Quality of Science and Mathematics Education in Grades K-12. Report of the National Research Council Committee on Indicators of Precollege Science and Mathematics Education. Washington, D.C.: National Academy Press. National Science Foundation. 1983. Educating Americans for the 21st Century: A Plan of Action for Improving Mathematics, Science and Technology Education for All American Elementaty and Secondary Students So That Their Achievement Is the Best in the World by 1995. A Report to the American People and the National Science Board. Washington, D.C.: National Science Board Commission on Precollege Education in Mathematics, Science and Technology. Otto, J. H., and A. Awls. 1985. Modern Biology. New York: Halt, Rinehart and Winston. Raizen, S. A., and Lo V. Jones. 1985. Indicators of Precollege Education in Science and Mathematics. A Preliminary Review. Report of the National Research Council Com- mittee on Indicators of Precollege Science and Mathematics Education. Washington, D.C.: National Academy Press. Robinson, J. T. 1979. A critical look at grading and evaluation practices. In M. B. Rowe, Ed. What Research Says to the Science Teacher. Vol. 1. Washington, D.C.: National Science Teachers Association. Schwab, J. J. 1963. Biology Teachers' Handbook. New York: John Wiley and Sons.
54 HIGH-SCHOOL BIOLOGY Stake, R. E., and J. Easley. 1978. Case Studies in Science Education. Vol. II. Urbana- Champaign, Ill.: Center for Instructional Research and Curriculum Evaluation and Committee on Culture and Cognition, University of Illinois, Urbana-Champaign. Stiggins, R. J. 1987. Design and Development of Performance Assessments. NCME Instructional Module. Educational Measurement: Issues and Practice. Washington, D.C.: National Council for Measurement in Education. Tamir, P. 1974. An inquiry oriented laboratory examination. J. Educ. Measure. 11:25-33. Tamir, P., R. Nussinovitz, and Y. Friedler. 1982. The design and use of a practical tests assessment inventory. J. Biol. Educ. 16:42-50.
8 Assessing Student Understanding of Biological Concepts CHARLES W. ANDERSON The students quoted below were juniors and seniors in a nonmajors' biology course at Michigan State University. On the average, they had completed 1.9 years of previous biology courses. The first pair of questions, given in multiple-choice form, concern their ideas about sources of energy for plants and animals. Cuestions: A bean plant needs energy to survive and grow. What is (are) the sources of the energy that a bean plant uses? A human also needs energy to survive and grow. Where do you think that a person gets the energy that he or she needs? (Circle all correct.) Student responses: S1: Bean plant: Air, water, sun, soil Person: Air, water, meat, potatoes S2: Bean plant: Air, water, sun, soil Person: Air, water, sun, exercise, meat, potatoes S3: Bean plant: Air, water, sun, soil Person: Air, water, meat, potatoes S4: Bean plant: Water, sun, soil Charles W. Anderson is an associate professor in the Department of Teacher Education, Michi- gan State University. His research focuses on teaching for understanding and conceptual change . . . 1n science earning. 55
56 HIGH-SCHOOL BIOLOGY Person: Air, water, sun S5: Bean plant: Air, water, sun, soil Person: Air, water, sun, exercise, meat, potatoes Not surprisingly, given the array of ideas that the students had about sources of energy, they also had a variety of ideas about energy conversion processes, such as photosynthesis: Question: How do you think that a biologist would define the term "photo- synthesis"? Student responses: S1: The conversion of light to energy. S2: Eking in inorganic material for use in the organism. S3: Changing sunlight energy into useful energy form. S4: The process by which a plant obtains energy by turning sunlight into CO2. S5: The process by which plants convert the sunlight into needed nutrients. The following question was designed to assess students' understanding of energy pyramids. Clearly, most students invoked different concepts. Question: A remote island in Lake Superior is uninhabited by humans, but supports populations of white-tailed deer and wolves. It is left undisturbed for many years. What will happen to the average size of the populations over time? (Multiple-choice predictions, open-response explanations.) Student responses: S1: This question cannot be answered because we have no idea of the amount of deer and wolves on the island and the time. S2: The deer will all die or be killed because of their white tails. The wolves will find it easy to find them. S3: On the average, there will be a few more wolves than deer, because the wolves will kill the deer for food. S4: On the average, there will be many more wolves than deer because wolves are carnivorous and deer would become food source. SS: On the average there will be many more wolves than deer. Survival of the fittest. The final example focuses on students' ideas of how the process of evolution occurs.
ASSESSING STUDENT UNDERSTANDING Question: 57 Cheetahs (large African cats) are able to run faster than 60 miles per hour when chasing prey. How would a biologist explain how the ability to run fast evolved in cheetahs? Student responses: S1: The cheetah's ability to run faster may be influenced by skeletal changes over many years. His legs may have become longer and he may be better adapted for running at high speeds because of his need to do so in order to survive. S2: Since cheetahs are smaller animals they are not very strong compared to other animals such as lions. Since they could not fight off animals effectively enough they have learned to escape their hunter. S3: The cheetah's running ability changed due to its environment. As they evolved they needed to run faster to catch faster animals. The tests from which these responses were taken were pretests admin- istered as part of a project to improve instruction in the course (Anderson et al., in press; Bishop and Anderson, in press). At the beginning of this project, I believed that our system of biology education was, if not working perfectly, at least working. The results of the pretests and posttests were discouraging, though, to someone with those beliefs. The tests quoted were taken from the middle of the stack of tests that I still have in my file folders; they are neither particularly better nor particularly worse than the other tests in the stack. The students' level of performance on the pretests was generally low; I saw little evidence of knowledge beyond that which my 12-year-old daughter is picking up from watching nature shows on television. Furthermore, there were no significant correlations between the level of performance and the amount of previous biology coursework that students had taken (the range was from less than 1 to more than 4 years). Those biology courses did not seem to be doing the students much good. These results led me to the position that I will take and elaborate on in this paper: Most students are not reaming anything useful in high- school biology courses. A few definitions of terms are in order here. By "most students" I mean perhaps the bottom 755b. I do not deny that the best-performing students are learning, and understanding, quite a lot from their biology courses. I define "useful" knowledge as knowledge that helps students do something other than pass tests knowledge that they can use in out-of-school contexts. Although my convictions arise from the experiences described above, they are certainly not the only evidence of the truth of the above assertion. When Yager and Yager (1985) tested students' ability to select correct definitions of terms from the biological and physical
58 HIGH-SCHOOL BIOLOGY sciences, they found evidence that seventh-graders did better than third- graders, but there was no improvement at all between seventh and eleventh grades, the time when most students take high-school biology! In the most recent studies of science achievement by the International Association for the Evaluation of Educational Achievement (1988), American high-school seniors were dead last among students in 13 countries ranked for assessed biological knowledge. These results lead me to two questions about assessment. The first arises from the fact that our present assessment system has declared these students to be successful biology students. They almost all graduated in the top half of their class, and they passed their previous biology courses. Why don't our tests reveal to us how little they are really learning? Second, how can we do a better job? We need to assess biology learning in ways that both give us valid descriptions of students' knowledge and help us to improve the practice of biology teaching. The remainder of this paper addresses these questions. CONCEALING STUDENTS' IGNORANCE Why don't our present assessment procedures, including both teacher- made and standardized tests, do a better job of revealing to us how little students are learning? I do not believe that it is difficult to devise assessment procedures that will reveal students' lack of learning. Almost any question that requires students to write or speak entire sentences (and many questions that do not) will work. However, the demands of producing tests that maximize efficiency and reliability (along with the vested interest that many people have in not seeing how poorly our system is working for non-elite students) have led us to create an elaborate assessment system that could hardly have been better designed to conceal students' lack of knowledge. We could take from our present system a set of object lessons in how to draw attention away from the absence of significant student learning. In particular, our present assessment and reporting procedures incorporate the following practices, each of which helps to obscure the fact that students are hardly learning anything. Don't Give Them Time to Forget During one of our studies at the middle-school level, we had to administer posttests after students had completed each of three units. My colleague Ed Smith was discussing with one of the teachers when we should administer the pastiest for a unit that his class would complete on a Thursday. "You'd better come on Friday," the teacher said. "There's no telling how much they will still remember by Monday." The most
ASSESSING STUDENT UNDERSTANDING 59 discouraging aspect of this story is that the teacher may well have been right. One characteristic of each of the studies cited above is that there was an appreciable delay between the time of instruction and the time of testing. Forgetting is probably partly responsible for the students' poor performance. Does this mean that the tests weren't "fair"? Not at all, if the purpose of testing is to assess useful knowledge. When do we expect the occasions to arise when students will use their biological knowledge? Surely they won't all be in the first week, or the first month, after the relevant concepts have been taught. The fact that deterioration of knowledge is a major problem is a sad commentary on our present biology curriculum, because it isn't a problem for everyone or for everything we teach. I haven't forgotten how to define "photosynthesis," even though my last biology course was before the last biology course of the students quoted above. The students studied reading and writing before they studied biology, but they haven't forgotten that. The difference between the memories that we retain and the ones that deteriorate has a lot to do with the usefulness of the knowledge. We forget or jumble up useless facts, while we remember the concepts, principles, and skills that we use to interpret and operate in the world. It appears that the students quoted above are trying (and usually not quite succeeding) to remember facts, not intellectual tools that they are accustomed to using to interpret the living world around them. Students will appear to remember more of those memorized facts if they are tested right away, but we are being deceived by their performance if we conclude that they have gained useful knowledge. Report Scores in Numerical Form The study that produced the student test responses quoted above was designed to improve a pair of courses that were sometimes taught by me and sometimes taught by colleagues in the Natural Science Department at Michigan State University. I thought that if they just looked at what their students were saying, they would see the need for substantial revision in the courses' curriculum and instruction. I gave one professor a sample of 10 posttests and asked him to look at them. I met with him again several weeks later, and he gave the tests back to me. I asked if he had read them. "No, I didn't," he said. "I didn't know how you wanted them interpreted." This same professor looked regularly at the item analyses for his multiple-choice tests, however, and constructed hypotheses (some of which I believed to be substantially erroneous) about why students missed questions. Numerical data reveal who is doing a little better, or who is doing a
60 HIGH-SCHOOL BIOLOGY little worse, or how students are doing on the average. They do not tell us very much, however, about how students are thinking or what they know. Actual test items (or interview questions) and actual student responses, especially longer written responses that reveal student reasoning, tend to confront the reader with the qualitative reality of students' thinking. Many teachers and policy-makers would prefer to hide behind a veil of numbers that leave some distance between them and this reality. Focus on Efficiency and Reliability at the Expense of Validity Several years ago I attended a colloquium presentation by the person in charge of the design of the science section of the Stanford Achievement Test. He described an elaborate procedure by which the test was devel- oped, moving from objectives to item pools, to item assessment, to the development of alternative forms of the test, and so forth. He used a single test item to illustrate this process, an item that ostensibly tested for student mastery of a "science process skill." It appeared to me that there were at least three ways to arrive at the correct response for this item, two of which did not involve use of the process skill at all. When I asked him how he knew what the item was really testing, he invoked the whole long test development process again. At no point during test development, however, did anyone ever ask a student how he or she arrived at an answer. It seems to me that the above incident revealed some basic differences in our assumptions about the nature and purposes of science achievement tests. This test development process emphasized efficiency and reliability; it produced a machine-scorable test that produced consistent student scores. My concerns, however, focused on validity; I wanted to know what the scores meant. The idea of using interviews to assess the validity of test items and scoring procedures is not original with me. Yarroch (1986) asked students how they arrived at their responses to items in the Michigan Educational Assessment Program science test. He found that students frequently were able to arrive at correct answers through incorrect reasoning. Less often, essentially correct reasoning led students to choose incorrect responses. Norris (in press) reviewed a series of studies using similar methods and generally arriving at similar conclusions: The reasoning that students actu- ally use to arrive at responses to multiple-choice questions may be different from what the test developers assume it is. When test items are not revealed, it is difficult to assess how big a problem this is. Apart from the issue of whether our current assessment procedures actually measure what they purport to measure is the question of whether what they purport to measure is useful knowledge. This question can be addressed at two levels. At one level, there is an issue offace validity: Does
ASSESSING STUDENT UNDERSTANDING 61 there seem to be a reasonable similarity between what we ask students to do on tests and what they might actually do with their knowledge in out-of-school contexts? At a deeper level, there is an issue of construct validity: Do the tests portray biological knowledge and learning in ways that are consistent with current scholarship in philosophy and psychology? On a more practical level, does the information obtained with current assessment procedures help us to develop appropriate policies or improve curriculum and instruction? In each respect, I believe that our current assessment procedures are lacking. With regard to face validity, I would simply observe that in out-of- school contexts, we sometimes speak or write about the living world in sentences. Yet my students have told me that it is possible for a biology major to graduate from Michigan State University without ever having to write a sentence on a test. Once they graduate, they will be expected to use their knowledge differently. Not since my student days has anyone asked me to answer a multiple-choice question about biology. With regard to construct validity, few people would actually claim that biological knowledge consists of a large number of independent and equally important bits. Yet when we give multiple-choice tests and treat the numbers of correct answers as interval data, this is precisely the as- sumption that is built into the technology of assessment. Many tests, both teacher-made and standardized, are, of course, accompanied by elaborate theoretical frameworks that describe biological knowledge in much more complicated terms. In this case, however, the medium often is the message. Students studying for a test, or teachers preparing their students, are likely to ignore the rhetoric and be guided in their preparations by the form of the test itself. DEVELOPING BETTER ASSESSMENTS OF BIOLOGICAL KNOWLEDGE AND LEARNING Criticizing current assessment practices is, of course, much easier than coming up with good alternatives. Nonetheless, a variety of alternative approaches to assessment have been developed. In this section, I will describe an approach that has been developed over the last 10 years by a research group at Michigan State University that includes my colleagues Ed Smith and Kathy Roth, several other professors and graduate students, and me. Other work in this area has been done by Rosalind Driver and colleagues at the University of Leeds (Driver and Erickson, 1983; Driver et al., 1985), James Stewart and colleagues at the University of Wisconsin (Stewart, 1983), and others. The nature of our assessment procedures has been determined by the larger goals that they served. Our research and development program has
62 HIGH-SCHOOL BIOLOGY had two essential goals. First, we have been interested in interpreting classroom instruction in understanding how students and teachers act and tank in classrooms and why some instructional strategies work better than others. Second, we were engaged in developing improved teaching methods and materials. ~ accomplish these goals, we needed rich and detailed descriptions of students' knowledge and thinking that were consistent with our philosophical and psychological understanding. Thus, we were willing to sacrifice some efficiency and reliability for richness of description and construct validity. Developing rich and psychologically sophisticated descriptions of stu- dents' knowledge is not an easy task. For example, although it is relatively easy to see that the students quoted above are deficient in formal biolog- ical knowledge, we wanted to go beyond that. We wanted to understand how they arrived at the responses that they gave. What did they know or believe, from whatever source, that led them to think as they did about the problems that we posed? In our attempts to develop assessment procedures that answered the above question, we drew on scholarship from a number of sources. The first of these was the history and philosophy of science (e.g., Mayr, 1982; Toulmin, 1972), which provided important ideas about the nature of sci- entific knowledge, as well as about the metaphorical similarities between systems of human knowledge and biological systems. A second source was social constructivist psychological theory and work applying it to problems of education (e.g., Collins et al., in press; Rogoff and Lave, 1984; Vy- gotsky, 1962, 1978), which provided ideas about the relationship between individual knowledge and social interactions. Finally, we drew heavily on other work that, like ours, approached problems of science education from a constructivist or conceptual-change orientation (e.g., Driver et al., 1985; Posner et al., 1982; West and Pines, 1985~. Describing Students' Knowledge and Learning In their present form, our assessment procedures consciously draw on biological metaphors (which we believe to be more appropriate than the computer metaphors that prevail in much cognitive scientific work) to describe human knowledge and learning. We think of human knowledge as consisting, like the living world, of many complex, interacting systems that can be characterized in terms of their structure, their functions, and the patterns of their development. Structure All human knowledge even the knowledge of apparently confused students like those quoted at the beginning of this paper is highly struc
ASSESSING STUDENT UNDERSTANDING 63 lured. Like some biological structures, the structures of human knowledge are complex and constantly changing. There is also an analogy in human knowledge to the hierarchically nested nature of biological structures. In particular, human knowledge has social, as well as individual, dimensions. Communities, including communities of scientists, work cooperatively to build knowledge structures that are far larger and more complex than any individual could ever master. The academic disciplines, including biology, are such socially constructed knowledge structures. Describing students' knowledge involves recognizing the complex in- terrelationships among their ideas, including relationships that go outside disciplinary boundaries. Indeed, it appears that formal biological knowl- edge can truly be meaningful to beginning students only if they can relate it systematically to the many ideas about the living world (some correct, some incorrect) that they already had before they began the formal study of biology. Although we find this characterization of the structures of students' biological knowledge to be metaphorically useful and consistent with current scholarship in philosophy and psychology, it suggests that the practical task of describing the knowledge structures of individual students is immensely difficult. Because these knowledge structures are so complex, diverse, and dynamic, in addition to being invisible, we have never seen a useful and practical approach to describing them. Our assessment procedures have therefore avoided attempts to develop complete descriptions of the structure of students' knowledge, relying instead on structural comparisons of the knowledge of different individuals (see the discussion of development below). Functions Biological facts, theories, and principles are not inert "content." They are more like intellectual tools or body organs, in that they have func- tions, as well as structures. In particular, biological knowledge helps us to describe, explain, make predictions about, and control the living world. Each of these functions is a social activity that involves the application of biological knowledge to living systems (see Anderson and Roth, in press). Description, explanation, prediction, and control are not, however, functions exclusively of scientific knowledge. Even young children who have no exposure to formal science instruction engage in these activities by using their personal and cultural knowledge. Biological concepts and principles make it possible for us to engage in these activities with far more power and precision than would otherwise be possible. Thus, for us a critical test of students' understanding is their ability to use biological
64 HIGH-SCHOOL BIOLOGY concepts and principles to describe, explain, predict, and control living systems. (Note that these functions of scientific knowledge are different from what other science educators sometimes refer to as science process skills or scientific thinking skills, in that the functions of scientific knowledge involve the use of existing knowledge, rather than the development of new knowledge.) In our assessment procedures, we recognize the importance of the functions of scientific knowledge in the ways that we specify instructional objectives. The objectives always specify ways in which students should be able to use biological knowledge to describe, explain, predict, and control living systems. Development Biological knowledge is constantly changing, both in individuals and in communities. I am particularly attracted to the analogies that ~ul- min (1972) draws between the development of scientific knowledge and processes of evolution or ecological succession. ~ulmin speaks of an "in- tellectual ecology," in which individual concepts are seen as analogous to populations in an ecosystem. The intellectual ecology of an individual or a scientific community changes gradually, through processes involving both cooperation and competition among concepts. A new concept can "take root" and thrive only if a complex of other concepts on which it depends is already in place; thus, students go through stages of intellectual devel- opment analogous to stages in ecological succession. In the early stages of development, students depend primarily on concepts that are part of our common cultural knowledge base. At later stages, they are able to in- corporate specialized scientific concepts and principles into their individual "intellectual ecologies" (see Posner et al., 1982~. In our work, we have tried to describe this process of intellectual development by drawing comparisons between students at different stages of development. Able 1 and Figures 1 and 2 show different ways that we have used to make those comparisons. Table 1 (from Anderson et al., 1987) is an example of our most common approach: a series of comparisons between common patterns in student thinking (Naive Conceptions) and the ways that we would like them to use scientific knowledge to think about the same issues (Goal Conceptions). Figure 1 (from Bishop and Anderson, in press) and Figure 2 (from Smith and Anderson, 1986) contrast naive conceptions and goal conceptions in diagram form. We regard these conceptual analyses of students' thinking as the most important outcome of our assessment procedures. These procedures do, however, also produce numerical data for the purposes of making com- parisons between different instructional treatments. In general, we report
ASSESSING STUDENT UNDERSTANDING TABLE 1 Respiration Issues and Conceptions 65 Issue Goal Conception Naive Conception Implicit definition of respiration Nature of food Function of food Source of energy Energy transformation Matter transformation Movements of reactants and products Nature of energy Respiration is the process by which all cells obtain energy from food. Food is matter that organisms can use as a source of energy. Food supplies the energy that cells need for life processes. The only source of energy for any organisms is the energy stored in food. Energy stored in food is released in a form that can be used by cells. Food is chemically combined with oxygen to create carbon dioxide and water, accompanied by the release of energy. Food and oxygen are supplied to all cells via the respiratory and circulatonr systems. Carbon dioxide and water are removed from cells by these same systems. Energy changes from one form to another: light~stored energy in food Energy for life processes ~heat. Respiration is breathing which occurs only in animals. Food is the stuff that organisms eat. Food keeps organisms alive. Organisms get energy from many different sources. Food energy is used directly (no notion of energy transformation). Food is digested and excreted. Bergen is changed into carbon dioxide. These two processes are not related to one another. Food goes to the stomach, gets digested and is excreted. Oxygen goes into the lungs and carbon dioxide comes out. (No notion of distribution to cells.) Energy is confused with matter, which contains energy, and gets used up (like fuel). the percentage of students who demonstrated mastery of each goal con- ception (Anderson and Smith, 1986; Anderson et al., in press; Bishop and Anderson, in press). Developing Tests and Analysis Procedures The table and figures are products of a fairly long and complex develop- ment process that produces topic-specific tests and analysis procedures for each test. These procedures are described in detail in the cited references. Briefly, they involve the following steps:
66 E oa ~0 O (5 ~ O \ a) C `,, > ~ ~ t_ i~ UJ C) =Z b_ a CO c Q Q tr C - C,0 a) ~ 4) ~ (d z oO ~ O / - Q Ct >, - ._ o, C a '_ :-#_ = .0 a) Z .- ° a, O ,!1 ~ ~ .0 0 ~, / . - Q \ / ~ C , ~_ ~ .0 _ ~ - 3 Q a) 0 Z ~ .o - ~o C) C ~ ~ a) ~\ {_-` C o O _ n3 ed 41) Q Z ~ 2 cn Eo ~a °~-°= :~§Q / / / a, O ~ > _ _ c 2._ Q ~n cn ~ ~ C o' C C - a, ._ . C O C 3 c ~n ~ a~ ~ Q C CO a, G.) C C C .- n C ~ ~ G) _ 00 'c a) 0 ~ 85 Q C ~ O <d Z _ o ._ {n C ._ O C O O Cl: ·= ~a, 4) _ cn 0 z O U] z o 6 - z UJ cn o z O IL z 111 ~: _ Q .> '_ _ Z 4~'Oro cn 3 ~ 3 ' ~ ~a ,,, > > > O [L C C C o o (~: (15 g ~ o5 m ~ Q O l n c ~ G, ~O ~2 C, c, ,, {~ o o E-' ~ ~ ~8 Z LL ~n y o o . C~ o ~n ~ ~, ~ =- -a' ~ ~ ·~: . o o ~q o~
67 - I 0- C~ ~/~ Gem ~ ~Z HIGH o ~ C ~ ~ ~ , llJ Q LIJ Q L 1 - 1 / In O Ocu ! _ - e', o ~ o .~ A- O .>-o E ~ so Z CD ~ ~ s a) G o s . _ ~ E ~ a) _ = In ~ a: a` O ~ In ~ o 8 . ~ ~ ._ lo: CQ a' 8 ou ._ at Ct ~4 o - to o Cal o ._ I_
68 HIGlI-SCHOOL BIOLOGY · Content anah,~s~s of the topic. We begin with an analysis of the biological knowledge to be tested as it is presented in relevant texts and as we understand it ourselves. Although texts often present biological knowledge as consisting primarily of facts and vocabulary words, we are interested in how students could use biological concepts and principles as intellectual tools that help them to make sense of the living world. Thus, the outcome of our content analysis is a set of behavioral objectives specifying how students should be able to use knowledge of the topic to describe, explain, predict, and control the world around them. · Developing preliminary tests or clinical interviews. The next stage in development is a preliminary exploration of student conceptions. In general, we begin this process by asking open-ended questions that require students to describe, explain, make predictions about, and control relevant living systems in their own terms. For example, early in the development of the photosynthesis test, we asked students to explain how a bean plant gets its food. The question about how cheetahs developed the ability to run fast (quoted at the beginning of this paper) is another example. · Developing questions that focus on cry al tssu es. The early open- ended questions lead us to hypotheses about critical differences between common student conceptions and canonical scientific conceptions. We test those hypotheses by developing and field-testing short-answer and multiple- choice questions that focus directly on the issue of interest. The question about energy for bean plants and humans is an example. The final test contains a mixture of open-response and forced-choice items. · Coding of student responses. After tests are administered, we begin the analysis process by developing and using systems to code student responses for critical characteristics. Sometimes a single response is coded for more than one characteristic. For example, each response to the cheetah question above was coded for three characteristics: whether the origin of new traits is attributed to a random process, such as mutation and sexual recombination, or to the organism's response to the environment; whether diversity in the ancestral population is recognized and assigned a role in the evolutionary process; and what process is suggested to account for population change (see Bishop and Anderson, in press). · Analysts of coded responses. Whether a student has mastered the goal conception for each issue (see Table 1) is assessed by calculating the weighted average of the evidence provided by all relevant questions. In general, a test contains several items relevant to each issue. Thus, the final outcome of the analysis is an assessment of the degree to which each student understands and is able to use each of the scientific goal conceptions.
ASSESSING STUDENT UNDERSTANDING Uses of the Assessment Procedures 69 As stated above, the assessment procedures described here were de- signed to be subordinate to other goals: the evaluation of classroom instruction and the development of improved teaching methods and ma- terials. They have worked well for those purposes. The descriptions of students' knowledge have allowed us to understand their responses to classroom instruction in ways that would otherwise have been impossible (Anderson and Roth, in press; Smith and Anderson, 1984~. Furthermore, they have helped us to develop teaching methods and materials, as well as approaches to teacher education, that are demonstrably more effective than those which currently prevail in school classrooms (see Anderson and Roth, in press; Roth and Anderson, 1987; Roth et al., 1988~. In our more successful development efforts, we have raised the percentage of students showing mastery of goal conceptions from the 0-20% range (when teach- ers used commercial methods and materials) to the 50-80% range (when teachers used the methods and materials that we had developed). CONCLUSION Most current assessment procedures, including both teacher-made and standardized tests, rely on techniques that emphasize efficiency and relia- bility. There are good reasons for this: resources are limited, we want to be fair to all students, and we need accurate and reliable data for policy purposes. However, in emphasizing efficiency and reliability, we have developed an array of assessment techniques that conceal students' lack of learning and that portray biological knowledge and learning in a woefully inadequate and distorted way. This is a tremendous price to pay for efficiency and reliability, inasmuch as the views of knowledge and learning built into our assessment techniques affect the thinking and behavior of teachers, students, and curriculum developers. In so doing, they contribute to the inadequacies and distortions of our present system of biological education. I have briefly described an alternative approach to assessment (not the only one) that sacrifices some efficiency and reliability for construct validity and richness in description of students' knowledge. I believe that it is a good trade. By portraying student knowledge and learning in philosophically and psychologically more sophisticated ways, these assessment techniques focus attention on critical problems in biology teaching and help us to develop solutions to those problems. I believe that other assessment systems would benefit from a similar shift in emphasis.
70 HIGH-SCHOOL BIOLOGY REFERENCES Anderson, C W., and K J. Roth. In press. Itaching for meaningful and self-regulated learning of science. In J. Brophy, Ed. Teaching for Meaningful Understanding and Self-Regulated Learning. Greenwich, Conn.: JAI Press. Anderson, C. W., and E. L. Smith. 1986. Children's Conceptions of Light and Color Understanding the Role of Unseen Rays. Research Series No. 166. East Lansing, Mich.: Institute for Research on Teaching, Michigan State University. Anderson, C. W., K J. Roth, R. E. Hollon, and T. D. Blakeslee. 1987. The Power Cell: Teacher's Guide to Respiration. Occasional Paper No. 115. East Lansing, Mich.: Institute for Research on Teaching, Michigan State University. Anderson, C. W., T. H. Sheldon, and J. DuBay. In press. The effects of instruction on non-majors' conceptions of respiration and photosynthesis. J. Res. Sci. Teach. (also available as Institute for Research on Teaching Research Series No. 164) Bishop, B. A., and C. W. Anderson. In press. Student conceptions of natural selection and its role in evolution. J. Res. Sci. Teach. (also available as Institute for Research on Teaching Research Series No. 165) Collins, A., J. S. Brown, and S. E. Newman. In press. Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L. B. Resnick, Ed. Learning and Knowledge: Issues and Agendas. Hillsdale, NJ.: Lawrence Erlbaum. Driver, R., and G. Erickson. 1983. Theories-in-action: Some theoretical and empirical issues in the study of students' conceptual frameworks in science. Stud. Sci. Educ. 10:37~0. Driver, R., E. Guesne, and A. Tiberghien. 1985. Children's Ideas in Science. Philadelphia: Open University Press. International Association for the Evaluation of Educational Achievement. 1988. Sci- ence Achievement in Seventeen Countries: A Preliminary Report. Elmsford, N.Y.: Pergamon Press. Mayr, E. 1982. The Growth of Biological Thought. Cambridge, Mass.: Belknap. Norris, S. P. In press. Using studies of thinking processes to develop multiple-choice tests of empirical reasoning competence. In D. N. Perkins, J. Segal, and J. F. Voss, Eds. Informal Reasoning and Education. Hillsdale, N.J.: Lawrence Erlbaum. Posner, G., K Strike, P. Hewson, and Vie Gertzog. 1982. Accommodation of a scientific conception: Toward a theory of conceptual change. Sci. Educ. 66:211-227. Rogoff, B., and J. Lave, Eds. 1984. Everyday Cognition. Cambridge, Mass.: Harvard University Press. Roth, K. J., and ~ W. Anderson. 1987. The Power Plant: Teacher's Guide to Photosynthesis. Occasional Paper No. 112. East Lansing, Mich.: Institute for Research on Teaching, Michigan State University. Roth, K. J., C. L. Rosaen, and P. E. Lanier. 1988. Mentor Teacher Project: Program Assessment Report. East Lansing, Mich.: Michigan State University. Smith, E. L., and C. W. Anderson. 1984. Plants as producers: A case study of elementary school science teaching. J. Res. Sci. Teach. 21:685-695. Smith, E. L., and C. W. Anderson. 1986. Alternative Student Conceptions of Matter Cycling in Ecosystems. Paper presented at the annual meeting of the National Association for Research in Science Teaching, San Francisco. Stewart, J. 1983. Student problem solving in high school genetics. Sci. Educ. 67: 523-540. Toulmin, S. 1972. Human Understanding. Princeton, N.J.: Princeton University Press. Vygotsky, L. S. 1962. Thought and Language. Cambridge, Mass.: MIT Press. Vygotsky, L. S. 1978. Mind in Society. Cambridge, Mass.: Harvard University Press. West L. H., and ~ L. Pines, Eds. 1985. Cognitive Structure and Conceptual Change. Orlando, Fla.: Academic Press. Yager, R. E., and S. O. Yager. 1985. The effects of schooling upon understanding of selected science terms. J. Res. Sci. Teach. 22:359-364. Yarroch, W. L. 1986. Content Validity of the 1985 Michigan Department of Education Pilot Science Examination. Paper presented at the annual meeting of the National Association for Research in Science Teaching, San Francisco.
9 The Advanced-Placement Biology Examination: Its Rationale, Development, Structure, and Results WALTER B. MACDONALD The advanced-placement (AP) biology course sponsored by the College Entrance Examination Board (College Board) is a national program that provides an opportunity for high-school students to pursue and receive credit for college-level biology coursework. The program is intended to replace biology courses that would normally be taken at the freshman or sophomore level in college and is based on the premise that college-level material can be taught successfully to able and well-prepared high-school students. The AP biology course is open to any high school that elects to participate; similarly, the AP biology examination is open to any student who wishes to take it. The AP examinations are administered once a year, in May, under standardized conditions at participating schools in the United States and many other countries. Most students take AP examinations in their own schools; others take them in multischool centers. DEVELOPMENT OF THE AP BIOLOGY COURSE AND EXAMINATION The policies-of the AP biology course and examination, like those of the AP courses and examinations In other subjects, are determined Walter B. MacDonald received his Ph.D. in ecology in 1983 from Rutgers University. Since join- ing the Educational Testing Service (ETS) in 1984, he has served as coordinator of the College Board's biology achievement test, as science coordinator of the College Board's Educational Equality Project, and as the College Board's member of ETS's Test Development Document Creation CI,D/DC) project. Currently, Dr. MacDonald is director of test development for the National Assessment of Educational Progress. 71
72 HIGH-SCHOOL BIOLOGY by representatives of College Board member institutions and agencies throughout the country, including public and private high schools, colleges, and universities. The preparation of the course is an ongoing process, and the design of each examination typically begins nearly 2 years before the actual administration. Operational aspects of the examination including the development of materials, scoring, and grading are managed by the Educational Testing Service (ETS). The AP Biology Development Committee, appointed by the College Board, is the "heart and mind" that prepares both the course description and the examination itself. The committee is made up of college professors and high-school AP teachers; these individuals are familiar with the aca- demic standards to which college freshmen or sophomores are held. The committee is the authority for subject-matter decisions that arise in the test construction process. Committee members bring to their tasks knowledge of biology curricula and of laboratory methods; they are cognizant of the abilities and understandings that are critical to mastery of biology and how students might be asked to demonstrate these abilities and understandings. COURSE DESCRIPTION The AP biology course is taught by high-school biology teachers with guidance from Advanced Placement Course Description Biology, a Col- lege Board publication prepared by the Development Committee. The course description provides broad guidelines for the content and skills to be included in the course and offers a recommended set of quantita- tive laboratory exercises. In addition, the publication contains information about the examination, sample questions illustrative of those included in the examination, a list of recommended textbooks, and other materials and resources helpful in preparing and teaching a college-level biology course to high-school students. AP biology teachers also receive assistance in developing and teaching their courses from other publications and from workshops and special conferences. Biology is a dynamic science; over the last few years, many new areas of inquiry have come to the forefront, while others previously emphasized in the discipline have receded. Sensing the need to reassess the content of biology instruction at the college level, the Development Committee surveyed the introductory biology courses at more than 80 colleges and universities across the country. Its primary goal was to obtain current information on what is taught so that the AP biology course syllabus could be revised to reflect collegiate course offerings more closely. On the average, respondents participating in the survey were able to categorize about 99% of what they taught into the 10 major biological categories presented in a questionnaire. The average percentage of course
THE ADYANCED-PLA CEMENT B. OLOGY EXAMINATION 73 time devoted to each of these categories and the average emphasis placed on 72 other subcategories and topics provided the information needed to update the AP biology curriculum. The survey of colleges and universities also affirmed the view that laboratories still are a central part of biology instruction and that certain topics are covered in these laboratories with great frequency. ~ maintain parity between AP biology and college-level courses, the Development Committee has included sample laboratory experiments in the description booklet to augment any other laboratory experiments already taught by AP teachers. Most of the experiments in the booklet are patterned after those often included in colleges. The current AP biology topics, the approximate percentages of emphasis, and the topics of 12 laboratory experiments are outlined in Figure 1. STRUCTURE OF THE EXAMINATION The AP biology examination is 3 hours long and is designed to mea- sure a student's knowledge and understanding of college-level biology. The examination consists of a 90-minute multiple-choice section with 120 ques- tions that examines the learning of representative facts and concepts drawn from across the entire curriculum and a 90-minute, free-response section consisting of four mandatory questions that address broader topics. The number of multiple-choice questions taken from each major topic of biol- ogy reflects the weighting of that topic as designated in the course syllabus. In the free-response portion of the examination, one essay question fo- cuses on molecules and cells, one on genetics and evolution, and two on organisms and populations. Any of these four questions may require the student to analyze and interpret data or information drawn from laboratory experiences, as well as lecture material; to design experiments; and to demonstrate the ability to synthesize material from several sources into a cogent and coherent essay. To allow students to show their mastery of lab- oratory science skills and knowledge, some questions in the multiple-choice section and one of the four essay questions may reflect the laboratory work and the objective associated with the AP biology laboratory exercises. The multiple-choice section of the examination counts for 60% and the free-response section 40% of the student's grade. In order to provide maximal information about differences in students' achievements in biology, the examinations are intended to have average scores of about 50-60% of the highest possible score for the multiple-choice and free-response sections. Using questions written by college faculty and AP teachers, the ex- amination is assembled by ETS consultants to both content and statistical specifications. Each examination contains both new questions and a set of questions that have been included on previous examinations. The set of
74 TOPICS I. Molecules and Cells A. Biological chemistry 1. Review of atoms, molecules, bonding, pH, water 2. Carbon, functional groups 3. Carbohydrates, lipids, proteins, nucleic acids 4. Chemical reactions, free-energy changes, equilibrium 5. Enzymes: coenzymes, cofactors, rates of activity, regulation B. Cells 1. Prokaryotic and eukaryotic cells 2. Plant and animal cells 3. Structure and function of cell membranes 4. Structure and function of organelles, subcellular, components of motility, cytoskeleton Cell cycle: Mitosis, cytokinesis C. Energy transformations 1. ATP, energy transfer, coupled reactions, chemiosmosis 2. C and C photosynthesis 3. G~ycolysis4 fermentation, aerobic respiration 11. Genetics and Evolution A. Molecular genetics 1. DNA: structure end replication 2. Eukaryotic chromosomal structure, nucleosome, transposable elements 3. RNA: transcription, mRNA editing, translation 4. Regulation of gene expressions 5. Mutations 6. Recombinant DNA, DNA cloning, hybridization, DNA sequencing 7. DNA and RNAviruses B. Heredity 1. Melosis 2. Menders laws, probability 3. 4. C. Evolution 1. Origin of life 2. Evidence for evolution 3. Natural selection 4. Hardy-Weinberg principle, factors influencing allelic frequencies 5. Speciation: isolating mechanisms, allopatry, sympatry, adaptive radiation 6. Patterns of evolution, gradualism, punctuated equilibrium Organisms and Populations A. Principles of taxonomy and systematics, five-kingdom system B. Survey of Monera, Protista, and Fungi C. Plants 1. Diversity; classification, phylogeny, adaptations to land; alternation of generations in moss, fern, pine, and flowering plants 2. Structure and physiology of vascular plants 3. Seed formation, germination, growth in seed plants 4. Hormonal regulation of plant growth 5. Plant response to stimuli: tropisms, photo-periodicity D. Animals 1. Diversity; classification, phylogeny, survey of acoelomate, pseudocoelomate, protostome, and deuterostome phyla 2. Structure and function of tissues, organs, and systems (emphasis on vertebrates), homeostasis, immune response 3. Gametogenesis, fertilization, embryogeny, development 4. Behavior E. Ecology 1. Population dynamics, biotic potential, limiting factors 2. Ecosystem and community dynamics: energy flow, productivity, species interactions,succession, biomes 3. Biogeochemical cycles Inheritance patterns: chromosomes, genes, alleles, interactions Human genetic defects 111. [IIGH-SCHOOL BIOLOGY PERCENTAGE GOALS LAB TOPICS 7% 10% 8% 9% 8% 8% 1% 2% 15% 7% 10% 5% 1% 25% 25% 50% 1. Diffusion and Osmosis 2. Enzyme Catalysis 3. Mitosis and Melosis 4. Photosynthesis 5. Cell Respiration 6. Molecular Biology 7. Genetics of Drosophila 8. Population Genetics and Evolution · 9. Transpiration 23% 9% 10. Physiology of the Circulatory System 11. Habitat Selection 12. Dissolved Oxygen and Primary Productivity FIGURE 1 College Board's advanced-placement biology course and laborato~y syllabus. From College Entrance Examination Board, 1988.
THE ADVANCED-PLACEMENT BIOLOGY EXAMINATION 75 previously used questions, called the equating set, is a "mini-test" assembled to both the content and statistical specifications for AP biology. The use of an equating set enables a new test to be equated to past tests. The analysis of student performance on an equating set allows statisticians to predict how previous AP students would have performed on a new examination or how the current AP students would have performed on past examinations. All the new questions used on a test are pretested on college students across the nation. The use of both pretested new questions and the equating set provides the statistical data to maintain an examination that is appropriate for college-level biology while keeping the level of examination difficulty relatively constant from year to year. While the multiple-choice section of the examination is machine- scored, the free-response section is hand-scored by over 100 readers chosen from among college and high-school biologists nationwide who are actively involved in introductory college-level biology courses or an AP equivalent. The training of readers ensures uniformity of grading and strict adherence to carefully developed standards. All essays are graded on a 10-point scale. The free-response score and the multiple-choice score are weighted and summed to produce a composite score with a 150-point maximum. Students are then assigned a grade of 5 to 1 based on a detailed analysis of the total scores for all students, on equating data from previously tested AP biology students, and on correlation checks to ensure test reliability. A score of 5 indicates that a student is extremely well qualified to pursue upper-level college biology courses, whereas a grade of 3 indicates average preparation. RESULTS Over the last 10 years, the number of students taking the AP biology examination has increased by about 11% per year. In 1988, about 31,000 students took the examination, compared with about 11,000 in 1978. In 1988, over 3,000 schools offered an AP biology course. That year, scores were sent to more than 1,000 colleges across the country. The results by sex, grade, type of school, and ethnicity are displayed in Able 1. Over the years, the AP biology examination has maintained a relatively constant level of difficulty. The data from the equating set tend to indicate that recent populations of AP biology students are slightly less able than past populations. The mean score has steadily declined from 3.35 in 1981 to 3.05 in 1988. The reported grades for AP students since 1981 are displayed in Figure 2. For 1988, 25.2% of the students scored 3, 23.5% scored 2, 23.4% scored 4, 15.4% scored 5, and 12.5% scored 1; thus, 64% scored 3 or higher. Over the last 8 years, the percentage of students who received a score of 4 has remained relatively constant, the percentage at 5 has slightly
76 HIGH-SCHOOL BIOLOGY TABLE 1 1988 National Summary Data for Biology AllFemaleMale11th12th StudentsStudentsStudentsGradeGrade Total N30,61215,65314,9598,61319,320 Mean3.052.873.233.143.01 Black Students N1,165770395294819 Mean2.172.072.962.242.12 White Students N22,09911,40510,6946,03614,408 Mean3.042.873.093.093.02 Asian Students N3,8751,7462,1291,2942,104 Mean3.393.233.513.523.33 Hispanic Students N872449423241559 Mean2.562.372.762.622.51 aData from College Entrance Examination Board, 1987. decreased, the percentage at 3 has greatly decreased, and the percentages at 2 and 1 have increased. The AP biology program has experienced tremendous growth over the last few years. There now are more students earning scores of 5, 4, and 3 than in the past. Unfortunately, many more students are earning scores of 2 or 1. This increase in the percentages of students at 2 and 1 may be due to the addition of many new schools with novice AP biology teachers. While it is rewarding to teach a college-level course in high school, it is not easy. Often it takes time for the novice AP biology teacher to develop the skills, level of preparedness, and enthusiasm typical of the veteran AP biology teacher. What topics do students who score a 2 or 1 not fully understand? A review of the multiple-choice questions on recent examinations shows that many of these students fail to comprehend such basic topics as os- mosis, plant-animal cell differences, function of cell organelles, differences between photosynthesis and respiration, DNA replication, RNA transcrip- tion, meiosis, inheritance patterns, natural selection, blood circulation, digestion, antigen-antibody relationships, and phylogenetic relationships. Most students who score a 3 show average understanding of these topics, whereas students who score 4 or 5 exhibit excellent understanding of these topics and many others. In 1987 and 1988, many of the students who scored a 2 or 1 could not score more than 1 on essay questions that asked them to:
THE ADVANCED-PLACEMENT BIOLOGY EXAMINATION Describe the production and processing of a protein that will be exported from a eukaryotic cell. Begin with the separation of the messenger RNA from the DNA template and end with the release of the protein at the plasma membrane [Educational Testing Service, 1987~. Or Discuss Mendel's laws of segregation and independent assortment. Explain how the events of meiosis I account for the observations that led Mendel to formulate these laws [Educational Testing Service, 19883. Or Discuss the processes of cleavage, gastrulation, and neurulation in the frog embryo; tell what each process accomplishes. Describe an experiment that illustrates the importance of induction in development [Educational Testing Service, 1988~. 77 Most students who scored a 3 could adequately answer these essay ques- tions, whereas the students who scored a 5 or 4 were more likely to write more elegant and complete answers. 40 35 30 25 20 llJ 15 10 Threes Fives ,~ Ones - , we O 1 1 1981 1982 1983 1984 1985 1986 1987 1988 1 1 ~ YEAR FIGURE 2 AP biology reported scores, 1981-1988.
78 HIGH-SCHOOL BIOLOGY SUMMARY Generally, the state of the AP biology program and the status of AP biology students are vein good. The majority of students are receiving a sound college-level course while attending high school. Validity and longitudinal studies have indicated that AP students perform as well as and often better than students taking the college course. Another important finding is that AP students tend to demonstrate higher achievement in college than their non-AP counterparts (Casserly, 1986~. While one might expect AP candidates to show higher achievement in college than non-AP students because AP candidates are, in general, more able students, many AP candidates are placed in higher-level, more demanding courses when they reach college. Studies also have shown that 90% of the students who were placed ahead felt well prepared for the advanced sequence of college-level courses in which they then enrolled (Casserly, 1968~. Other longitudinal studies have shown good correlation between scores on the AP biology examination and subsequent grades in introductory and upper-level biology courses in college (Willingham and Margaret, 1986~. An area of current and future concern is the increasing percentages of students who score 1 or 2 on the examination. It Is hoped that the teacher preparation required to teach a college-level course In high school will catch up with the swelling population of AP students and precipitate a decrease in the percentages of students receiving scores of 1 and 2, while increasing the percentages of students receiving scores of 3, 4, and 5. While participation by m~nor~-group students in AP courses has increased over the last few years, a greater effort should be made to increase the participation of black and Hispanic students In AP biology courses. AP courses are a rewarding challenge that should be made available to all able students. REFERENCES Casserly, P. L. 1968. What college students say about advanced placement. College Board Rev. 69: Fall. Casserly, P. L. 1986. Advanced Placement Revisited. College Board Report 86-2. New York: College Entrance Examination Board. College Entrance Examination Board. 1987. Advanced Placement Program National Summary Reports. New York: College Entrance Examination Board. College Entrance Examination Board. 1988. Advanced Placement Course Description- Biology May 1989. New York: College Entrance Examination Board. Educational Testing Service. 1987. The 1987 AP Biology Examination. Princeton, N.J.: Educational Testing Service. Educational Testing Service. 1988. The 1988 AP Biology Examination. Princeton, N.J.: Educational Testing Service. W'llingham, W., and M. Margaret. 1986. Four Yeam Later: A Longitudinal Study of Advanced Placement Students in College. College Board Report 86-2. New York: College Entrance Examination Board.
1 n The Development of Interest in Science JON D. MILLER There Is broad agreement in the United States that scientific literary is a good thing, that we don't have enough of it, and that it is especially Important for our young people to have a lot more of it. There Is also broad agreement that schools are the place where young people should get their scientific literacy and that formal institutions of education are failing to produce a minimal level of scientific literacy In an acceptable proportion -- r- _ _ of our young people. Most Americans, to borrow from the Declaration of Independence, find these truths to be self-evident. As educators and scientists, however, we cannot accept truths as self-evident, but must seek to understand better the roots of scientific literacy and the social, economic, and political consequences of scientific illiteracy. Having spent the last several years studying data about young people's knowledge of and attitude toward science and having talked with a large Jon D. Miller is professor of political science and director of the Public Opinion Laboratory, Northern Illinois University. He received an A.B. from Ohio University, an M.A. from the Uni- versity of Chicago, and a Ph.D. in political science from Northwestern University. Dr. Miller directs research focused on the development of political and social attitudes in young adults and adults and on political behavior in democratic systems. He directs a major longitudinal study of the development of interest and competence in science and mathematics among middle-school and high-school students in the United States. The Longitudinal Study of American Youth, including the work reported in this paper, is sup- ported by National Science Foundation grant MDR-8550085. All the analyses, opinions, and conclusions offered are those of the author and do not necessarily reflect the views of the Na- tional Science Foundation or its staff. 79
80 HIGH-SCHOOL BIOLOGY number of students and teachers, I am convinced that functional scientific literacy requires some level of formal science and mathematics instruction. Informal learning programs like museums and television shows can augment formal instruction and stimulate interest in it, but these informal experi- ences cannot effectively replace or substitute for formal science instruction. Further, it is clear to me that functional scientific literacy requires the ability to read about science and technology to be able to sustain literacy in the decades after the end of formal instruction. The basic problem is that formal instruction in science and mathemat- ics has become voluntary in most American high schools and that attitudes have developed that discourage the vast majority of young Americans from attempting formal coursework in chemistry, physics, and mathematics be- yond first-year algebra. Only 15% of last year's American high-school graduates had completed a physics course during their high-school experi- ence, and only 30% had taken a chemistry course. Forty-five percent had avoided any contact with algebra throughout their 4 years of high school. And these figures apply only to students who graduated, excluding the sizable proportion that dropped out before graduation. Further, the data indicate that young women avoid science and mathematics at almost double the rate of young men. The problems are serious. As scientists and educators, we must ask why so many young Americans decide not to study science and mathematics during their high-school years, and it is this question that has driven most of my recent work in this area. It is critically important that we come to understand the reasons for this pattern of science and mathematics avoidance. The British government recently addressed this problem by mandating that all British students take science and mathematics every year that they are in school and that they be tested through a national testing program to measure results. Compulsion is one solution, but with 16,000 independent school boards in the United States, compulsion is not an alternative available to us, regardless of its merits. If we are to do a good job with science and mathematics education in the United States, we must first understand the root sources of the attitudes of young Americans toward science and mathematics and seek to address those issues effectively. The Longitudinal Study of American Youth (LSAY) is one effort to understand better the process of socialization and development of attitudes toward science and technology and citizenship. The LSAY builds on a previous cross-sectional study by Miller et al. (1980) and on the relevant literature. The LSAY will follow a national sample of seventh-graders and a parallel sample of tenth-graders for the next 4 years, collecting data from the students, their parents, their teachers, and related school staff. The base-year student data collection for the LSAY was completed during the 1987-1988 school year.
THE DEVELOPMENT OF INTEREST IN SCIENCE MEASUREMENT OF INTEREST IN SCIENCE 81 One approach to the problem of student avoidance of science is to examine the general attitude of students toward science. Apart from courses or specific encounters with science, most students have a general attitude or disposition toward science. The data collected by the LSAY provide an opportunity to construct a unidimensional measure of attitude toward science and to seek to understand the factors that contribute to fostering that attitude. The base-year LSAY data collection included a series of items designed to tap each student's general attitude toward science. The full set of items was examined by both factor analysis and reliability tests, and the following five agree-disagree items were identified as a unidimensional measure of attitude toward science: . I enjoy science. I am good at science. I usually understand what we are doing in science. Science is useful in everyday problems. I will use science in many ways as an adult. Each student was asked to strongly agree, agree, disagree, or strongly disagree with- each of these items. An analysis of the marginal distribution of these items found a generally positive attitude toward science (see Bible 1~. A solid majority of high-school sophomores agreed that they liked science and felt that they understood it, but just over one-third thought that science would be helpful in their adult activities. The combination of these five items constitutes an index of general attitude toward science. The index is simply the number of agreements (strong or regular) with this set of items. The distribution of the students across the range of 0-6 was relatively even, with about one-third of the students scoring 0 or 1 on the index, one-third scoring 2 or 3, and one-third scoring 4 or 5. The mean score was 2.4, and the median score was 3. It is likely that students scoring high on this index will be more likely to take advanced science courses and to engage in more informal science learning activities than students who score low. While we will have to await the second and third cycles of the LSAY for the individual change data to test that hypothesis, it is possible to use the base-year data to understand better the influence of home, school, and each student's life goals on his or her general attitude toward science. SOME FACTORS ASSOCIATED WITH ATTITUDE TOWARD SCIENCE While the aggregate distribution of student attitudes toward science is interesting, it is important to know more about which students hold
82 HIGH-SCHOOL BIOLOGY TABLE 1 Distribution of 2,829 Tenth-Grade Students on Five Attitude Items, 1987 Strongly Not Strongly Agree Agree Sure Disagree Disagree (%) (%) (%) (%) (%) I enjoy science. 17 40 14 19 9 tam good at science. 13 41 20 18 7 I usually understand what we are doing in science. 12 47 17 15 6 Science is useful in everyday problems. 8 29 35 20 6 I will use science in many ways as an adult. 11 24 39 16 9 more positive attitudes and which students hold more negative attitudes. We would expect that students who hold positive attitudes toward science would be most likely to enroll in advanced science courses and to carry more information from their courses into adulthood. We cannot test those hypotheses until we have obtained additional cycles of measures from the LSAY sample, but we can examine some of the characteristics associated with holding a positive attitude toward science among high- school sophomores. The most proximate source of influence on a student's general attitude toward science might be expected to be the science course in which he or she is enrolled. While students like all citizens-experience a wide array of technologies in their daily lives, it is primarily through formal science classes that students encounter science. Of course, some students may also experience science in science museums or on television or in books, but for most students, those experiences are far less frequent than class experiences. As a starting point, it is useful to recall that most high schools require only 2 years of science and that not all tenth-grade students are enrolled in a science course. Most high schools offer students some choice in science courses, and some students elect to take a general science course while others move directly into biology. An examination of the course-taking patterns of the LSAY sophomore cohort found that 84% were enrolled in a science course and that 59% had enrolled in a biological science course, which is almost universally taught as a laboratory course at the high-school level. An additional 13% were enrolled in chemistry, and 1% were in a physics course; both are usually taught as laboratory sciences. Seven percent of sophomores were enrolled in a physical-science course, and 4%
THE DEVELOPMENT OF INTEREST IN SCIENCE 83 in a general science course, neither of which normally involves extensive student laboratory work. An examination of course-taking patterns by student sex, student edu- cational expectations, and parental educational achievement indicated that sophomores with clear intentions to complete at least a baccalaureate were significantly more likely to be enrolled in chemistry than students without college. aspirations. Sophomores not planning to go to college were more likely to be enrolled in general science, physical science, or no science at all. There appears to be no significant sex difference when parent education and student educational aspiration are held constant. Beyond enrollment, it is important to know- what each student thinks about the science courses to which he or she is exposed. In the LSAY, each student was asked to list each course that he or she was taking in the fall semester and to rate each course on eight dimensions (interest in subject of course, clarity of teacher, clarity of textbook, difficulty of course, whether course challenged student to think, likely utility of course to student's expected occupation, use of computers, and number of hours of homework each week). An examination of the data from the tenth- grade cohort indicated a substantial level of variance in these measures, suggesting that students were able and willing to differentiate among the different facets of each course and among the courses they were taking. A factor analysis indicated that two of the dimensions captured a general attitude toward the course. One dimension concerned the student's interest in the subject matter of the course. A second dimension concerned each student's perception of the likely utility of the course in his or her career. As to interest in subject matter, students were asked to grade their level of interest in letter-grade terms. An A denoted a high level of interest in the subject matter, a C denoted an average level of interest, and an F denoted little or no interest. As to perceived utility, the same grade-card scoring was employed, with A meaning that the student thinks the course will be very useful in his or her career and F meaning that the course would be of no use. All five letter grades were available for use. For this analysis, I have converted these responses into traditional grade-point averages (GPAs), assigning 4 points for an A, 3 for a B. and 0 for an F. The index of attitude toward science course is the mean grade given by each student on the interest and utility dimensions. Using this index, the tenth-grade cohort appears to hold generally positive attitudes toward their science courses. Using the same parent education, student aspiration, and sex context used to examine enrollment, the LSAY data indicate that high-school sophomores assign a B- to their science courses (see Able 2~. In contrast to the 2.7 GPA assigned to science courses, the same sophomore cohort assigned GPAs of 3.1 to English, 2.9 to mathematics, and 2.5 to social studies. Science courses, it would appear,
84 HIGH-SCHOOL BIOLOGY TABLE 2 Evaluation of Science Courses by a National Sample of Public-High-School Sophomores, 1987 Mean Score for Student's Parents' Expected Student's Other Education Education Sex Biology Chemistry Science High schoolLess than Male 2.2 (226) 2.3 (12) 2.6 (87) orlesscollege Female 2.3 (206) 2.5 (21) 2.3(82) College Male 2.6 (128) 2~8 (30) 2~6 (34) degree Female 2.6(142) 2~9 (35) 2.6 (27) Graduate Male 3~2 (76) 2~8 (32) 2.4(10) degree Female 3.1(134) 3~1(44) 2~9 (16) CollegeLess than Male 1.8 (28) a (3) a (8) degreecollege Female 2~4 (43) - (2) 2.5 (19 College Male 2.7 (87) 3.1(30) 2.9 (24) degree Female 2.3 (86) 3-0(35) 2.3 (20) Graduate Male 3.1(94) 3.2 (47) 2~9 (16) degree Female 3~0 (98) 2~8 (45) 3.3 (18) All public-high-school sophomores 2.6 (1,348) 2.9 (336) 2.6 (361) -aToo few cases available to calculate a reliable mean. are viewed by sophomores more positively than social studies and less positively than English or mathematics. All four distributions, however, approximate normality, suggesting that some students hold very positive and very negative attitudes toward all four course areas. Looking at the distribution of student attitudes toward science courses within the same parent education, student aspiration, student sex frame- work used to examine science-course enrollments, it appears that students expecting to complete a graduate degree hold the most positive attitudes toward science and that there are no systematic sex differences. While these multivariate tabulations are helpful in providing general impressions of the influence of each of these variables on the distribution of student attitudes toward science courses or toward science generally, we would like to know both the absolute and relative influence of each of these (and perhaps other) variables on student attitude toward science. It is possible to obtain a more precise measurement of the relationship of these and other background variables to student attitudes through the construction of a path model.
THE DEVELOPMENT OF INTEREST IN SCIENCE A MULTIVARIATE MODEL 85 The primary variable of interest to us is each student's general attitude toward science. A five-item index of student attitude toward science was introduced above, and the multivariate model will seek to understand the influence of several independent variables on the distribution of this attitude. The trichotomous distribution of the index described earlier will be used in this model. In the preceding section, we identified student attitude toward science courses as the most proximate independent variable and looked at the distribution of student attitudes toward science classes. For this model, a single index of attitude toward science courses has been constructed, using the mean value of the attitude toward the science class in which the student is enrolled. In a very few cases, students were enrolled in more than one science class simultaneously, and in those cases the mean rating of the more advanced science class was used in the index. Approximately 400 sophomores were not enrolled in any science class, and they have been dropped from this analysis. Among the approximately 2,800 sophomores enrolled in a science course, 43% gave the course a C or lower, 29% a B. and 29% an ~ In the preceding tables, we have examined the distribution of student attitudes by the level of parental education, the level of education each student expects to complete, and the sex of the student. All three of these variables will be retained in the construction of a path model. The level of parent education will be dichotomized into less than a baccalaureate and the completion of a baccalaureate or more; 32% of the parents included In this analysis held a baccalaureate or more. The level of education expected by each student will be dichotomized into less than a baccalaureate and a baccalaureate or more; 66% of the sophomores included in this analysis expect to earn a baccalaureate or more. Fifty-two percent of the students in this analysis are female. ~ explore more fully the impact of family practices and values on each student's attitude toward science, two additional variables will be added to the model. The first variable seeks to measure the degree to which parents encourage-or push science. Each LSAY student was asked to mark a series of statements about his or her parents' attitudes and behaviors. A factor analysis of this battery of items identified five items that characterize parent science push. The index of parent science push is the number of student agreements with the following statements: My parents want me to learn about computers. My parents have always encouraged me to work hard on science. My parents buy me mathematics and science games and books. My parents expect me to do well in science.
86 HIGH-SCHOOL BIOLOGY My parents think that science is a very important subject. For this analysis, the index of parent science push was dichotomized into parents who were reported by their student to do three or more of the five activities and parents who were reported to do fewer. Forty percent of the parents in the study were reported to do three or more of the science push activities. The second family variable concerns the religious values of the parents. One parent from each LSAY family was interviewed by telephone in the spring of 1988, during the second semester of each student's sophomore year. A small set of religious-value questions were asked and subsequently used to create a typology of religious values. Parents who agreed with both the following statements were classified as religious conservatives: There is a personal God who hears the prayers of individual men and women. The Bible is the actual word of God and is to be taken literally, word for word. Parents who disagreed with one or both of these statements were clas- sified as religious moderates or liberals. For this analysis, parents were dichotomized into religious conservatives and others. Fifty-seven percent of LSAY parents were classified as religious conservatives. If there is a perceived conflict between science and religious values, we would expect to find it occurring most often among religious conservatives. The inclusion of student course attitude, student educational aspi- ration, parent science push, parent religious attitudes, parent education, and student sex in a single model allows the exploration of the relative influence each of these measures in the context of the relative impact of all the other variables-on each student's general attitude toward science. A path model (Goodman, 1978; Fienberg, 1980) is a convenient method of looking at these relationships and displaying the results in a relatively comprehensible format. The path model to predict students' general attitude toward science indicates an interesting network of direct and indirect influences (see Figure 1~. The variables are placed in an approximate temporal order. The current student attitude toward science is the object of our concern and the predictive object of the model. Student course experience is the most proximate independent variable and is placed closest to the dependent variable. Student educational aspirations may be thought of as having been formed before the immediate experience of courses and as being somewhat longer-standing in nature. This variable, therefore, is placed to the left of student course attitude, but to the right of the other variables. Similarly, parent science push may be viewed as of longer standing and likely before the formation of student educational aspirations. Student
THE DEVELOPMENT OF INTEREST IN SCIENCE \ W~ ,~ .40 _~VV .55 ~0, / - ~ ' \ \ - ~.50 87 ~SCIENCE) .07- ` ATT J - FIGURE 1 A path model to predict student attitude toward science, 1987. sex, parent education, and parent religious attitudes are considered as background variables that have been extant for most, if not all, of each student's life. These three variables are placed on the far left of the model. Looking at the direct paths, the model indicates that four variables have a direct influence on student attitude toward science. The strongest path comes from student attitude toward his or her science course. The path coefficient is Goodman's (1972) coefficient of multiple-partial determination (CMPD), and the value of .50 indicates that 5055 of the mutual dependence in the direct model can be attributed to student course attitude. (CMPD is a proportional reduction-of-error measure. The CMPD uses the difference between the number of likelihood-ratio chi-squares in the independence model and any other model to measure the improvement in estimation attributable to any given model. Goodman suggests that the CMPD is analogous to a multiple R2in a regression model. Goodman suggests that in the analysis of ordinal and nominal variables, it is preferable to use the term "mutual dependence" to refer to the deviation from true independence. When two variables are unrelated, we refer to them as being independent. When two variables are not independent, Goodman refers to them as being mutually dependent and measures this mutual dependence in units of likelihood-ratio chi-squares.) In contrast, parent science push accounts for only 756 of the mutual dependence in the direct model. Student sex and student educational aspiration account for even smaller portions of the mutual dependence in the direct model. Parent education and parent
88 HIGH-SCHOOL BIOLOGY religious attitudes do not have a direct path to student general attitude toward science. It is important to understand that the direct paths are residual paths, in that these relationships express the influence of each of the independent variables on the dependent variable, holding constant the direct and indirect influence of all the other independent variables included in this analysis. The best way to understand this point is to look at an example. A review of the influence of parent education will be helpful. Beginning at the left side of the model, the path coefficient of .40 between parent education and parent science push indicates that college-educated parents are significantly more likely to push science with their children than are non- college-educated parents. Similarly, the coefficient of .55 between parent education and student educational aspiration indicates that the children of college-educated parents are significantly more likely to plan to earn a baccalaureate than are the children of other parents. In short, better- educated parents foster higher educational aspirations and push science with their children, and the sizes of the two coefficients indicate that both these relationships are strong. Following this network of influence, parent science push is strongly associated with student course attitude, accounting for 36% of the mutual dependence in the prediction of student course attitude. Parent science push is significantly associated with the level of student educational aspi- ration, but accounts for only 11% of the mutual dependence in predicting student educational aspiration. The level of student educational aspiration is associated with student course attitude, accounting for a quarter of the mutual dependence in the prediction of student course attitude, which we noted earlier is the strong direct predictor of student general attitude toward science. Looking at the whole network of direct and indirect influences, it is clear that the level of parent education does play a significant but indirect role in influencing student general attitude toward science. The influence of parental education in fostering higher educational aspirations and in pushing science creates attitudes and goals that are conducive to liking science courses, which, in turn, appears to be strongly associated with a positive general attitude toward science. The linkage is indirect, but important. Although the residual direct influence of student sex was small, an examination of the indirect paths indicates substantially greater influence. The path coefficient of-.19 between student sex and parent science push indicates that male students are more likely to have reported that their parents engaged in science push activities than are female students. In subsequent analyses of the parent interviews, we will explore the parent reports of science pushing activities, but for this analysis, these results
THE DEVELOPMENT OF INTEREST IN SCIENCE 89 indicate that student sex accounts for about 19% of the mutual dependence in the level of parent science push, with sophomore boys reporting the higher level of parent science pushing activities. In contrast, the coefficient of .07 between student sex and student educational aspirations indicates that sophomore girls are more likely to plan to earn a baccalaureate than sophomore boys, accounting for about 7% of the mutual dependence in the prediction of student educational expectations. The absence of a path between student sex and student course attitude means that there was not a significant difference in science course attitudes between sophomore boys and girls, holding constant parent education, parent religious attitude, parent science push, and student educational aspiration. The path mode indicates that there are significant, but weak, rela- tionships among parent religious attitude, parent science push, and student educational aspirations. The-.10 coefficient between parent religious at- titude and parent science push means that parents who hold conservative religious views are slightly less likely to push science than are parents holding moderate or liberal religious views. The-.07 coefficient between parent religious attitude and student educational aspiration means that the students of parents with conservative religious views are slightly less likely to plan to complete a baccalaureate than other students, holding constant parent education, student sex, and parent science push. The absence of direct paths between parent religious attitude and either student science course attitude or student general attitude toward science indicates that there is not a residual direct effect of parent religious views on either of those variables. Given the sizes of the coefficients, the indirect influence of parent religious attitude on student general attitude toward science is very small. Finally, the model suggests that parent science push plays an important role in fostering positive student attitude toward science courses and science generally. Parent science push is the strongest predictor of positive student science course attitude in this model, accounting for 36~o of the mutual dependence in the prediction of student science course attitude. CONCLUSIONS AND RECOMMENDATIONS Returning to our original concern about the attitudes of students to- ward science and the failure of American high-school students to enroll in advanced science and mathematics classes in adequate numbers, this anal- ysis of the data from the Longitudinal Study of American Youth indicates that students' attitude toward their science course is the most proximate and most important short-term influence on more general student atti- tudes toward science. Since most high-school students experience a biology course early in their high-school program, one important impact of biology
go HIGH-SCHOOL BIOLOGY courses is that on student attitudes toward science generally. In subsequent analyses, we will explore in greater detail which facets of the science-course experience appear to have the greatest impact on general attitudes toward . ~ A ~ ~ ~ . ~ ~ ~ ~ ~ ~ ~ ~ ^~ _ _ science, but from the analysis reported above it is clear that the experl- ence of the sophomore student in his or her science class has a significant influence in the more general attitudes of the student toward science. In addition to the influence of science courses, the analysis indicated that parent science push and student educational aspirations also have significant, but far weaker, influences on students' general attitude toward science. The level of parent education has a substantial indirect influence on the later formation of student attitude toward science. Parent religious attitude has little net influence on students' general attitude toward science. Student sex appears to have mixed effects. Sophomore girls are significantly more likely to plan to earn a baccalaureate than sophomore boys, but boys report greater parent science push than girls. It is clear that both classrooms and parents play important, but differ- ent, roles in stimulating positive general attitudes toward science. Parent education and parent science push clearly contribute to holding higher educational aspirations and to liking high-school science courses. These factors, in turn, appear to foster more positive general student attitudes toward science. For the purpose of this analysis, classrooms and parents were treated as two separate variables. Unfortunately, in practice, they also appear to function relatively independently. This analysis suggests that one approach to increasing student interest In science might be an increased parental involvement in the science program. Some parents already push science with their children. Other parents may wish to encourage their students In science, but lack the educational background or self-confidence to do so. Increased parental involvement In high-school science programs should be explored as one avenue to focusing and using parental Influence in the most productive manner possible. REFERENCES Fienberg, S. E. 1980. The Analysis of Cross-classified Categorical Data. 2nded. Cambridge, Mass.: MIT Press. Goodman, L. A. 1972. A general model for the analysis of surreys. Amer. J. Social. 77:1035-1086. Goodman, Lo A. 1978. Analyzing Qualitative/Categorical Data: Log-Linear Models and Latent Structure Analysis. Cambridge, Mass.: Abt Books. Miller, J. D., R. W. Suchner, and A. Voelker. 1980. Citizenship in an Age of Science. New York: Pergamon Press.
11 What High-Schoo! Juniors Know About Biology: Perspectives from NAEP, the Nation's Report Card INA ~ S. MULLIS LEVELS OF PROFICIENCY Since 1969, the National Assessment of Educational Progress (NAEP) has been conducting regular assessments of student achievement in a variety of school subjects. As part of its most recent science assessment, in 1986, NAEP assessed the science proficiency of a nationally representative sample of eleventh-grade students composed of 11,744 respondents. The assessment included multiple-choice and open-ended questions about their knowledge, skills, and understanding in four science content areas-the life sciences (biology), physics, chemistry, and earth and space sciences- and their grasp of the nature of science (National Assessment of Educational Progress, 1987~. The assessment was also conducted at grades 3 and 7 and was designed to monitor trends in achievement. A comprehensive report of the results is contained in The Science Report Card (Mullis and Jenkins, 1988~. The data were analyzed with Item Response Theory scaling techniques and summarized on a composite science scale ranging from 0 to 500 (Beaton Ina ~ S. Mullis, deputy director of the National Assessment of Educational Progress (NAEP), The Nation's Report Card, was a coauthor of The Science Report Card. Elements of Risk and Recovery. She was principal investigator for NAEP's study of higher-order thinking skills assess- ment techniques in science and mathematics and is serving on the advisory board for assessment for the National Center for Improving Science Education. 91
92 HIGH-SCHOOL BIOLOGY TABLE 1 Have You Taken Biology? Response Percentage Science Proficiency Yes 89 296 (1~0) No 11 268 (1.8) aJackknifed standard errors are in parentheses. et al., 1988; Mullis and Jenkins, 1988~. Eighty-nine percent of the eleventh- graders reported having taken biology, and their average science proficiency was substantially higher than that of students who had not taken the course (see Able 1~. In addition to average science proficiency, to provide a basis for inter- preting the results on the NAEP scale, NAEP defined science proficiency at five levels on the science scale. ~ characterize these levels, science specialists analyzed the types of items that discriminated between adjacent performance levels on the NAEP science scale and described the skills held by students performing at five anchor points (150, COD, 250, 300, and 350~. Able 2 provides a brief characterization of performance at each anchor point and gives the percentage of high-school juniors performing at or above each level. Virtually all eleventh-grade students performed at or above Level 200, indicating an understanding of simple scientific principles. In the area of biology, these students displayed a rudimentary knowledge of the structure and function of plants and animals. They are likely to recognize the characteristics of common aquatic birds and know that the wolf and dog are closely related, that a mouse does not lay eggs, and that the main function of the heart is to pump blood to all parts of the body. Also, as typified by Level 250, most (85%) are likely to be familiar with food chains, to understand that light and water affect plant growth, and to be able to identify how some diseases are transmitted. While most high-school juniors attained the three lowest proficiency levels, fewer than half reached Level 30~a level characterized by more specific scientific knowledge and the ability to analyze scientific procedures and data. Further, only 6% of the students at this grade level demon- strated the ability to infer relationships and draw conclusions using detailed scientific knowledge. In addition to highlighting the distribution of students across the five levels, Able 2 reveals large performance gaps between males and females and particularly between white students and their black and Hispanic peers. For example, half the males reached Level 300 in 1986, compared with only
93 C~ ._ Ct U. . - Ct m ~1 ~0 00 0` . ~ e~ 2 C) ·CJ V' .2 o ¢ o ¢ 4 - V) o Ct P" [T] ~; E~ ._ 3 Ct - - Ct o ._ .c C~ a ~o ~o 0 ~i ~i 0 0 _ _ _ _ _ oo o ~o ox o ~_ {', ,_ ', o. ~o o 1 _ _ _ _ o ~o o ~o ~o . · . o ~o _ _ _ o ~o~ o ~, o _ _ _ ~oo o o o o o o - o~ ox ~_ o · . . . . o ~o o o _ _ _ _ _ ax 0 o ~0 0 oo o o o o ~o o o _ _ _ _ _ so ~o ~o ~oo o ~o ,~ O ~.° ca ~ ~ i D ~ S o o o o o o ~o ~ ta C~ Ct .~ ~2 Ct Ct 4 - U, ._ Ct Ct1
94 HIGH-SCHOOL BIOLOGY one-third of the females. And, while half the white students reached this level, only about 9% of the black students and 14% of the Hispanic students did so. Although 93% of the white students reached Level 250, only about half the black and two-thirds of the Hispanic eleventh-graders did. RESULTS BY SEX ANI) RACE-ETHNICI1Y ON THE LIFE-SCIENCES SUBSCALE ~ construct the composite science scale, NAEP computed results for the different content-area subscales-one of which was life sciences (biol- ogy). The meaning of the science subscales cannot be known in absolute terms; that is, one cannot determine how much learning in chemistry equals how much learning in the life sciences. The subscales do, however, permit an analysis of the relative strengths and weaknesses of students in different population groups within each science content area. Like the proficiency results on the composite science scale, the re- sults on the life-sciences subscale indicated that white eleventh-graders outperformed Hispanic eleventh-graders, who outperformed their black counterparts. Given these results, a natural question revolves around the impact of any potential differences in biology course-taking for these groups of students. However, as shown in Table 3, although slightly fewer His- panic students than white or black students reported having taken biology, the percentages were equivalent for white and black students (Mullis and Jenkins, 1988~. In all three racial-ethnic groups, students who had taken the course performed much better on the life-sciences subscale than those who had not, but course-taking did little to lessen the performance gaps between these groups of students. The gaps remained essentially constant. Also In keeping with results on the composite science scale, males performed better than females on the life-sciences subscale, although it is TABLE 3 Average Proficiency of Eleventh-Grade Students on the Life-Sciences Subscale, by Biology Course-Takin~a Percentage Proficiency Proficiency of Students of Students of Students Student Who Have Who Have Who Have Not Group Taken Biology Taken Biology Taken biology Male Female White rib. Black 88 (1.0) 89 (0.9) 89 (1.0) 82 (2.3) 89 (1.3) 298 (1.3) 291 (0.9) 302 (0.8) 269 (1.6) 259 (1.7) 271 (3.0) 265 (2.5) 276 (2.5) 248 (2.4) 239 (3.0) aJackknifed standard errors are in parentheses.
WHAT HIGH-SCHOOL JUNIORS KNOW ABOUT BIOLOGY 95 interesting that this difference did not appear to be substantial until grade 11. In fact, at grade 3, girls had a slight edge in performance-an edge that shifted in favor of boys by grade 7. Able 3 also indicates that just as many females as males take biology in high school. However, while both male and female students who had com- pleted biology performed significantly better on the life-sciences subscale than those who had not taken the course, sex differences in performance remained essentially unchanged, irrespective of biology course-taking. Patterns of high-school performance of groups of students defined by sex or race-ethnicity appear to be established early in the schooling process. For example, although the sex gap was not evident in the area of biology for third-graders, there was a substantial discrepancy in performance among racial-ethnic groups, with white students having significantly higher proficiency levels on the life-sciences subscale than their black and Hispanic classmates. Forty-six percent of the seventh-grade students described life science as the primary area of study in their science classes. However, these students did not perform as well on the life-sciences subscale as the one- quarter of their classmates who reported studying general science, although students studying life science did outperform the 6% of the students who reported study emphasizing physical science and the 11% studying earth science. By seventh grade, for both students studying life science and those studying general science, performance gaps were apparent between males and females, with the males having the higher levels of achievement. The gaps among the three racial-ethnic groups were substantial, with white students outperforming their black and Hispanic peers. SELECTED ITEM-BY-ITEM RESULTS At grade 11, 59 items were included in the life-sciences subscale, and the performance results for some of these items are provided below to illus- trate the composition of this subscale and to present some specifics about what high-school juniors know about biology. The items are categorized in four groups for the purposes of discussion: ecological relationships, cell structures and functions, energr transformation, and genetics. It should be emphasized that the items that follow are only illustrative of the skills, knowledge, and understanding tapped in the 1986 science assessment and are not intended to be an inclusive account of all that high-school students should know about or be capable of doing in biology (Mullis and Jenkins, 1988~.
96 HIGH-SCHOOL BIOLOGY Ecological Relationships Eleventh-grade students appeared to have some knowledge of their environment and a grasp of basic ecological concepts. For example, in response to the item below, 81% of the students correctly identified the fox as the predator In the food web presented. >' Fox Rabbit ' Grass _,' Field Mouse With respect to the field mouse in the food web above, what is the fox considered? · A predator 0 A prey o A producer o A decomposer In response to several other ecology-related items, 80% of the high- school juniors recognized acid rain as a kind of pollution, and 77% knew something about the effects of insecticides. In contrast, only about one-third (31%) of the students recognized a recommended method for controlling soil erosion, and fewer than one-fifth (19%) correctly identified a graph of the world's population growth. For the items described above, the response patterns for groups were similar to those shown on the life-sciences subscale overall: eleventh- grade males performed significantly better than females, and white students performed significantly better than either black or Hispanic students. Cell Structures and Functions Although two-thirds of the eleventh-graders recognized a diagram of a group of cells as a tissue, only one-third were able to identify the basic function of the cell membrane. Fewer still (approximately one-fourth) were able to apply their knowledge to specify how a cell membrane works or explain the distinguishing features of a plant cell based on a diagram. Although white students tended to perform better than black or Hispanic students on these questions on cell structures and functions, the differences in performance between males and females were minimal. Energy Transformation While 90% of the high-school juniors recognized that `'junk food" is high in calories and low in nutrients, fewer than two-thirds appeared to
WHAT HIGH-SCHOOL JUNIORS KNOW ABOUT BIOLOGY 97 have a general understanding of photosynthesis. For example, 60% of the students responded correctly to the item below. Which of the following best explains why marine algae are most often restricted to the top 100 meters in the ocean? They have no roots to anchor them to the ocean floor. They are photosynthetic and can live only where there is light. o The pressure is too great for them to survive below 100 meters. O The temperature of the top 100 meters of the ocean is ideal for them. A slightly higher percentage of the students appeared to know that plants produce oxygen and tend to grow toward light (68% and 67%, respectively). Somewhat fewer students (57%) were able to apply their knowledge to respond correctly to an item on the role of red blood cells in transporting oxygen. In this area, again, the sex gap was not significant, perhaps because females were more likely than males to report experience in working with plants and animals. As with performance on the groups of items previously described, however, there were significant differences in the performance of white, black, and Hispanic students on items pertaining to cell structures and functions. Genetics As made evident by their performance on items in this category, genet- ics was a relatively difficult area for the eleventh-grade students assessed. Approximately half the students demonstrated a basic understanding of recessive genes, and 57% could identify the probability that parents with a certain genetic structure would produce blue-eyed children. Forty-seven percent of the students responded correctly to the item below. A female white rabbit and a male black rabbit mate and have a large number of baby rabbits. About half of the baby rabbits are black, and the other half are white. If black fur is the dominant color in rabbits, how can the appearance of white baby rabbits best be explained? o The female rabbit has one gene for black fur and one gene for white fur. The male rabbit has one gene for black fur and one gene for white fur. o The white baby rabbits received no genes for fur color from the father. o The white baby rabbits are result of accidental mutations. Fewer students (28%) were able to use their knowledge of natural resistance to assess the implications of genetics research, as shown in
98 . HIGH-SCHOOL BIOLOGY the example below. Although males and females performed similarly on questions related to genetics, white students tended to outperform their black and Hispanic classmates. Recombinant DNA research has produced a variety of organisms with big economic potential. For which of the following reasons are concerned citizens hesitant to permit the use of these organisms outside of the labo ratory? Production of such organisms will involve the production of haz ardous by-products. Most scientific research is perceived to be dangerous. o The organisms could die outside of a laboratory environment. The introduction of organisms new to the Earth could upset the ecological balance. SUMMARY Nearly all (89%) of the approximately 12,000 high-school juniors as- sessed by NAEP in 1986 reported having taken a course in biology. Students who had taken the course performed significantly better than those who had not on both the life-sciences subscale and the composite scale representing performance in all the science content areas assessed. Given that most high-school juniors have taken biology, their un- derstanding of the life sciences appears quite limited. Virtually all these students exhibited the kinds of knowledge that may be gained from ev- eryday experiences; however, substantially fewer displayed more detailed knowledge and understanding. While an instructional emphasis on some aspects of ecology is suggested in the performance results, students ap- peared to have had relatively limited or ineffective exposure to other areas in the life sciences. For example, students displayed little understanding of cell structures and functions, genetics, and energy transformation. On the basis of their lack of knowledge, skills, and understanding and their inability to apply those they do possess, it is likely that our high-school juniors do not grasp the larger concepts that most science educators believe to be the foundation of a strong education in biology, including systems and cycles of change, heredity, diversity, evolution, structure and function, and organization. These findings, while troubling in themselves, are given further weight by evidence that substantial disparities exist in the performance of groups defined by sex and by race-ethnicity. While just as many females as males had taken biology, course-taking in this subject did not appear to lessen the sex-related performance gap. Rather, the gap remained as large among students who had taken the course as among those who had not. Similar patterns were found in the disparities across racial-ethnic groups. For
WHAT HIGH-SCHOOL JUNIORS KNOW ABOUT BIOLOGY 99 each of the three groups analyzed by NAEP white, black, and Hispanic students- students who had taken biology outperformed those who had not taken the course. However, biology course-taking did not appear to lessen the performance gaps between white students and their black and Hispanic peers. REFERENCES Beaton, ~ E., et al. 1988. Expanding the New Design: The NAEP 1985-86 Technical Report. Princeton, N.J.: Educational Testing Service. Mullis, I. V. S., and L. Jenkins. 1988. The Science Report Card: Elements of Risk and Recovery. Princeton, N.J.: National Assessment of Educational Progress, Educational Testing Service. National Assessment of Educational Progress. 1987. Science Objectives: 1985-86 Assess- ment. Princeton, N.J.: National Assessment of Educational Progress, Educational Testing Service.
1 ~ The NABT-NSTA High-School Biology Examination: Its Design and Rationale BARBARA SCHULZ EVOLUTION OF THE BIOLOGY TEST The evolution of the national high-school biology test is especially interesting. Early in this decade, there was much discussion about the need for such a test by both the National Science Teachers Association (NSTA) and the National Association of Biology Teachers (NABT). Each group ini- tiated preliminary discussions about whether such a test should be written, how it would be written, who would write it, and how it might be abused. In 1982, the high-school division of NSTA, under the direction of Linda Perez of Texas and Angelina Romano of New Jersey, conducted a needs assessment among the membership. The response was overwhelmingly in favor of a test. At the same time, NABT appointed a small committee to discuss the feasibility of such a test. This group, including Joe McInerney of Colorado and Ken Bingman of Kansas, discussed the notion of developing a test bank of questions available to the membership. In 1984, the NSTA board of directors passed a motion that NSTA and NAB T proceed with the development of a national test. The motion also asked that the president Barbara Schulz has been a high-school science teacher for 22 years and department chair since 1974. She was a recipient of the Outstanding Biology TeacherAward in 1981 and the Presidential Award for Excellence in Science Teaching in 1983, and she was a semifinalist for the Teacher in Space Award. Ms. Schulz was president of the Washington Science Teachers Association in 1987- 1988. 100
TlIE NAB T-NSTA HIGH-SCHOOL BIOL~YE~INATION 101 of each organization appoint four persons to serve on a joint test devel- opment committee. So it was that the first NABT-NSTA test development committee met in Chicago in June 1985 to begin the test construction. It was with a great deal of hesitation that this joint committee of nine biology educators proceeded with the test development. Some states have competence tests; others are considering such a move. We, the professional biology teachers, feel best qualified to design the test and help to set the direction of the biology curriculum. With the great demand for accountability being felt by all, it was clear that a test would be developed. The following statement of rationale and purpose was prepared and printed in both News & News and The Science Teacher Journal In 1985: A Standardized Test for First-Year High School Biology Rationale and Statement of Purpose There is an increasing demand for accountability in science education, and science educatom, through their professional organizations, should assume responsibility for establishing the mechanisms for that accountability, lest the responsibility fall to lay persons with vested interests. One such mechanism is a student education instrument. Accordingly, the National Association of Biology Teachers and the National Science Teachers Association are collaborating on a project to develop a standardized test for high school biology. This objective, year-end test will be intended for first-year high school biology students and will address a core of basic biological concepts, processes, and thinking skills. The joint committee has agreed on the following principles: a. The test should be used to improve science education; questions will be oriented toward inquiry and other higher-level cognitive functions; b. The test should not be used to evaluate teachem; c. The test should not become an end in itself, that is, the biologic content reflected in the test items should not be interpreted as the final word on a complete conceptual framework for an introductory biology course; and d. The test should be updated every two years. VALIDATION BY THE MEMBERSHIP Having declared that the purpose of this test is to drive curriculum forward, the committee looked at the question of content and level of difficult. The following concepts were decided on: Cell structure and function Sample concept: Biological systems vary in their degree of spe- cialization.* B. Bioenergetics Sample concept: Biological systems cannot exist without energy input.* *Biology Assessment Review Workshop. Biology Test Domains, Objectives and Content Speci- fications. Wisconsin Department of Public Instruction. 1983.
102 HIGH-SCH~L BIOLOGY C. Genetics Sample concept: Organisms pass on characteristics to the next generation through genetic material.* D. Evolution Sample concept: Organisms change through time. E. Systems, physiology to morphology Sample concept: Structure and function complement each other in biological systems. F. Ecology Sample concept: Organisms are interdependent, and their inter- actions result in the flow of energy and the cycling of matter. Taxonomy Sample concept: Biological systems are grouped on the basis of similarities that reflect evolutionary history.* H. Behavior Sample concept: The response of an organism to its environment has both a genetic and an environmental basis. I. Science, technology, and society Sample concept: Advances in science and technology have impli- cations for personal and societal decision-making. It was also decided that the following list of processes and skills should be represented in the way questions were designed: 1. Inquiry Process science Experimental design "Science as a way of knowing" (John ~ Moore) History Probabilistic thinking Creative problem-solving These concepts, processes, and skills were published in the same article with the rationale. A response card was published concurrently in both journals, and readers were asked to validate the conceptual framework and intent of the test. The test committee received a good response; more than 400 cards were returned. Also, sessions were held at the three NSTA regional meetings, the NSTA national convention, and the NAB T national convention for the purpose of concept validation. As a result of the meetings, the conceptual area of science, technology, and society was added to the test by popular demand. A call for test questions from the membership was made and a good response received. Each question was reviewed by a college-level content specialist and a high-school biology teacher for validity and appropriateness. From the solicited questions, 120 3. 4. 5. 6. 7.
THE NAB T-NSTA HIGH-SCHOOL BIOLOGY EXAMINATION 103 were selected for review and field-testing. Of those, 80 were selected for the final test. TEST CONSTRUCTION Armed with more than 300 questions, the committee turned to the issue of test content. The following numbers of questions were agreed on: Concept Cell structure and function Bioenergetics Genetics Evolution Systems, physiology to morphology Ecology Mono my Behavior Science, technology, and society No. questions 8 10 12 12 8 8 6 8 8 In addition, the committee decided that each concept area should be written at three levels of difficulty, ranging from knowledge to synthesis. A field test of 120 items was given in the spring of 1986 to students in 12 states covering all regions of the country. The Lertap test analysis was done on the field-test data. Eighty questions were then selected for inclusion in the first edition of the test. A second field test was done on the 80-item test in the fall of 1986 after some editing of the positively correlated distracters. THE RESULTS ARE IN By the spring of 1987, the test was published, and more than 30,000 copies were sold. The Educational lasting Service (ETS), Princeton, New Jersey, was hired to do the data analysis on the first commercial edition of the test. ETS analyzed 895 answer sheets from Form A and 1,075 answer sheets from Form B. Form A Form B Mean 43.8 40.7 Standard deviation 13.3 12.4 Median 44.3 40.2 Reliability (alpha) 0.91 0.91 Committee member Juliana Texley provided the analysis shown in Figures 1, 2, and 3.
104 10 5 o <' HIGH-SCHOOL BIOLOGY DATA - FORM B _ ~ rid . 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 ETn~ FIGURE 1 Jest Form B. 20 an c, 1 5 Oh 11 o 1 0 z G LL 5 o FIGURE 2 Test Form STUDENT SCORES DATA - FORM A Inn <5 10 15 20 Pnl I 55 60 65 70 75 80 30 35 40 45 50 STUDENT SCORES The test committee was pleased with the results. There was a strong interest by science teachers in using this test, and we have learned much from it. The committee made some minor revisions to correct misleading language in the 1987 test. The revised edition will be available for the next 2 years. In 1990, at the request of the memberships of NABT and NSTA, a completely new test will be designed and tested. The same format will be followed.
105 DATA - FORM A REVISED THE NAB T-NSTA HIGH-SCHOOL BIOLOGY EXAMINATION 25 In Z 20 g lL 15 o z IIJ llJ 10 5 o FIGURE 3 1bst Form A Revised. an n n n <5 6-1011-15 16-20 21-25 26-30 31-35 36-40 41-45 46-50 51-55 56 60 61-65 >65 STUDENT SCORES CONTINUED CHANGE THROUGH TIME The test will be recreated every 2 years. New committee members will rotate on to the test committee to serve a 3-year term, the time it takes to build, field-test, and publish the test. Questions will continuously be sought from practicing high-school biology teachers. Thus, the curricular emphasis within the test can change to reflect the concepts deemed relevant by biology teachers. The test may help to focus a solid core curriculum in biology, stated in terms of broad and significant concepts, rather than encyclopedic facts. The biology test, as an effort of professional associations, is not a product, but an on-going process. The first administration of the test has generated at least as many questions for the committee as we provided for our students. As the data from the first test were analyzed, the group was already planning a schedule for soliciting and field-testing new items for future tests. The committees will change, and the evaluation will evolve with the help and input of members. And as they do, we will learn more about our students, our classrooms, and ourselves. After 2 years of development, the first test results for biology are in, and another benefit of the test development process has also surfaced. We quickly realized that we have been given a valuable insight into what students know- about the science of biology. Scanning the answers of over 2,000 randomly selected subjects, the committee was able to peek into some widely held misconceptions and to hypothesize about the classroom procedures that might be perpetuating them. ~ a large extent, the test's validity derives from the many years of joint experience the committee
106 HIGH-SCHOOL BIOLOGY members share; their interpretations of the data they have received from the first test administration come largely from the same experience base. The group hopes to engender further discussion to improve not only the test, but the process of biology education. A PEEK INTO STUDENTS' MINDS Every teacher knows that the hardest part of test construction is choosing the wrong answers (the distracters or foils)-not too easy to spot, not so outlandish that no one would choose them. When the constructors of the national biology test received their item analyses, one of their most significant data sets was the percentage of students that chose each of the wrong answers. When one foil attracted a very large number of respondents, the obvious question always came up: Why? In a few cases, the foil was found to be marginally correct, given a slightly nontraditional reading, in a way the group had not foreseen. This type was changed by editing. But there was nothing "right" about many of the most frequently chosen foils. What was happening, it seemed, was that the foil touched . on a widespread student misconception or a teaching technique that often misfired. Some of the common errors seemed mnemonic; they seemed to result from verbal associations that we repeat too often in the teaching of biology: · When we asked students to complete the phrase. "The cell is a unit of structure and a unit of ," in the first field test, we were amazed to find that the majority chose "organ system." Had we taught the sequence "cell, tissue, organ, system" by rote once too often? Students demonstrated common vocabulary confusions, such as mistaking "cell membrane" and "cell wall." · When we asked a question about meiosis, the most popular choice was one that contained the word 'Gamete''-even though the sense of the answer was completely incorrect. The results suggested to the committee that far too many of our students are relying on word associations to weave their way through biology. Do such tricks work on classroom tests? Do we encourage them? Other common errors that the students demonstrated told us that some of our most important conceptual goals were often not met. In ques- tions about evolution, the Lamarckian explanation for an adaptation was consistently chosen as often as the explanation based on natural selection. This result is backed up by a number of research studies that show that the idea of inheritance of acquired characteristics is both intuitively appealing and surprisingly persistent in biology students of all ages. Similarly, the concepts of energy and entropy were difficult for students,
THE NAB T-NSTA HIGH-SCHOOL BIOLOGY EXAMINATION 107 despite the relatively simple and straightforward wording of questions. The idea that energy dissipates and does not cycle in the environment was a difficult one for students in several contexts. Perhaps too much emphasis was placed on cycles and too little on energy. We saw evidence of misconceptions that were and are text-perpetuated. Students believe (on the basis of misinformation in many texts) that muta- tions are always recessive and weak (Mahadeva and Randerson, 1982~. We also found that it was dangerous to assume that students had ex- perienced some of the more common laboratory investigations in first-year biology texts. Students found two questions about surface-to-volume ratio very difficult; it seemed that they had not explored the relationship between cell-membrane size and cell division. Any questions about hypertonic and hypotonic solutions were quite challenging when the terms themselves were not used. It seemed that students relied on the words, rather than the experiences, to influence their predictions. OTHER VARIABLES IN THE TESTING PROCESS In analyzing an evaluation tool, test-makers must be conscious of the other factors that can contribute to the variance in student performance. In the construction of the national biology test, the authors paid careful attention to the reading level and vocabulary of each question. In many cases, judicious editing was effective. But there was still evidence that the longer questions were harder than the shorter ones a result that was not expected. What was surprising, and what may provide the basis for more detailed research by the group, was evidence suggesting that questions involving visual or graphic analysis were harder as a group than the others in the instrument. The students who took the test seemed consistently confused by graphs and diagrams. In one item set based on an enzyme graph, the independent variable was increasing left to right, but students commonly erred by assuming that the enzymes represented left to right were in the order of their presence in the alimentary canal; that is, many believed the first were mouth enzymes, the second stomach enzymes. In a diagram of predator- prey relationships in a prairie, many students guessed that the prey of coyotes in that community would be jackrabbits-despite the evidence provided in a graph and clear directions to answer the question from that graph. In 1988, the committee examined seven demographic questions relevant to performance on the high- school biology examination. Each of the questions was preceded onto the test forms by students in self-selected classrooms and analyzed by means of one-way analysis of variance at a 0.05 level of significance. We randomly selected 882 tests. Although Forms A
108 [IIGH-SCHOOL BIOLOGY and B were distributed that year, only Form A responses were available in numbers suitable for random selection for analysis. (Previous analysis of test scores indicated that the forms were parallel, since only the order of the answers had been changed.) Of the students in the sample, ninth-graders and eleventh-graders did significantly better than the tenth-graders who would normally be enrolled in standard-level biology classrooms. In analyzing the data further, we found that students who indicated that they "never" experienced laboratory work did significantly more poorly than those who did laboratory work "some of the time." The frequency of laboratory work was not an important factor. However, those who had a laboratory experience did better than those with no laboratory experience or those who said they had laboratory all the time. While this identifies laboratory experience as necessary, it also brings into question student perception of "seldom," "frequent," and "most of the time." There is little evidence of standardization among advanced-biology sections, and some of these students may have been in courses tailored to individual research. The committee found no significant difference based on structure of schools. However, there was some evidence that students in smaller schools-500 or fewer performed significantly better. Our results on item difficulty gave us a clue to what was and what was not generally taught in the classrooms where our normative data were developed. Botany questions were uniformly more difficult for students than zoology questions. Mendelian genetics was surprisingly easy; modern genetic engineering was often very difficult. Taxonomy questions were the easiest (even though the test did not ask any specific taxa). And questions about the societal implications of modern biology and environmental prob- lems like acid rain were answered correctly by very few subjects, suggesting that teachers may be reluctant to add this emphasis to their curriculum. FUTURE TESTING For the immediate future, the committee has opted to add clearer pic- tures and diagrams for students who need such help. In years to come, both teaching and testing may be enhanced by far more visual stimuli; videotape and real-life examples may help students to reason more effectively with broader comprehension. Perhaps the most important implication of such a national test is not the result, but the point from which the committee started. With the recognition that we can't teach and students can't really learn everything in the commercial texts, the joint position of the associations is that the test establishes a core of nine concepts that should be a part of every student's first-year biology experience. It was this list and not
THE NABT-NSTA HIGH-SCHOOL BIOLOGY EXAMINATION 109 the questions themselves that seemed to elicit the most interest in the members, many of whom would rely on such a statement to guide their own choices. REFERENCE Mahadeva, M., and S. Randerson. 1982. Mutation mumbo jumbo. Sci. Teach. 49~3~:34-38.
