control group consists of students in a comparable classroom taught the same science and engineering topics but not with EiE curriculum materials and tools. In the ideal scenario, EiE has a large pool of teachers from which part are admitted to the EiE project and the other part remain as control groups. This process does not always work because of constraints of funding and time. Another example criterion is that teachers express increased efficacy and interest in teaching engineering to their students.
Randomized, controlled studies with external evaluators are the preferred method for evaluating and comparing efforts in education, said Lachapelle. NSF, for example, prefers this approach when seeking summative assessments in projects it funds. Unfortunately this type of assessment is very expensive to execute because there is usually a need for a fairly large number of students in order to randomize whole classrooms into different testing groups. Also external evaluators are an added cost and bear their own pros and cons. Although external evaluators are likely to be more objective in their assessments, they do not have the advantage of an ongoing relationship with the teachers, administrators, and students whom they are engaging and thus may miss subtleties that more familiar evaluators might observe.
In her discussion, Lachapelle cautioned that assessments and evaluations of computational thinking activities and materials require clearly specified learning objectives, which in turn require some community consensus regarding the content of computational thinking—that is, what is it that the community wants children at various ages to know (from early elementary school to college)? In the EiE context, some learning objectives include being able to identify a process, to explain what a process is in an engineering context, and to explain why the order of steps in a process is important.
She also argued that the learning objectives should align with psychological and developmental learning progressions, since doing so provides some guidance over time as to where students should be at each stage. Thus, learning objectives are and should be the object of research and design. She noted that EiE does extensive literature searches and local interviews with kids before beginning the design of each of its units in order to learn more about what kids know. For example, for a unit on sinking and floating, developers would do a literature search and then interview local students by asking them things like, “Do you know what it means to float?,” “Do you understand why things float?,” and so on.
Finally, Lachapelle commented that their assessments are also designed to address student attitudes toward science and engineering. Broadly speaking, these assessments indicate that girls tend be interested in engineering things when framed as helping to improve people’s lives and boys tend to be interested in engineering things when framed in terms of constructing engineering artifacts.