National Academies Press: OpenBook

Indicators for Monitoring Undergraduate STEM Education (2017)

Chapter: Appendix A Public Comments on Draft Report and Committee Response

« Previous: 7 Implementing the Indicator System
Suggested Citation:"Appendix A Public Comments on Draft Report and Committee Response." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 177
Suggested Citation:"Appendix A Public Comments on Draft Report and Committee Response." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 178
Suggested Citation:"Appendix A Public Comments on Draft Report and Committee Response." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 179
Suggested Citation:"Appendix A Public Comments on Draft Report and Committee Response." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 180
Suggested Citation:"Appendix A Public Comments on Draft Report and Committee Response." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 181
Suggested Citation:"Appendix A Public Comments on Draft Report and Committee Response." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 182

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

PREPUBLICATION COPY, UNCORRECTED PROOFS Appendix A Public Comments on Draft Report and Committee Response In order to obtain broad input into its work, the committee publicly released a draft report for comment in August, 2016 after completing Phase I of the study. This draft report was intended to elicit feedback from the interested public in order to ensure that the committee was comprehensively covering the relevant terrain and also proposing reasonable goals and objectives that could be monitored over time without imposing undue data collection burdens. The interim report was available on the committee’s website, with a 7-week period for comment. Public comments were sought to obtain perspectives and insights from researchers and practitioners knowledgeable about undergraduate STEM reform and education statistics. The public comment draft included a conceptual framework for the indicator system, identified goals and objectives for improving undergraduate STEM education at both 2- and 4- year institutions, and reviewed existing systems for monitoring undergraduate STEM education: Table A-1 shows the draft goals and objectives on which the committee sought comment. Based on the committee’s consideration of what information from the public would be most useful for the second phase of the study, the report included a series of questions for readers to respond to, as follows: 1. The committee proposes 5 goals to improve the quality of undergraduate STEM education (see Chapter 2). Is this the right set of goals? Should any be deleted or other goals added? Why do you suggest this change? 2. The committee identifies 14 objectives around which national indicators for undergraduate STEM education will be developed in Phase II of the study (see Chapter 2). Is this the right set of objectives? Should any be deleted or other objectives added? Why do you suggest this change? 3. The committee discusses various data sources on undergraduate STEM (see Chapter 3). Are these the right data sources? Should any be deleted or other sources added? Why do you suggest this change? 4. Are there larger issues related to measuring and improving quality in undergraduate STEM that are missing from the committee’s proposed conceptual framework, goals, and objectives? 5. How and where, if at all, do you see the national indicators to be developed in Phase II being used to improve undergraduate STEM? Individuals and representatives of organizations were encouraged to submit their responses to these questions through an online questionnaire that was posted with the public comment draft. The committee received 32 comments through the website questionnaire and 2 comments through letters. To supplement the input from the online questionnaire, the committee convened a day- long public meeting in October 2016, which included responses from invited individuals and institutions, as well as open microphone sessions for all meeting participants. The meeting drew just over 100 people, 62 in person and 40 by the webcast. Following the public meeting, the committee reviewed all of the feedback and identified possible revisions. The committee used the Phase II of the study to revise its work in response to the concerns and suggestions it had received, resulting in this final report. The rest of this AppA-1

appendix summarizes the feedback and describes the steps taken to revise the initial conceptual framework and the preliminary review of data sources and monitoring systems. Those revisions are reflected in this final report. Overarching Issues Several themes emerged across all comments received, through the website, letters, and at the October meeting:  Role of Discipline-Based Education Research (DBER). Several commenters ask that we consider the role DBER in improving the quality of undergraduate STEM. They asked the committee to consider adding DBER to one of our objectives, and to consider it as a potential indicator of the use of evidence-based educational practices.  Meaning of Goal 1. Several commenters raised concerns about the wording of Goal 1, which was “Increase numbers of STEM majors,” and offered suggestions for other ways of approaching the goal.  Evidence-Based Practices. Commenters asked for a more thorough explanation of the meaning of “evidence-based practices” and suggested broader, more expansive use of the term.  Expanded Definitions of Equity. Commenters praised the committee’s attention to diverse learners, but several asked that the committee broaden this group to include students with learning disabilities, first-generation college students, and other populations, along with discussion of the ethical dimensions of diversity and inclusion.  Unit of Analysis. Commenters raised questions and offered suggestions about the most appropriate unit of analysis for measuring improvement in STEM. For example, some called for indicators at the department or institutional level, while others expressed concern that individual institutions would be held accountable for national- level indicators of equity, diversity, and inclusion.  Defining STEM and Related Terms. Commenters asked for expanded definitions of STEM, STEM literacy, and STEM learning to emphasize the role of the social sciences, the social and civic application of STEM knowledge, the development of ethics, positive attitudes, and “21st century” skills and to more closely integrate STEM with the humanities.  Future Indicators. A few commenters noted that the framework, goals, and objectives are relevant to current undergraduate STEM but may need to be updated in the future, as student populations and higher education institutions change.  Use of Existing Rubrics. Many commenters were concerned that the draft did not give more prominence to the PULSE Vision and Change reform initiative, which has developed rubrics to measure progress toward some of the proposed objectives.  Data Sources. A few commenters noted that the Science and Engineering Indicators report is not an original data source. In addition, several pointed to the PULSE rubrics as a potential data source. AppA-2

Committee Response In response to these comments, the Committee made several revisions to the interim goals and objectives shown in Table A-1 (above): DBER Although the committee decided not to identify DBER as a specific objective or indicator, discussion of DBER was added to the chapter on use of evidence-based educational practices (see Chapter 3). Goal 1 The committee revised the wording of Goal 1 from “Increase numbers of STEM majors” to “Ensure adequate numbers of STEM professionals,” and added language clarifying that demand varies across the different STEM disciplines (see Chapters 1 and 5). Evidence-Based Practices The committee defined “evidence-based practices” (see Chapter 1) and described them with review of the literature and detailed examples (see Chapter 3). Equity The committee expanded its focus on equity to include students with disabilities and first–generation college students (see Chapter 4). Unit of Analysis The committee clarified its focus on national indicators, and the nation as a whole as its primary unit of analysis, as called for in the study charge (see Chapter 1). Definition of STEM Considering comments about the lack of clarity around STEM literacy, the committee discussed the recent report on science literacy (National Academies of Sciences, Engineering, and Medicine, 2016b). Given that report’s findings about the difficulty of defining science literacy, as well as the challenge of specifying a level of STEM literacy that all students should master, the committee decided to drop STEM literacy as a formal goal. However, the committee explains that its vision for undergraduate education includes all students’ developing a basic understanding of STEM concepts and skills (see Chapter 1). While recognizing the value of the social sciences and the humanities and the benefits of an integrated liberal arts education, the committee concluded it needed to maintain its focus on undergraduate STEM education, as required by the study charge. Additionally, in response to calls to address the future of STEM education, this report discusses the growth of online education and assessment, noting that as these technologies advance, new indicators of STEM learning may be needed (see Chapters 1 and 7). Finally, to clarify and focus the overarching goals, the committee decided to drop the goal of continuous improvement, but added continuous improvement as an objective within the goal of increasing mastery of STEM concepts and skills by increasing students’ engagement in evidence-based educational practices (see Chapter 3). Existing Rubrics In response to many comments about PULSE, the committee added more discussion of PULSE and other existing reform initiatives (see Chapters 1 and 3). Data Sources In response to comments about data sources, the committee distinguished between original data sources and compilations of statistics and data (see Chapter 6), clarified its focus on national-level indicators (see Chapter 1), and considered a few specific PULSE rubrics that have the potential to be used to gather information on a national basis for the proposed indicators (see Chapter 3). AppA-3

PREPUBLICATION COPY, UNCORRECTED PROOFS TABLE A-1 Goals and Objectives in the Draft for Public Comment [FOR CONSISTENCY WITH FINAL REPORT, I DELETED PERIODS IN THE OBJECTIVES] Strategies to Advance Objective and Possible Goal Framework Objective Measures 1. Input 1.1 Multiple pathways into and through  Variety of entry/exit points Increase STEM programs  Guided pathways (map of courses) Numbers  Inter-institution articulations  Preparation support  Developmental Education approach  Bridge programs Process 1.2 High retention of students in STEM  Co-curricular supports for completion of core disciplines beyond core foundational foundational courses courses  Core course/unit completion  Advising  Mentoring  Living/learning communities  Career development/advising  Evidence-based instructional practices Process 1.3 Appropriate general education  Core proficiency in math, language and experiences for STEM students’ communication, and digital foundational preparation fluency/computational thinking Outcome 1.4 STEM credential attainment  Variety of credentials,  Outcome data  Change in attainment numbers over time 2. Ensure Process 2.1 Equity of access to high-quality  Recruitment Diversity undergraduate STEM education  Admissions processes and support  Bridge programs  Preparatory (developmental education) courses Outcome 2.2 Representational equity in STEM  Variety of credentials credential attainment  Outcome data AppA-4

 Change in attainment numbers over time 3. Process 3.1 Use of evidence-based STEM  Active learning instructional strategies Evidence educational practices both in and out of  Formative assessment Based (EB) classrooms  Advising and mentoring Education  Co-curricular opportunities/experiences  Internships  Engage in relevant interdisciplinary big questions  Authentic practice  Backward design of courses and programs  Aligned assessments  Data driven course and program improvements Process 3.2 Equitable access to evidence-based  Mentoring and advising STEM educational practices both in and  Diversity of instructional staff out of classrooms  Numbers of students experiencing evidence- based practices Environment 3.3 Support for instructors to use  Infrastructure evidence based teaching methods  Professional development  Recognition  Adequate time Environment 3.4 Institutional culture that values  Happens at all institutional levels undergraduate STEM education  Valid robust evidence-based teaching evaluation system  Teaching and learning in mission and official documents  Reward system aligned with instruction Outcome 3.5 STEM learning for students in STEM  Happens at course and program levels educational experiences  Ensure adequate depth in STEM disciplinary skills and knowledge (competencies)  All students will gain in the ability to be AppA-5

lifelong, independent and resourceful learners  Adaptable to the demands of new projects, jobs or careers 4. Outcome 4.1 Access to foundational STEM  Number of STEM courses completed by STEM experiences for all students, to develop students Literacy STEM literacy  Degree of achievement of STEM literacy  Change in attainment numbers over time Outcome 4.2 Representational equity in core  Number of STEM courses completed by STEM literacy outcomes students  Degree of achievement of STEM literacy  Change in attainment numbers over time 5. Environment 5.1 Ongoing data-driven improvement  Data-driven Institutional learning Ensure  Data systems that allow better tracking of Continuous students across multiple higher education Improvement institutions AppA-6

Next: Appendix B Possible Formulas for Calculating Selected Indicators »
Indicators for Monitoring Undergraduate STEM Education Get This Book
×
Buy Paperback | $55.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Science, technology, engineering and mathematics (STEM) professionals generate a stream of scientific discoveries and technological innovations that fuel job creation and national economic growth. Ensuring a robust supply of these professionals is critical for sustaining growth and creating jobs growth at a time of intense global competition. Undergraduate STEM education prepares the STEM professionals of today and those of tomorrow, while also helping all students develop knowledge and skills they can draw on in a variety of occupations and as individual citizens. However, many capable students intending to major in STEM later switch to another field or drop out of higher education altogether, partly because of documented weaknesses in STEM teaching, learning and student supports. Improving undergraduate STEM education to address these weaknesses is a national imperative.

Many initiatives are now underway to improve the quality of undergraduate STEM teaching and learning. Some focus on the national level, others involve multi-institution collaborations, and others take place on individual campuses. At present, however, policymakers and the public do not know whether these various initiatives are accomplishing their goals and leading to nationwide improvement in undergraduate STEM education.

Indicators for Monitoring Undergraduate STEM Education outlines a framework and a set of indicators that document the status and quality of undergraduate STEM education at the national level over multiple years. It also indicates areas where additional research is needed in order to develop appropriate measures. This publication will be valuable to government agencies that make investments in higher education, institutions of higher education, private funders of higher education programs, and industry stakeholders. It will also be of interest to researchers who study higher education.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!