National Academies Press: OpenBook

Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop (2012)

Chapter: 3 Key Questions Identified by Discussion Groups

« Previous: 2 Summary of Presentations
Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×

3

Key Questions Identified by Discussion Groups

At the conclusion of the workshop, a rapporteur from each of the six afternoon discussion groups provided an oral summary of the group’s discussion. The summary comments are organized below according to elements of the statement of task for the Panel for Review of Best Practices in Assessment of Research and Development Organizations. (The panel’s statement of task is presented above on page 1 of this report.) The following summary comments are phrased as a set of questions that might be considered during assessments.

Organizational Context: Does the Organization’s Current and Planned Portfolio
Align with Its Mission, and Are the Organization’s Plans and Strategies Aligned
with the Needs of Its Customers and Stakeholders?

In their presentations, John Sommerer, Roy Levin, and J. Stephen Rottler highlighted the importance of recognizing and addressing the context within which a given R&D organization exists. James Turner emphasized that assessments should include measurement against organizational goals and intentions. Sommerer emphasized the importance of a clearly articulated vision of what the parent or organization is trying to achieve according to established milestones. Gilbert Decker emphasized that the following are key management functions: providing the resources available to support high-quality work, effectively delivering the services and products required to fulfill the organization’s goals and mission and to address the needs of its customers, and maintaining a current and planned R&D portfolio that supports the organization’s mission. William Banholzer noted that an industrial organization must remain mindful of three questions: What do people want? What will people pay for? What can they afford? Banholzer noted that assessments should consider the importance of R&D time frames and should address the question, Who are the customers and stakeholders, and how do they define success? Workshop discussants also identified the following questions:

•  Does the assessment reflect understanding of the principle that context is, indeed, fundamentally important, and that the whole organization should be assessed, not only individual programs, projects, units, or people?

•  Is the organizational environment created to produce outcomes?

•  What is the organization’s definition of success?

•  What are the appropriate time frames for research and development efforts being assessed?

•  Is the assessment in synchrony with changing time frames?

•  How is the relationship between the organization and its customers and stakeholders being addressed?

•  Does the agency that funds the research have an appreciation for research?

•  What is the level of direct interaction with customers?

Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×

•  Are there mechanisms in place to cut stagnant and unnecessary programs in order to prevent the dilution of the quality of more important programs?

•  How should the impact of programs in basic research be assessed?

•  How should innovation be assessed?

•  How should spin-offs and transitions be measured?

•  How are publication citations and patents—measures of the transition of knowledge—assessed?

•  Are assessment metrics focused on outcomes as well as on activities?

•  Are both historical impact and predictions of future success considered?

How Good Is the Technical Merit and Quality of the Science and
Engineering Work?

J. Stephen Rottler emphasized that there has been a need within his organization to shift from quantitative to qualitative assessment informed by data. Workshop discussants identified the following questions:

•  Is there external oversight of the assessment, even if the assessment is not being conducted by an external review board?

•  Is there a strong internal review to ensure that the product is not trivial before being submitted to external review? Is this applied to publications (especially for scientific publications) as well as to programs?

•  To allow for candor without worry about giving offense or meeting with reprisal, especially in small scientific communities, do external reviews include processes to preserve anonymity?

•  Are the terms of external review board members appropriate (generally between 3 and 5 years)?

•  Does the review team have a balance of expertise and backgrounds?

•  Do the review team members have good community reputations?

•  Does the chair of the review team show good judgment?

•  Has a clear tasking charge been provided for the assessment team?

•  Has the audience for assessment reports been identified?

•  Have mechanisms for both formal and informal communication of assessment findings been established?

•  Is benchmarking included in the assessment as a useful means for assessing process factors—that is, how things are accomplished within the organization?

What Are the Elements of Technical Management That Affect the
Quality of the Work?

Each of the six presenters examined elements of technical management that affect the quality of the R&D scientific and technical work. Workshop discussants identified the following questions:

Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×

•  Does the organization document processes and outputs when metrics are desired? Are the metrics that are gathered appropriate to permitting an examination of the data and trends for making decisions about actions to take?

•  How do organizational decision makers perform judgments based on assessment of risk?

•  Is a record kept of anecdotes, which often communicate accomplishments better than quantitative metrics such as publications and patents?

•  Is the organization’s management assessed? Does the assessment ask staff how well management is performing? Are the senior managers technically competent? Does the organization have in place mechanisms to remove pathological managers who will not hire individuals more competent than themselves?

Are Adequate Resources Available to Support High-Quality Work?

John Sommerer noted that a vision without resources is a hallucination. He and Roy Levin emphasized that human capital is a fundamental resource and that innovation requires that researchers be given some latitude and discretion with respect to their projects. Workshop discussants identified the following questions:

•  How well does the organization support the education and development of staff?

•  How well does the personnel selection and assessment system provide and maintain good performance? Are there any constraints on the system (e.g., constraints imposed within the federal context)?

•  Are incentives in place to recognize individuals and teams?

•  Are there efforts within the organization to seek external recognition?

•  Does the organization promote teamwork? Does management connect with the team to discover talent, through social networking and an open-door policy?

•  Does management communicate with junior researchers?

•  As a measure of teamwork, are common cross-discipline terminologies used by the staff and the management?

•  Do the staff members possess both technical and social skills?

•  As a measure of organizational flexibility, how do the initial academic degrees of staff compare with their current work tasks?

•  How does the organization inspire stellar performers? Does the organization have a rigorous and transparent process of rewards and acknowledgments in which the staff have confidence and faith?

•  Does the organization have mechanisms for moving aside ossified individuals and those who block the performance of others—mechanisms whereby nonperformers can be flushed out?

•  Are “wild ducks” (brilliant oddballs) identified and embraced?

•  Are rewards other than money available to staff?

Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×

•  Is there a mentoring system that creates a direct interaction between mentor and mentee in order to enhance productivity?

•  If the organization applies a force ranking system to compare the goals and productivity of individual employees with the productivity and goals of others, is the size of the organization adequate to allow meaningful comparison of similar individuals?

Is the Organization Able to Respond Flexibly to Changing Economic, Political,
Social, and Technological Contexts?

John Sommerer noted that the intersection of vision, people, and alignment is addressed by an examination of the organization’s agility, flexibility, and adaptability in the face of changing pressures, budgets, and external contexts. Roy Levin emphasized that the rationale for the existence of a research organization is to provide the capacity to deal quickly with the unknown and unexpected. He noted that disruptive technologies, new competitors, and new business models can occur suddenly and must be responded to. Workshop discussants identified the following questions:

•  Does the assessment reflect the differences in context for federal, industrial, academic, and national laboratory settings, which involve very different customers, stakeholders, missions, and goals?

•  What can be learned from assessment methods applied in other countries?

•  Does the organization foster participation in the global R&D community (e.g., by providing resources to prepare publications and attend conferences)?

Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×

This page intentionally left blank.

Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×
Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×

This page intentionally left blank.

Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×
Page 18
Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×
Page 19
Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×
Page 20
Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×
Page 21
Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×
Page 22
Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×
Page 23
Suggested Citation:"3 Key Questions Identified by Discussion Groups." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13527.
×
Page 24
Next: Appendix A Agenda for the Workshop »
Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Institute of Standards and Technology (NIST)--recognizing that information and insights gained through continual examination of practices for organizational assessment are useful for decision makers at organizations across the deferral, industrial, academic, and national laboratory sectors-recently requested that the National Research Council (NRC) organize a panel to review best practices in assessment of research and development (R&D) organizations. In response, the NRC established the Panel for Review of Best Practices in Assessment of Research and Development Organizations.

The panel was charged to consider means of assessing the following in a manner that satisfies the requirements of NIST to perform effective assessments but also identifies assessment methods that can be applied selectively to other R&D organizations. These methods include: technical merit and quality of the science and engineering work, the adequacy of the resources available to support high-quality work, the effectiveness of the agency's delivery of the services and products required to fulfill its goals, the degree to which the agency's current and planned R&D portfolio supports its mission, as well as the agency's flexibility to respond to changing economic, political, social and technological contexts.

As one means of data gathering, among others that the panel is performing toward development of a final report of its findings, the panel organized a planning committee for a workshop on best practices in assessment of R&D organizations. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop reviews the workshop conducted at the Keck Center of the National Academies in Washington, D.C., on March 19, 2012.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!