National Academies Press: OpenBook
« Previous: 2 Framework for Examination of Assessment Processes
Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

3

Assessing Management

Management of a research and development (R&D) organization is principally charged with two key tasks: establishing the vision and strategic plan for the organization and ensuring the preparedness of the organization to meet current commitments and future opportunities.1 In addition to supporting these management tasks, assessments may, for some organizations, serve the additional function of fulfilling statutory and other requirements. Appendix E provides a discussion of such requirements for U.S. government agencies and U.S. government funds, including the Government Performance and Results Act of 1993 (GPRA), P.L. 103-62; the GPRA Modernization Act of 2010 (GPRAMA), P.L. 111-352; and the Office of Science and Technology Policy (OSTP) and Office of Management and Budget (OMB) FY—Administration Research and Development Budget Priorities document, which generally specifies that R&D programs will be assessed for quality, relevance, and performance, and indicates that budget decisions will be made based on these assessments.

CUSTOMERS AND STAKEHOLDERS

As part of an effective assessment process, the organization identifies its stakeholders and customers and ensures that their needs are addressed to their satisfaction. Definitions of “customer” and ”stakeholder” vary. A customer is often viewed as someone, within or outside the organization, who purchases products from the organization or its elements. A stakeholder may be viewed as an entity that can impact the organization’s vision, mission, plans, or resources. Customers may differ from or be a subset of stakeholders. Definitions are arguable, but the point remains that an effective assessment determines whether the organization has developed a clear and meaningful identification of its set of customers and stakeholders and the means for identifying and satisfying their needs.

Planning its research portfolio requires that the organization obtain inputs from stakeholders and customers. Reaching these individuals is important but in some cases may be difficult—especially the task of reaching customers. Sometimes a means of aggregating customers is available. For industry, trade associations can be asked to survey their members. Likewise, some scientific or engineering societies may have formal means of collecting information about the needs of their members. In another example, the state organizations concerned with accuracy in weights and measures are brought together in the National Conference on Weights and Measures (NCWM). The NCWM holds regular meetings during which problems facing the community are brought into focus. For organizations with very broad scope, such aggregation of customers and stakeholders is essential. For example, farmers are aggregated in various ways to interact with state and national technical organizations. One of these, the U.S. Department of Agriculture (USDA) Extension Service, applies very good

____________________________

1 RAND Corporation. 2012. Improving Army Basic Research: Report of an Expert Panel on the Future of Army Laboratories. RAND Arroyo Center, Santa Monica, Calif.

Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

processes for learning about problems in the field and for transitioning research findings to the farmer. In some other government organizations, the customers are focused narrowly and are easy to reach more directly. For the Department of Defense (DOD) the warfighters are represented by the training schools and the developers and keepers of military doctrine. The military organizations can communicate with these groups and can actually rotate their staff to and from these customer organizations. Holding regular planning meetings with its customers helps to ensure that the organization’s programs are relevant.

Effective processes for addressing customer and stakeholder satisfaction include the following:

•  A systematic process to identify all stakeholders and make their identities part of the public record;

•  The identification of individual(s) within the organization to be assigned the responsibility for interaction with each of the stakeholders or with organizations representing the stakeholders (e.g., industrial consortia or associations);

•  A well-defined, open process for sharing information with stakeholders at all stages of R&D, including planning, execution, and delivery of results; and

•  On a regular basis, internal review by management of this system of stakeholder interaction, and at intervals, review by outsiders using standing or ad hoc committees of expert advisers.

Appendix F describes the history of the process applied by the Army Research Laboratory (ARL) to assess the relevance of its R&D. This case study demonstrates an approach to identifying and responding to the needs of an organization’s stakeholders. Appendix G provides a discussion of stakeholder relationships for laboratories at the DOD, the Department of Energy (DOE), and NIST.

RESOURCE MANAGEMENT

An effective assessment is conducted in the context of the organization’s mission. In order to satisfy its mission, an organization will seek to be prepared to handle its current and future workload, and this will have a successful combination of the following elements:

•  R&D portfolio—a collection of projects that are most likely to lead to the successful accomplishment of the organization’s mission;

•  Resources—a workforce with an appropriate skill balance; the needed physical plant and equipment; and sufficient funding to enable the mission;

•  Organizational leadership and management structure appropriate to the mission; and

•  Planning for the future—the preparedness needed to ensure that the required resources will be in place as the mission evolves.

These elements are properly considered in context. Academic research focuses on generating new knowledge with relatively few mission objectives, whereas government and industrial research organizations have fairly clearly defined missions. An effective management assessment also recognizes externally imposed limitations, including but not limited to regulatory and budgetary restrictions.

Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

Portfolio

At all stages of R&D, it is important that the institution construct and manage its portfolio to maximize the probability of success. In basic research, it is important that the portfolio cover those areas that are likely to be important to the ultimate mission and that the assessors look at the portfolio and indicate whether there are areas that may be missing and whether there are areas covered that may not be very relevant. (These are sometimes very subjective evaluations.) In the product development portfolio, the areas are often well specified, but it is important to consider whether or not the correct technologies are being evaluated for possible use in a future product. As explained above, portfolios can be assessed during the planning phase, ongoing research phase, and retrospectively.

Three definable elements to consider in assessing the quality of a technical portfolio are (1) current projects and their relevance to the mission; (2) anticipation of opportunities; and (3) alignment of the planned future portfolio to mission, opportunities, and budget.

Generally an R&D organization has some way of cataloging its research portfolio. At the extreme of little specificity is funding per group with descriptions of the group responsibilities and recent accomplishments. This format is most common at the more basic or fundamental end of the R&D spectrum. At the more applied extreme are examples in some industrial organizations in which each project is specified in great detail, including timelines and anticipated return on investment.

In many respects surveying the research portfolio of a single organization (or a unit in that organization) is a straightforward process. Individual projects are grouped under programs and evaluated in concert with stakeholders’ needs and expectations. This process is often done through organized meetings. It may be a continuing process that includes formal oversight from those stakeholders as well as outside reviewers and consultants. Industrial organizations may rely heavily on metrics involving financial return, whereas government organizations may focus more on delivering needed value to stakeholders, consistent with mission statements.

When more than one organization is involved in the portfolio being examined, because of differences in the missions, history, and/or policies, a more complex situation of portfolio management occurs. In that case, a project may have different objectives within the context of each organization.

Workforce and Physical Resources Management

An effective assessment considers the adequacy of both the workforce and the physical facilities and equipment available to address the mission. Because the skills and capabilities of the workforce and the equipment and facilities available to them are key factors affecting the quality of the R&D work performed, workforce and physical resources are discussed in more detail in the Chapter 4, which discusses assessing the quality of the work. It is noted here that it is the responsibility of management to ensure that effective assessments of the available and needed workforce and physical resources are performed and that the results of these assessments are addressed with an eye toward meeting the requirements of the organization’s vision, mission,

Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

stakeholders, and customers.2,3

Organizational Leadership and Management

As stated above, understanding the context within which an organization conducts its R&D is fundamentally important during its assessment. Effective management structure will be appropriate to the nature of the work. Basic research typically requires a very flat management structure, significant individual freedom in selection of research directions, and a management very receptive to suggestions (although projects involving very large experimental resources such as accelerators may require more structure). Product development typically requires a more hierarchical structure in order to ensure mission progress, although a fast-to-market product requires a streamlined structure.

A research organization is an open system with input (the external environment), throughput (the organization and its components), and output (performance by individuals, groups [units], and the total organization). A feedback loop connects output with input and provides a continuous process.

Organizations perform many functions, activities, and events that define their existence. What happens on a daily basis determines what the organizations are. These many aspects of organizations can be quantitatively and interactively overwhelming. Therefore, a conceptual framework that summarizes and simplifies this organizational complexity can be helpful for more effective leadership and management. One such framework is that proposed by the Burke-Litwin model.4,5

The following, based on the Burke-Litwin model, is an abbreviated set of questions proposed for consideration in an evaluation of leadership and management; the mechanisms for addressing them can vary: Do senior leaders monitor the external environment (consumers, stakeholders, technology, scientific community), gather relevant information, and share it across the organization? Does the research organization have a clear statement of purpose that shows a clear link to the external environment? Does the organization have a clear process for executing the mission? What long-range planning and forecasting are done to address future needs (technology, staff, facilities, equipment)? Are employee satisfaction factors addressed so that the employees focus on their work and good performers remain and attract others? Are those selected for leadership chosen according to clear and evidence-based criteria such as technical and managerial competence, self-awareness, relevant experience, learning agility, vision, and energy? Are creativity and performance recognized and rewarded? Are people, vision, and mission aligned? Does the organizational structure encourage communication? Are organizational members provided with the resources they need? Are managers and other staff appropriately involved in decision making? Are there clear, reliable, and valid metrics for

____________________________

2 National Research Council, 2010. Capabilities for the Future: An Assessment of NASA Laboratories for Basic Research. The National Academies Press, Washington, D.C.

3 Y. Saad, 2012. “Review of IBM’s Technical Review Path.” Presentation to the National Research Council’s Panel for Review of Best Practices in Assessment of Research and Development Organizations, March 20, Washington, D.C.

4 W.W. Burke and G.H. Litwin, 1992. A causal model of organizational performance and change. Journal of Management 18 (3):523-545.

5 W.W. Burke, 2010. Organization Change: Theory and Practice. Sage Publications, Thousand Oaks, Calif.

Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

evaluating individual performance, work unit performance, managerial performance, and performance of the organization overall?

A more detailed list of questions, also based on the Burke-Litwin model, is presented in Appendix H. Many of the questions presented here and in Appendix H may be addressed in a self-evaluation process by a management that is interested in continuous improvement. If the purpose of the review is internal—that is, requested by management to assist it in an assessment process, it can be handled by self-assessment or by invited external peers. However, for an externally mandated evaluation of all aspects of the organization, some questions have to be investigated by external reviewers.

Some of the questions listed here and in Appendix H need to allow for anonymous input from staff. This can be done in several ways, including the use of “skip-level” meetings, which are meetings between external reviewers and non-management staff in the absence of managers, during which anonymity is assured to the staff. These meetings can be valuable for identifying potential problems that may not have come to the attention of management. Some assessments organized by the National Research Council have used skip-level meetings, and it has been found that they are most successful when a few external reviewers meet with relatively small groups of employees at similar points in their careers (a small number of staff, so that there are opportunities for most to contribute) and at varying levels of leadership. It is important to communicate to the staff involved that any issues identified at such a meeting might be discussed with management, but that no individual attendee will be identified. It is helpful for the reviewers to have suitable open-ended questions to ask the group. For example: What aspects of the job make you eager to come in each day—and what aspects of the job make you dread coming in? It is important that the reviewers present attempt to prevent staff input from being monopolized by a single individual, and if one individual seems to focus on a small set of issues, it is important to try to determine if complaints are widely shared. It is also important that reviewers avoid leading questions that are designed to elicit specific responses. Information derived from these meetings is best used for the development of hypotheses for further investigation rather than as data from which conclusions can be immediately drawn, given that there are issues associated with sample size and selection.

Other means of garnering input from staff include the following: a parallel organization exercise (a temporary arrangement whereby formal authority is suspended and people can speak openly without fear of retribution), the purpose being to address issues not adequately addressed within the formal structure; and an informal relations survey (an organizational-network analysis consisting of a computer-generated map of the informal organization, identifying individuals to whom others reach out regardless of position or rank).

Planning for the Future

It is important that management continually plan for the future so that the organization is always in a position to fulfill its mission. This means that the right resources and leadership will be in place when needed. The demographics of the workforce will be tracked so that there will be leaders in place as retirements and departures occur, and people with new skill sets will be recruited and trained so that the staff is ready to deal with new technologies. An effective assessment considers the adequacy of this planning as well as plans for the necessary physical infrastructure to support the new skill sets and technologies.

Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

SUMMARY OF FINDINGS

The key components of the management of an organization considered in an effective assessment are the R&D portfolio, resources, the organizational leadership and management structure, and planning for the future.

Research is characterized by the fact that the result of any particular approach is not known until the research has been performed. Hence it is important that a research program—even one with a specific mission—examine several alternatives in a portfolio of projects. Similarly, in development it is usually not known which technology will lead to the best (most cost-effective, efficient) solution, and so it is important to consider a portfolio of technologies. For an effective assessment, resources will be evaluated to determine if they are adequate for the proposed or ongoing program, but the evaluation will also recognize the external resource limitations (budgetary, regulatory, personnel). To be effective, research leadership and structure will be adapted to the environment in which it operates—no one model suffices for all styles of research organizations. The future always brings improved science and technology, and so if the parent organization is to remain competitive, it is important that its R&D units continuously upgrade the skill sets of its workforce and match these with adequate research facilities.

Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 23
Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 24
Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 25
Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 26
Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 27
Suggested Citation:"3 Assessing Management." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 28
Next: 4 Assessing Technical Quality »
Best Practices in Assessment of Research and Development Organizations Get This Book
×
Buy Paperback | $37.00 Buy Ebook | $29.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Research and development (R&D) organizations are operated by government, business, academe, and independent institutes. The success of their parent organizations is closely tied to the success of these R&D organizations. In this report, organizations refers to an organization that performs research and/or development activities (often a laboratory), and parent refers to the superordinate organization of which the R&D organization is a part. When the organization under discussion is formally labeled a laboratory, it is referred to as such. The question arises: How does one know whether an organization and its programs are achieving excellence in the best interests of its parent? Does the organization have an appropriate research staff, facilities, and equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does it lead to successful new concepts, products, or processes that support the interests of its parent?

This report offers assessment guidelines for senior management of organizations and of their parents. The report lists the major principles of assessment, noting that details will vary from one organization to another. It provides sufficient information to inform the design of assessments, but it does not prescribe precisely how to perform them, because different techniques are needed for different types of organizations.

Best Practices in Assessment of Research and Development Organizations covers three key factors that underpin the success of an R&D organization: (1) the mission of the organization and its alignment with that of the parents; (2) the relevance and impact of the organization's work; and (3) the resources provided to the organization, beginning with a high-quality staff and management.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!