3

Assessing Management

Management of a research and development (R&D) organization is principally charged with two key tasks: establishing the vision and strategic plan for the organization and ensuring the preparedness of the organization to meet current commitments and future opportunities.1 In addition to supporting these management tasks, assessments may, for some organizations, serve the additional function of fulfilling statutory and other requirements. Appendix E provides a discussion of such requirements for U.S. government agencies and U.S. government funds, including the Government Performance and Results Act of 1993 (GPRA), P.L. 103-62; the GPRA Modernization Act of 2010 (GPRAMA), P.L. 111-352; and the Office of Science and Technology Policy (OSTP) and Office of Management and Budget (OMB) FY—Administration Research and Development Budget Priorities document, which generally specifies that R&D programs will be assessed for quality, relevance, and performance, and indicates that budget decisions will be made based on these assessments.

CUSTOMERS AND STAKEHOLDERS

As part of an effective assessment process, the organization identifies its stakeholders and customers and ensures that their needs are addressed to their satisfaction. Definitions of “customer” and ”stakeholder” vary. A customer is often viewed as someone, within or outside the organization, who purchases products from the organization or its elements. A stakeholder may be viewed as an entity that can impact the organization’s vision, mission, plans, or resources. Customers may differ from or be a subset of stakeholders. Definitions are arguable, but the point remains that an effective assessment determines whether the organization has developed a clear and meaningful identification of its set of customers and stakeholders and the means for identifying and satisfying their needs.

Planning its research portfolio requires that the organization obtain inputs from stakeholders and customers. Reaching these individuals is important but in some cases may be difficult—especially the task of reaching customers. Sometimes a means of aggregating customers is available. For industry, trade associations can be asked to survey their members. Likewise, some scientific or engineering societies may have formal means of collecting information about the needs of their members. In another example, the state organizations concerned with accuracy in weights and measures are brought together in the National Conference on Weights and Measures (NCWM). The NCWM holds regular meetings during which problems facing the community are brought into focus. For organizations with very broad scope, such aggregation of customers and stakeholders is essential. For example, farmers are aggregated in various ways to interact with state and national technical organizations. One of these, the U.S. Department of Agriculture (USDA) Extension Service, applies very good

____________________________

1 RAND Corporation. 2012. Improving Army Basic Research: Report of an Expert Panel on the Future of Army Laboratories. RAND Arroyo Center, Santa Monica, Calif.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 23
3 Assessing Management Management of a research and development (R&D) organization is principally charged with two key tasks: establishing the vision and strategic plan for the organization and ensuring the preparedness of the organization to meet current commitments and future opportunities.1 In addition to supporting these management tasks, assessments may, for some organizations, serve the additional function of fulfilling statutory and other requirements. Appendix E provides a discussion of such requirements for U.S. government agencies and U.S. government funds, including the Government Performance and Results Act of 1993 (GPRA), P.L. 103-62; the GPRA Modernization Act of 2010 (GPRAMA), P.L. 111-352; and the Office of Science and Technology Policy (OSTP) and Office of Management and Budget (OMB) FY--Administration Research and Development Budget Priorities document, which generally specifies that R&D programs will be assessed for quality, relevance, and performance, and indicates that budget decisions will be made based on these assessments. . CUSTOMERS AND STAKEHOLDERS As part of an effective assessment process, the organization identifies its stakeholders and customers and ensures that their needs are addressed to their satisfaction. Definitions of "customer" and "stakeholder" vary. A customer is often viewed as someone, within or outside the organization, who purchases products from the organization or its elements. A stakeholder may be viewed as an entity that can impact the organization's vision, mission, plans, or resources. Customers may differ from or be a subset of stakeholders. Definitions are arguable, but the point remains that an effective assessment determines whether the organization has developed a clear and meaningful identification of its set of customers and stakeholders and the means for identifying and satisfying their needs. Planning its research portfolio requires that the organization obtain inputs from stakeholders and customers. Reaching these individuals is important but in some cases may be difficult--especially the task of reaching customers. Sometimes a means of aggregating customers is available. For industry, trade associations can be asked to survey their members. Likewise, some scientific or engineering societies may have formal means of collecting information about the needs of their members. In another example, the state organizations concerned with accuracy in weights and measures are brought together in the National Conference on Weights and Measures (NCWM). The NCWM holds regular meetings during which problems facing the community are brought into focus. For organizations with very broad scope, such aggregation of customers and stakeholders is essential. For example, farmers are aggregated in various ways to interact with state and national technical organizations. One of these, the U.S. Department of Agriculture (USDA) Extension Service, applies very good 1 RAND Corporation. 2012. Improving Army Basic Research: Report of an Expert Panel on the Future of Army Laboratories. RAND Arroyo Center, Santa Monica, Calif. 23

OCR for page 23
processes for learning about problems in the field and for transitioning research findings to the farmer. In some other government organizations, the customers are focused narrowly and are easy to reach more directly. For the Department of Defense (DOD) the warfighters are represented by the training schools and the developers and keepers of military doctrine. The military organizations can communicate with these groups and can actually rotate their staff to and from these customer organizations. Holding regular planning meetings with its customers helps to ensure that the organization's programs are relevant. Effective processes for addressing customer and stakeholder satisfaction include the following: A systematic process to identify all stakeholders and make their identities part of the public record; The identification of individual(s) within the organization to be assigned the responsibility for interaction with each of the stakeholders or with organizations representing the stakeholders (e.g., industrial consortia or associations); A well-defined, open process for sharing information with stakeholders at all stages of R&D, including planning, execution, and delivery of results; and On a regular basis, internal review by management of this system of stakeholder interaction, and at intervals, review by outsiders using standing or ad hoc committees of expert advisers. Appendix F describes the history of the process applied by the Army Research Laboratory (ARL) to assess the relevance of its R&D. This case study demonstrates an approach to identifying and responding to the needs of an organization's stakeholders. Appendix G provides a discussion of stakeholder relationships for laboratories at the DOD, the Department of Energy (DOE), and NIST. RESOURCE MANAGEMENT An effective assessment is conducted in the context of the organization's mission. In order to satisfy its mission, an organization will seek to be prepared to handle its current and future workload, and this will have a successful combination of the following elements: R&D portfolio--a collection of projects that are most likely to lead to the successful accomplishment of the organization's mission; Resources--a workforce with an appropriate skill balance; the needed physical plant and equipment; and sufficient funding to enable the mission; Organizational leadership and management structure appropriate to the mission; and Planning for the future--the preparedness needed to ensure that the required resources will be in place as the mission evolves. These elements are properly considered in context. Academic research focuses on generating new knowledge with relatively few mission objectives, whereas government and industrial research organizations have fairly clearly defined missions. An effective management assessment also recognizes externally imposed limitations, including but not limited to regulatory and budgetary restrictions. 24

OCR for page 23
Portfolio At all stages of R&D, it is important that the institution construct and manage its portfolio to maximize the probability of success. In basic research, it is important that the portfolio cover those areas that are likely to be important to the ultimate mission and that the assessors look at the portfolio and indicate whether there are areas that may be missing and whether there are areas covered that may not be very relevant. (These are sometimes very subjective evaluations.) In the product development portfolio, the areas are often well specified, but it is important to consider whether or not the correct technologies are being evaluated for possible use in a future product. As explained above, portfolios can be assessed during the planning phase, ongoing research phase, and retrospectively. Three definable elements to consider in assessing the quality of a technical portfolio are (1) current projects and their relevance to the mission; (2) anticipation of opportunities; and (3) alignment of the planned future portfolio to mission, opportunities, and budget. Generally an R&D organization has some way of cataloging its research portfolio. At the extreme of little specificity is funding per group with descriptions of the group responsibilities and recent accomplishments. This format is most common at the more basic or fundamental end of the R&D spectrum. At the more applied extreme are examples in some industrial organizations in which each project is specified in great detail, including timelines and anticipated return on investment. In many respects surveying the research portfolio of a single organization (or a unit in that organization) is a straightforward process. Individual projects are grouped under programs and evaluated in concert with stakeholders' needs and expectations. This process is often done through organized meetings. It may be a continuing process that includes formal oversight from those stakeholders as well as outside reviewers and consultants. Industrial organizations may rely heavily on metrics involving financial return, whereas government organizations may focus more on delivering needed value to stakeholders, consistent with mission statements. When more than one organization is involved in the portfolio being examined, because of differences in the missions, history, and/or policies, a more complex situation of portfolio management occurs. In that case, a project may have different objectives within the context of each organization. Workforce and Physical Resources Management An effective assessment considers the adequacy of both the workforce and the physical facilities and equipment available to address the mission. Because the skills and capabilities of the workforce and the equipment and facilities available to them are key factors affecting the quality of the R&D work performed, workforce and physical resources are discussed in more detail in the Chapter 4, which discusses assessing the quality of the work. It is noted here that it is the responsibility of management to ensure that effective assessments of the available and needed workforce and physical resources are performed and that the results of these assessments are addressed with an eye toward meeting the requirements of the organization's vision, mission, 25

OCR for page 23
stakeholders, and customers.2,3 Organizational Leadership and Management As stated above, understanding the context within which an organization conducts its R&D is fundamentally important during its assessment. Effective management structure will be appropriate to the nature of the work. Basic research typically requires a very flat management structure, significant individual freedom in selection of research directions, and a management very receptive to suggestions (although projects involving very large experimental resources such as accelerators may require more structure). Product development typically requires a more hierarchical structure in order to ensure mission progress, although a fast-to-market product requires a streamlined structure. A research organization is an open system with input (the external environment), throughput (the organization and its components), and output (performance by individuals, groups [units], and the total organization). A feedback loop connects output with input and provides a continuous process. Organizations perform many functions, activities, and events that define their existence. What happens on a daily basis determines what the organizations are. These many aspects of organizations can be quantitatively and interactively overwhelming. Therefore, a conceptual framework that summarizes and simplifies this organizational complexity can be helpful for more effective leadership and management. One such framework is that proposed by the Burke- Litwin model.4,5 The following, based on the Burke-Litwin model, is an abbreviated set of questions proposed for consideration in an evaluation of leadership and management; the mechanisms for addressing them can vary: Do senior leaders monitor the external environment (consumers, stakeholders, technology, scientific community), gather relevant information, and share it across the organization? Does the research organization have a clear statement of purpose that shows a clear link to the external environment? Does the organization have a clear process for executing the mission? What long-range planning and forecasting are done to address future needs (technology, staff, facilities, equipment)? Are employee satisfaction factors addressed so that the employees focus on their work and good performers remain and attract others? Are those selected for leadership chosen according to clear and evidence-based criteria such as technical and managerial competence, self-awareness, relevant experience, learning agility, vision, and energy? Are creativity and performance recognized and rewarded? Are people, vision, and mission aligned? Does the organizational structure encourage communication? Are organizational members provided with the resources they need? Are managers and other staff appropriately involved in decision making? Are there clear, reliable, and valid metrics for 2 National Research Council, 2010. Capabilities for the Future: An Assessment of NASA Laboratories for Basic Research. The National Academies Press, Washington, D.C. 3 Y. Saad, 2012. "Review of IBM's Technical Review Path." Presentation to the National Research Council's Panel for Review of Best Practices in Assessment of Research and Development Organizations, March 20, Washington, D.C. 4 W.W. Burke and G.H. Litwin, 1992. A causal model of organizational performance and change. Journal of Management 18 (3):523-545. 5 W.W. Burke, 2010. Organization Change: Theory and Practice. Sage Publications, Thousand Oaks, Calif. 26

OCR for page 23
evaluating individual performance, work unit performance, managerial performance, and performance of the organization overall? A more detailed list of questions, also based on the Burke-Litwin model, is presented in Appendix H. Many of the questions presented here and in Appendix H may be addressed in a self-evaluation process by a management that is interested in continuous improvement. If the purpose of the review is internal--that is, requested by management to assist it in an assessment process, it can be handled by self-assessment or by invited external peers. However, for an externally mandated evaluation of all aspects of the organization, some questions have to be investigated by external reviewers. Some of the questions listed here and in Appendix H need to allow for anonymous input from staff. This can be done in several ways, including the use of "skip-level" meetings, which are meetings between external reviewers and non-management staff in the absence of managers, during which anonymity is assured to the staff. These meetings can be valuable for identifying potential problems that may not have come to the attention of management. Some assessments organized by the National Research Council have used skip-level meetings, and it has been found that they are most successful when a few external reviewers meet with relatively small groups of employees at similar points in their careers (a small number of staff, so that there are opportunities for most to contribute) and at varying levels of leadership. It is important to communicate to the staff involved that any issues identified at such a meeting might be discussed with management, but that no individual attendee will be identified. It is helpful for the reviewers to have suitable open-ended questions to ask the group. For example: What aspects of the job make you eager to come in each day--and what aspects of the job make you dread coming in? It is important that the reviewers present attempt to prevent staff input from being monopolized by a single individual, and if one individual seems to focus on a small set of issues, it is important to try to determine if complaints are widely shared. It is also important that reviewers avoid leading questions that are designed to elicit specific responses. Information derived from these meetings is best used for the development of hypotheses for further investigation rather than as data from which conclusions can be immediately drawn, given that there are issues associated with sample size and selection. Other means of garnering input from staff include the following: a parallel organization exercise (a temporary arrangement whereby formal authority is suspended and people can speak openly without fear of retribution), the purpose being to address issues not adequately addressed within the formal structure; and an informal relations survey (an organizational-network analysis consisting of a computer-generated map of the informal organization, identifying individuals to whom others reach out regardless of position or rank). Planning for the Future It is important that management continually plan for the future so that the organization is always in a position to fulfill its mission. This means that the right resources and leadership will be in place when needed. The demographics of the workforce will be tracked so that there will be leaders in place as retirements and departures occur, and people with new skill sets will be recruited and trained so that the staff is ready to deal with new technologies. An effective assessment considers the adequacy of this planning as well as plans for the necessary physical infrastructure to support the new skill sets and technologies. 27

OCR for page 23
SUMMARY OF FINDINGS The key components of the management of an organization considered in an effective assessment are the R&D portfolio, resources, the organizational leadership and management structure, and planning for the future. Research is characterized by the fact that the result of any particular approach is not known until the research has been performed. Hence it is important that a research program-- even one with a specific mission--examine several alternatives in a portfolio of projects. Similarly, in development it is usually not known which technology will lead to the best (most cost-effective, efficient) solution, and so it is important to consider a portfolio of technologies. For an effective assessment, resources will be evaluated to determine if they are adequate for the proposed or ongoing program, but the evaluation will also recognize the external resource limitations (budgetary, regulatory, personnel). To be effective, research leadership and structure will be adapted to the environment in which it operates--no one model suffices for all styles of research organizations. The future always brings improved science and technology, and so if the parent organization is to remain competitive, it is important that its R&D units continuously upgrade the skill sets of its workforce and match these with adequate research facilities. 28