Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 23
3
Assessing Management
Management of a research and development (R&D) organization is principally charged
with two key tasks: establishing the vision and strategic plan for the organization and ensuring
the preparedness of the organization to meet current commitments and future opportunities.1 In
addition to supporting these management tasks, assessments may, for some organizations, serve
the additional function of fulfilling statutory and other requirements. Appendix E provides a
discussion of such requirements for U.S. government agencies and U.S. government funds,
including the Government Performance and Results Act of 1993 (GPRA), P.L. 103-62; the
GPRA Modernization Act of 2010 (GPRAMA), P.L. 111-352; and the Office of Science and
Technology Policy (OSTP) and Office of Management and Budget (OMB) FY--Administration
Research and Development Budget Priorities document, which generally specifies that R&D
programs will be assessed for quality, relevance, and performance, and indicates that budget
decisions will be made based on these assessments.
.
CUSTOMERS AND STAKEHOLDERS
As part of an effective assessment process, the organization identifies its stakeholders and
customers and ensures that their needs are addressed to their satisfaction. Definitions of
"customer" and "stakeholder" vary. A customer is often viewed as someone, within or outside
the organization, who purchases products from the organization or its elements. A stakeholder
may be viewed as an entity that can impact the organization's vision, mission, plans, or
resources. Customers may differ from or be a subset of stakeholders. Definitions are arguable,
but the point remains that an effective assessment determines whether the organization has
developed a clear and meaningful identification of its set of customers and stakeholders and the
means for identifying and satisfying their needs.
Planning its research portfolio requires that the organization obtain inputs from
stakeholders and customers. Reaching these individuals is important but in some cases may be
difficult--especially the task of reaching customers. Sometimes a means of aggregating
customers is available. For industry, trade associations can be asked to survey their members.
Likewise, some scientific or engineering societies may have formal means of collecting
information about the needs of their members. In another example, the state organizations
concerned with accuracy in weights and measures are brought together in the National
Conference on Weights and Measures (NCWM). The NCWM holds regular meetings during
which problems facing the community are brought into focus. For organizations with very broad
scope, such aggregation of customers and stakeholders is essential. For example, farmers are
aggregated in various ways to interact with state and national technical organizations. One of
these, the U.S. Department of Agriculture (USDA) Extension Service, applies very good
1
RAND Corporation. 2012. Improving Army Basic Research: Report of an Expert Panel on the Future of Army
Laboratories. RAND Arroyo Center, Santa Monica, Calif.
23
OCR for page 24
processes for learning about problems in the field and for transitioning research findings to the
farmer. In some other government organizations, the customers are focused narrowly and are
easy to reach more directly. For the Department of Defense (DOD) the warfighters are
represented by the training schools and the developers and keepers of military doctrine. The
military organizations can communicate with these groups and can actually rotate their staff to
and from these customer organizations. Holding regular planning meetings with its customers
helps to ensure that the organization's programs are relevant.
Effective processes for addressing customer and stakeholder satisfaction include the
following:
A systematic process to identify all stakeholders and make their identities part of the
public record;
The identification of individual(s) within the organization to be assigned the
responsibility for interaction with each of the stakeholders or with organizations
representing the stakeholders (e.g., industrial consortia or associations);
A well-defined, open process for sharing information with stakeholders at all stages
of R&D, including planning, execution, and delivery of results; and
On a regular basis, internal review by management of this system of stakeholder
interaction, and at intervals, review by outsiders using standing or ad hoc committees
of expert advisers.
Appendix F describes the history of the process applied by the Army Research
Laboratory (ARL) to assess the relevance of its R&D. This case study demonstrates an approach
to identifying and responding to the needs of an organization's stakeholders. Appendix G
provides a discussion of stakeholder relationships for laboratories at the DOD, the Department of
Energy (DOE), and NIST.
RESOURCE MANAGEMENT
An effective assessment is conducted in the context of the organization's mission. In
order to satisfy its mission, an organization will seek to be prepared to handle its current and
future workload, and this will have a successful combination of the following elements:
R&D portfolio--a collection of projects that are most likely to lead to the successful
accomplishment of the organization's mission;
Resources--a workforce with an appropriate skill balance; the needed physical plant
and equipment; and sufficient funding to enable the mission;
Organizational leadership and management structure appropriate to the mission; and
Planning for the future--the preparedness needed to ensure that the required
resources will be in place as the mission evolves.
These elements are properly considered in context. Academic research focuses on generating
new knowledge with relatively few mission objectives, whereas government and industrial
research organizations have fairly clearly defined missions. An effective management
assessment also recognizes externally imposed limitations, including but not limited to regulatory
and budgetary restrictions.
24
OCR for page 25
Portfolio
At all stages of R&D, it is important that the institution construct and manage its portfolio
to maximize the probability of success. In basic research, it is important that the portfolio cover
those areas that are likely to be important to the ultimate mission and that the assessors look at
the portfolio and indicate whether there are areas that may be missing and whether there are areas
covered that may not be very relevant. (These are sometimes very subjective evaluations.) In
the product development portfolio, the areas are often well specified, but it is important to
consider whether or not the correct technologies are being evaluated for possible use in a future
product. As explained above, portfolios can be assessed during the planning phase, ongoing
research phase, and retrospectively.
Three definable elements to consider in assessing the quality of a technical portfolio are
(1) current projects and their relevance to the mission; (2) anticipation of opportunities; and (3)
alignment of the planned future portfolio to mission, opportunities, and budget.
Generally an R&D organization has some way of cataloging its research portfolio. At the
extreme of little specificity is funding per group with descriptions of the group responsibilities
and recent accomplishments. This format is most common at the more basic or fundamental end
of the R&D spectrum. At the more applied extreme are examples in some industrial
organizations in which each project is specified in great detail, including timelines and
anticipated return on investment.
In many respects surveying the research portfolio of a single organization (or a unit in
that organization) is a straightforward process. Individual projects are grouped under programs
and evaluated in concert with stakeholders' needs and expectations. This process is often done
through organized meetings. It may be a continuing process that includes formal oversight from
those stakeholders as well as outside reviewers and consultants. Industrial organizations may
rely heavily on metrics involving financial return, whereas government organizations may focus
more on delivering needed value to stakeholders, consistent with mission statements.
When more than one organization is involved in the portfolio being examined, because of
differences in the missions, history, and/or policies, a more complex situation of portfolio
management occurs. In that case, a project may have different objectives within the context of
each organization.
Workforce and Physical Resources Management
An effective assessment considers the adequacy of both the workforce and the physical
facilities and equipment available to address the mission. Because the skills and capabilities of
the workforce and the equipment and facilities available to them are key factors affecting the
quality of the R&D work performed, workforce and physical resources are discussed in more
detail in the Chapter 4, which discusses assessing the quality of the work. It is noted here that it
is the responsibility of management to ensure that effective assessments of the available and
needed workforce and physical resources are performed and that the results of these assessments
are addressed with an eye toward meeting the requirements of the organization's vision, mission,
25
OCR for page 26
stakeholders, and customers.2,3
Organizational Leadership and Management
As stated above, understanding the context within which an organization conducts its
R&D is fundamentally important during its assessment. Effective management structure will be
appropriate to the nature of the work. Basic research typically requires a very flat management
structure, significant individual freedom in selection of research directions, and a management
very receptive to suggestions (although projects involving very large experimental resources
such as accelerators may require more structure). Product development typically requires a more
hierarchical structure in order to ensure mission progress, although a fast-to-market product
requires a streamlined structure.
A research organization is an open system with input (the external environment),
throughput (the organization and its components), and output (performance by individuals,
groups [units], and the total organization). A feedback loop connects output with input and
provides a continuous process.
Organizations perform many functions, activities, and events that define their existence.
What happens on a daily basis determines what the organizations are. These many aspects of
organizations can be quantitatively and interactively overwhelming. Therefore, a conceptual
framework that summarizes and simplifies this organizational complexity can be helpful for
more effective leadership and management. One such framework is that proposed by the Burke-
Litwin model.4,5
The following, based on the Burke-Litwin model, is an abbreviated set of questions
proposed for consideration in an evaluation of leadership and management; the mechanisms for
addressing them can vary: Do senior leaders monitor the external environment (consumers,
stakeholders, technology, scientific community), gather relevant information, and share it across
the organization? Does the research organization have a clear statement of purpose that shows a
clear link to the external environment? Does the organization have a clear process for executing
the mission? What long-range planning and forecasting are done to address future needs
(technology, staff, facilities, equipment)? Are employee satisfaction factors addressed so that the
employees focus on their work and good performers remain and attract others? Are those
selected for leadership chosen according to clear and evidence-based criteria such as technical
and managerial competence, self-awareness, relevant experience, learning agility, vision, and
energy? Are creativity and performance recognized and rewarded? Are people, vision, and
mission aligned? Does the organizational structure encourage communication? Are
organizational members provided with the resources they need? Are managers and other staff
appropriately involved in decision making? Are there clear, reliable, and valid metrics for
2
National Research Council, 2010. Capabilities for the Future: An Assessment of NASA Laboratories for Basic
Research. The National Academies Press, Washington, D.C.
3
Y. Saad, 2012. "Review of IBM's Technical Review Path." Presentation to the National Research Council's
Panel for Review of Best Practices in Assessment of Research and Development Organizations, March 20,
Washington, D.C.
4
W.W. Burke and G.H. Litwin, 1992. A causal model of organizational performance and change. Journal of
Management 18 (3):523-545.
5
W.W. Burke, 2010. Organization Change: Theory and Practice. Sage Publications, Thousand Oaks, Calif.
26
OCR for page 27
evaluating individual performance, work unit performance, managerial performance, and
performance of the organization overall?
A more detailed list of questions, also based on the Burke-Litwin model, is presented in
Appendix H. Many of the questions presented here and in Appendix H may be addressed in a
self-evaluation process by a management that is interested in continuous improvement. If the
purpose of the review is internal--that is, requested by management to assist it in an assessment
process, it can be handled by self-assessment or by invited external peers. However, for an
externally mandated evaluation of all aspects of the organization, some questions have to be
investigated by external reviewers.
Some of the questions listed here and in Appendix H need to allow for anonymous input
from staff. This can be done in several ways, including the use of "skip-level" meetings, which
are meetings between external reviewers and non-management staff in the absence of managers,
during which anonymity is assured to the staff. These meetings can be valuable for identifying
potential problems that may not have come to the attention of management. Some assessments
organized by the National Research Council have used skip-level meetings, and it has been found
that they are most successful when a few external reviewers meet with relatively small groups of
employees at similar points in their careers (a small number of staff, so that there are
opportunities for most to contribute) and at varying levels of leadership. It is important to
communicate to the staff involved that any issues identified at such a meeting might be discussed
with management, but that no individual attendee will be identified. It is helpful for the
reviewers to have suitable open-ended questions to ask the group. For example: What aspects of
the job make you eager to come in each day--and what aspects of the job make you dread
coming in? It is important that the reviewers present attempt to prevent staff input from being
monopolized by a single individual, and if one individual seems to focus on a small set of issues,
it is important to try to determine if complaints are widely shared. It is also important that
reviewers avoid leading questions that are designed to elicit specific responses. Information
derived from these meetings is best used for the development of hypotheses for further
investigation rather than as data from which conclusions can be immediately drawn, given that
there are issues associated with sample size and selection.
Other means of garnering input from staff include the following: a parallel organization
exercise (a temporary arrangement whereby formal authority is suspended and people can speak
openly without fear of retribution), the purpose being to address issues not adequately addressed
within the formal structure; and an informal relations survey (an organizational-network analysis
consisting of a computer-generated map of the informal organization, identifying individuals to
whom others reach out regardless of position or rank).
Planning for the Future
It is important that management continually plan for the future so that the organization is
always in a position to fulfill its mission. This means that the right resources and leadership will
be in place when needed. The demographics of the workforce will be tracked so that there will
be leaders in place as retirements and departures occur, and people with new skill sets will be
recruited and trained so that the staff is ready to deal with new technologies. An effective
assessment considers the adequacy of this planning as well as plans for the necessary physical
infrastructure to support the new skill sets and technologies.
27
OCR for page 28
SUMMARY OF FINDINGS
The key components of the management of an organization considered in an effective
assessment are the R&D portfolio, resources, the organizational leadership and management
structure, and planning for the future.
Research is characterized by the fact that the result of any particular approach is not
known until the research has been performed. Hence it is important that a research program--
even one with a specific mission--examine several alternatives in a portfolio of projects.
Similarly, in development it is usually not known which technology will lead to the best (most
cost-effective, efficient) solution, and so it is important to consider a portfolio of technologies.
For an effective assessment, resources will be evaluated to determine if they are adequate for the
proposed or ongoing program, but the evaluation will also recognize the external resource
limitations (budgetary, regulatory, personnel). To be effective, research leadership and structure
will be adapted to the environment in which it operates--no one model suffices for all styles of
research organizations. The future always brings improved science and technology, and so if the
parent organization is to remain competitive, it is important that its R&D units continuously
upgrade the skill sets of its workforce and match these with adequate research facilities.
28
