organizations, such as NIST, there are too many customers for individual contacts. NIST’s clientele includes the scientific and engineering communities; a variety of federal, state, and local organizations; and other countries. NIST of necessity deals through intermediary organizations such as trade groups, scientific and engineering societies, and aggregations of state and local government interests. It is important that an assessment of an organization include the efficacy of such interactions, using techniques such as polling and face-to-face meetings.
Any effective assessment is done in the context of the organization’s mission. In order to satisfy its mission, an organization needs to be prepared to handle its current and future workload. This means that it will have a successful combination of the following:
• R&D portfolio—a collection of projects that are most likely to lead to successful accomplishment of the organization’s mission;
• Resources—a workforce with an appropriate skill balance; the needed physical plant and equipment; and sufficient funding to enable accomplishment of the mission;
• Organizational leadership and management structure appropriate to the mission; and
• Planning for the future—the preparedness needed to ensure that the required resources will be in place as the mission evolves.
These elements are properly considered in context. Academic research focuses on generating new knowledge with relatively few mission objectives, whereas government and industrial research organizations have fairly clearly defined missions. Any effective management assessment also recognizes externally imposed limitations, including but not limited to regulatory and budgetary restrictions.
At all stages of R&D, it is important that the institution construct and manage its portfolio to maximize the probability of success. In basic research, it is important that the portfolio cover those areas that are likely to be important in the long term to achieving the mission, and that the assessors look at the portfolio and comment on whether there are areas that may be missing and whether there are areas that may be covered but not be relevant. In product development the areas are often well specified, but it is important to consider whether or not the correct set of technologies is being applied to achieving the desired results. As noted above in the discussion on the context of the evaluation, portfolios can be assessed during the planning phase, ongoing research phase, and retrospectively.
There are three elements to consider when assessing the quality of the research portfolio: (1) current projects and their relevance to the mission; (2) anticipation of opportunities; and (3) alignment of the planned future portfolio to mission, opportunities, and budget. Every R&D organization has some systematic way of listing its investments. At the extreme of offering little specificity is identification of funding per group, with descriptions of the group responsibilities and recent accomplishments. This format is most common at the more basic or fundamental end of the R&D spectrum. At the more applied extreme are examples in some industrial