Appendix C
Excerpts from the Phase-One Report
This appendix contains key excerpts from the committee's phase-one report, World-Class Research and Development, Characteristics for an Army Research, Development, and Engineering Organization. The excerpts consist of the Executive Summary and the tables (Tables C-1 through C-5), which provide the 25 characteristics and 100 metrics of world-class performance.
Executive Summary
The U.S. Army intends to conduct "world-class" research and pursue advances in technology to maintain superiority in land warfare. This report, which was prepared for an Army sponsor, defines the characteristics of a world-class research, development, and engineering (RD&E) organization.
Background and Approach
The Natick Research, Development and Engineering Center (RDEC) is a major element of the U.S. Army Soldier Systems Command, which, in ram, is a major subordinate command of the U.S. Army Materiel Command. The Natick RDEC vision is to be a world-class RD&E team that provides global customers with the essentials of life. The RDEC is organized into the following five directorates, which reflect the essential elements of its mission: (1) survivability, (2) sustainability, (3) mobility, (4) science and technology, and (5) advanced systems concepts.
The technical director of the Natick RDEC requested the assistance of the National Research Council in shaping the RDEC's future role and direction. A committee of the National Research Council (known as the Natick Standing Committee) was asked to (1) define world-class research, development, and engineering; (2) identify the major components of world-class research, development, and engineering; and (3) identify measurable qualitative and quantitative characteristics (and associated metrics) that must be met in order for an Army RDEC to declare itself world-class. The characteristics and metrics will be used later by the committee for an assessment of the Natick RDEC.
The committee recognized that the phrase world-class is widely used and has different meanings to different people. Although providing a general definition of world-class is relatively easy, defining the term for a research and development organization, particularly an Army RDEC, is more difficult. The committee attempted to develop a definition that takes into account the Army's mission and the RDEC's role in fulfilling that mission. Although an Army RD&E organization has unique features (e.g., it exists principally to serve the Army) that
distinguish it from academic and industry research and development centers, the committee found that research and development centers that are considered world-class share similar, measurable characteristics.
World-Class Research and Development
To define world-class research and development (R&D), the committee drew on material from general discussions with representatives of industry, academia, and government. The committee also examined relevant literature and four examples of widely respected organizations (i.e., Motorola, Milliken & Company, Intel, and FedEx). The committee determined that a world-class R&D organization is one that is recognized by peers and competitors as among the best in the field on an international scale, at least in several key attributes.
The committee observed that world-class R&D organizations maintain performance by creating and sustaining certain critical competitive advantages (e.g., a strategic focus on unique competencies of the organization). These competitive advantages result from excellence in five key attributes, which are often called "pillars." The pillars are (1) customer focus, (2) resources and capabilities, (3) strategic vision, (4) value creation, and (5) quality focus. These pillars are founded, in ram, on a demonstrated commitment to achieving world-class performance.
The major components of a world-class R&D organization are, therefore, demonstrated commitment, the five pillars, and the competitive advantages. Of these, the committee believes that the base—a demonstrated commitment—is the most important. Without it, aspirations to achieve world-class performance will be doomed.
World-Class Army Research, Development, and Engineering
The uniqueness of an Army RD&E organization makes it difficult to find similar peer and competitive organizations on which to base performance comparisons. Therefore, a definition of a world-class Army RD&E organization must also recognize the unique aspects of an organization's vision, mission, and strategy. For example, the Natick RDEC mission, which flows from its vision of becoming a world-class organization, is to (1) maximize the soldier's survivability, sustainability, mobility, combat effectiveness, and quality of life through the research, development, and engineering of items such as rations, clothing, shelters, and airdrop systems; (2) provide the necessary research, development, and engineering to integrate several combat-essential elements (e.g., survivability, sustainability, and mobility) into the soldier system; and (3) perform
similar, related functions for other Department of Defense services and federal agencies (e.g., be the center of excellence for food science and technology). The strategy includes developing highly skilled personnel, acquiring quality equipment and facilities, and establishing consistent and stable funding. However, this strategy—and ultimately the vision—are necessarily influenced by the current environment, which includes shrinking budgets and levels of personnel.
Taking the factors listed above into account, the committee determined that a world-class Army RD&E organization is one that excels in several key attributes by matching core competencies to its mission, thereby fulfilling the needs of soldiers as well as, or better than, similar organizations anywhere in the world. To achieve and maintain world-class performance, an organization must identify and develop the necessary core competencies. For an Army RD&E organization, these include the ability to move quickly from developing to fielding new, applied technologies. The technological capability must encourage continued development of new, superior products.
The committee believes that the pillars of a world-class R&D organization provide the most convenient means of articulating the prominent aspects of world-class performance. The five pillars are also applicable to Army RD&E organizations. The pillars are described below.
- Customer focus is the ability to identify, anticipate, and respond to customer needs both now and in the future. The focus is on internal customers as well as external customers.
- Resources and capabilities are the assets and talents with which the organization creates value for the customer.
- Strategic vision is a mental view of the type of organization that senior-level management would like the enterprise to become. This vision must be communicated indelibly to all personnel and translated into key elements that will make the vision a reality.
- Value creation is the ability to produce or increase benefits perceived by customers so they feel they are getting more value than they expected or previously received.
- Quality focus is the ability to continue striving for higher quality. The commitment to quality often results in breakthroughs.
Characteristics and Metrics
The characteristics of a world-class RD&E organization are derived from the five pillars. The committee judged that 25 characteristics are most relevant to an Army RD&E organization. These characteristics are discussed below according to the pillar under which they fall.
Customer Focus Pillar
The characteristics of this pillar are customer satisfaction, customer involvement, and market diversification. Both types of customers (e.g., internal product development teams and soldiers external to the RD&E organization) can be surveyed to ascertain how satisfied they are with the technological solutions and products delivered. Customer involvement in setting program objectives and following progress can also be evaluated. Although an Army RDEC must focus on the primary markets it serves, the committee believes that some market diversification is proper for any RD&E organization. Indeed, in the private sector world-class RD&E organizations seek to exploit fully the results of their research and product development. The extent of market diversification by Army RDECs can be determined; but diversification must also be considered carefully because Army RDECs exist primarily to support their Army missions and rely on government funding, which is usually authorized only to satisfy specific needs. Satisfaction, involvement, and the nature of market diversification indicate how well an RDEC is connected with and focused on the long-term and short-term needs of customers.
Resources and Capabilities Pillar
The characteristics of this pillar are the quality of personnel; facilities and infrastructure; budget; RD&E capabilities, skills, and talents; use of external resources; important technologies; information technology; and organizational climate. Internal and external reviews (e.g., by management of the RD&E organization or by higher headquarters) can be conducted to assess the organization's resources and capabilities. These reviews may include analyses of the core technical programs, evaluations of employee morale and the research climate, and assessments of the ability to reach ''make versus buy'' decisions. The quality and quantity of the human, physical, and financial resources and core capabilities of the RD&E organization indicate the ability and power to achieve world-class results. A positive organizational climate usually correlates with high productivity.
Strategic Vision Pillar
The characteristics of this pillar are alignment of vision and mission, anticipatory strategic planning, stakeholder buy-in, and leadership. Internal and external (e.g., peer) reviews can determine if the strategic vision of the RD&E organization and the mission are aligned and whether anticipatory strategic planning is sufficient to develop future Army and joint service products rapidly. To assess stakeholder buy-in, the stakeholders must first be identified.
Assessments of strategic vision should include the organizational leadership to ensure that the organization's vision is understood by staff and stakeholders alike. The quality of the strategic vision will give a reading of the enduring capability of the organization to plan and achieve world-class results.
Value Creation Pillar
The characteristics of this pillar are a proper portfolio, product performance, cycle time and responsiveness, and the value of work in progress. Value creation is often a perception based on a comparison of previous products (or lack thereof) with current products. Reviews of the breadth of effort (i.e., the portfolio) and other characteristics are important for making a meaningful assessment. Reviews can be conducted using both internal and external evaluations. The extent to which the RD&E organization produces outstanding, meaningful results reflects the impact of the organization.
Quality Focus Pillar
The characteristics of this pillar are the capacity for breakthroughs, continuous improvement, commitment to quality, structured processes, a learning environment, and the quality of research. The capacity for scientific and engineering breakthroughs can be assessed, in part, by reviewing past performance (e.g., how many breakthroughs have already occurred). Continuous improvement and structured processes (i.e., the ability to work in a disciplined and organized fashion) can be assessed by reviewing processes and results. The commitment to quality must be assessed at all levels, from topmost management down. Reviews can determine the ability of the staff and the organization as a whole to learn and use knowledge to achieve outstanding results. Finally, research quality can be assessed by expert review. Measurements of all these characteristics can give an overall assessment of the focus on quality in an Army RD&E organization.
Measuring the Characteristics
Metrics can be developed to measure various aspects of input, processes, output, and outcomes in the past, present, and future. Using the wrong metrics may limit performance or lead to inappropriate results. For an RD&E organization, the metrics should foster improvement and be related to the vision and mission. With these factors in mind, the committee developed a set of metrics that can be used as part of an assessment of the Natick RDEC. Beyond measuring the extent to which the RDEC exhibits world-class performance, the metrics can
also be helpful for self-evaluation or for evaluations of other RD&E organizations by higher-level Army commands.
To describe adequately the many facets of RD&E performance, the committee chose metrics with qualitative descriptors for four levels of performance. These levels are poor, adequate, good, and excellent. The committee believes that a predominance of excellent performance is necessary for an organization to be deemed world-class. The committee also considered the concept of best-in-class, which is the level of performance beyond excellent. This level is not included in the metrics because descriptors would apply to unique situations.
The 100 metrics (i.e., 25 characteristics, with four metrics each) are tabulated in the body of the report according to the characteristics to which they belong. They are sorted by pillar (e.g., there are 12 metrics for the customer focus pillar, four for each of the three characteristics). Assessment results can be summarized in tables or figures, which include overall assessments for each pillar.
It should be noted that the committee has implicitly given equal weight to all five pillars. Under some circumstances, it may be appropriate to assign greater weight to one pillar or another. Other adaptations (e.g., for self-assessment) could also be considered.
Conclusions and Recommendations
Conclusion 1. The phrase "world-class" is widely used to describe products and services. This phrase, however, can reasonably mean different things to different people. Therefore, if the phrase "world-class" is to be useful as a vision, it must be defined, tailored, and characterized in detail.
Conclusion 2. A world-class R&D organization is one that is recognized by peers and competitors as among the best in the field on an international scale, at least in several key attributes.
Conclusion 3. A world-class Army RD&E organization is one that excels in several key attributes by matching core competencies to its mission,-thereby fulfilling the needs of soldiers as well as, or better than, similar organizations anywhere in the world.
Conclusion 4. Efforts to reach or maintain world-class performance require the demonstrated commitment of the full chain of command, from topmost management to the lowest level.
Conclusion 5. World-class R&D organizations are likely to excel in certain fundamental attributes, which are based on demonstrated commitment. These attributes, often called pillars, are customer focus, resources and capabilities, strategic focus, value creation, and quality focus.
Conclusion 6. The five pillars are the basis of 25 Characteristics that the committee believes are most relevant to an Army RD&E organization.
Conclusion 7. Metrics with qualitative descriptors for four levels of performance (i.e., poor, adequate, good, and excellent) of the 25 characteristics are the preferred means of determining the extent to which an RD&E organization has achieved world-class performance.
Conclusion 8. Good or excellent performance for each characteristic, and excellent overall performance for all five pillars, are believed to be necessary for an organization to be judged world-class.
Recommendation 1. The concepts, characteristics, and metrics developed in this study should be used to assist the committee to assess the Natick RDEC.
Recommendation 2. These concepts, characteristics, and metrics should be considered by the Army or outside reviewers for use in assessing other Army RD&E organizations.
Recommendation 3. Army RD&E organizations should consider using these concepts, characteristics, and metrics for self-evaluation.
Recommendation 4. The concepts developed in this study should be considered by RD&E organizations in general for making assessments and self-evaluations. Some tailoring of the characteristics and metrics will probably be needed to suit specific organizations, be they inside or outside the Department of Defense.
Recommendation 5. The concept of a world-class organization should be used principally as an internal focusing mechanism for achieving excellence rather than as an external mechanism for advertising the virtues of an organization.
TABLE C-1 Metrics of the Customer Focus Pillar
Characteristics |
Performance Level |
Metrics |
Customer Satisfaction |
Poor |
Less than satisfied or dissatisfied with
|
|
Adequate |
Satisfied with all of a-e (met expectations) |
|
Good |
Very satisfied with a-e (exceeded expectations) |
|
Excellent |
Delighted with a-e (beyond normal expectations) |
Customer Involvement |
Poor |
No consideration was given to involving either internal or external customers in program planning, evaluation, or early "results" (prototype) testing. |
|
Adequate |
Internal or external customers are at times consulted on various aspects of research programs or are involved primarily in program reviews. |
|
Good |
Internal or external customers are from time to time involved in setting program objectives and following progress; there are opportunities for customer feedback. |
|
Excellent |
Customers feel completely involved, almost like partners in the effort; they feel they can and do have a major impact in the life-cycle development of the product or service. |
Market Diversification |
Poor |
Although diversification is addressed in strategic and business plans, senior management has not effectively broadened the customer base; products are developed only for the Army; few joint service RD&E programs are in place. |
|
Adequate |
RD&E programs result in products for the Army and the other uniformed services; the organization provides products to other federal agencies; some of the budget is devoted to developing partnerships with industry and academia. |
|
Good |
The organization is assigned DoD lead on joint programs; a significant amount of the budget is devoted to partnerships with industry and academia; research partnerships yield products that fulfill military needs and fill a void in the needs of other federal agencies. |
|
Excellent |
As a center of excellence, the organization's products serve a wide range of customers, including DoD, other U.S. government organizations, and global allies of the United States; much technology is transferred between the organization and the private sector; industrial and academic partnerships result in the rapid transfer of cutting-edge technology between the organization and its partners; high-quality products are developed, manufactured, and distributed to global customers. |
Table C-2 Metrics of the Resources and Capabilities Pillar
Characteristics |
Performance Level |
Metrics |
Personnel Quality |
Poor |
Work is below standard throughout the organization; there are inadequate technical skills; program planning and management are poor. |
|
Adequate |
Work meets expectations; work force has adequate skills to get results in a timely manner; opportunities to improve and upgrade skills are minimal; few resources are programmed for improving technical skills. |
|
Good |
Work usually exceeds expectations; newly hired employees bring critical new skills and capabilities; present RD&E personnel devote at least a small percentage of their work time to upgrading or acquiring skills, and this training is reflected in annual performance appraisals; the Army gives special recognition to RD&E personnel; personnel are well connected with the scientific and technical community outside the organization. |
|
Excellent |
Work consistently exceeds expectations (of those with major interests in the work of the organization); new skills and capabilities are regularly introduced into the organization; newly hired personnel bring new, state-of-the-art methods into the organization; personnel are encouraged to devote a significant amount of their work week to improving and acquiring technical skills; personnel are recognized for their accomplishments by individuals and organizations outside the Army; career structures support the development of technologists in a wide range of needed disciplines; personnel are noted for effective use of both external and internal resources. |
Budget |
Poor |
Research and program support budgets are constrained; research programs are consistently underfunded; out-year budget projections are flat or decrease; mid-year budget cuts are routine; programs are abandoned, with resulting inefficiencies. |
|
Adequate |
Although budgets are at the recommended levels, major research programs are constantly in jeopardy because of uncertainties about year-to-year funding; no new major construction or programs are funded even though budgets finally prove to be adequate for maintaining ongoing programs. |
|
Good |
The organization consistently finds ways to get more done with less; resources are leveraged with other government agencies; the organization periodically takes the lead in DoD-wide or similar programs; collaborative programs with industry and academic groups are cultivated; some funding is provided to support new research initiatives, acquire new equipment, and construct or renovate laboratory facilities. |
|
Excellent |
The outstanding work of the organization is recognized by prompt funding at desired levels; the organization is asked to accelerate RD&E programs and initiate new missions when additional funding is available; program managers obtain the absolute best value with their budget; resources leveraged with other organizations and agencies are recognized as force multipliers; the organization maintains a backlog of high-quality yet-to-be-funded projects. |
Characteristics |
Performance Level |
Metrics |
RD&E Capabilities, Skills, Talents |
Poor |
Technical skills, capabilities, and talents are inadequate to support current and future customer requirements; few new techniques and skills are acquired via new hires or continuing education and retraining of personnel; personnel cannot fully operate, maintain, or utilize available equipment; continuing education is not promoted, encouraged, or funded. |
|
Adequate |
Plans are developed and funding is provided for maintaining the present core capabilities for the future; personnel are trained to operate and maintain equipment and use equipment as specified by the manufacturer; personnel skills are recognized as current and competent for their technical specialties. |
|
Good |
The organization possesses the skills and talents to fulfill customer requirements for the foreseeable future; new and innovative techniques, skills, and processes are incorporated into the RD&E processes; newly acquired skills result in improved product engineering, manufacturing, or performance; new personnel are recruited to bring state-of-the-art techniques into the organization; personnel are encouraged to participate in formal continuing education programs; members of the research staff are encouraged to participate in professional societies, serve on external committees, etc.; program managers recognize new skills that will benefit their programs, and they plan for the acquisition of these skills and talents. |
|
Excellent |
The research and support staffs are recognized as possessing superb technical and administrative skills and talents; many members of the support staff are recognized as artisans of their trade; research personnel incorporate state-of-the-art techniques into their research and develop pioneering methods of their own; a clearly articulated plan describes how needs and voids in core capabilities are identified and filled; new capabilities that must be developed are also addressed and acted upon; a growing inventory of skills is maintained. |
Use of External Resources |
Poor |
Work is contracted outside the organization on an ad hoe basis with little or no planning; contract managers do not ensure that statements of work are fulfilled on time or on budget; products and services provided by contractors and partners contribute incrementally to the organization's mission. |
|
Adequate |
Partnerships are developed with a wide range of groups to enable work to be done outside the organization; work done by others is contracted primarily based on the other party's willingness to do the work; products and services obtained from external sources fulfill the statement of work; products complement internal research programs. |
|
Good |
The organization is recognized as a "smart buyer" of services and work of other parties; personnel appreciate the quality of the work that is contracted; the extent of leverage (i.e., the ratio of the cost to do the work at the organization to the contracted cost) is appreciated; external research programs enhance internal programs and result in leap-ahead technology. |
|
Excellent |
Partnerships and contracts with organizations recognized as the best in their field complement RD&E programs and result in leap-ahead (and occasional breakthrough) technological advances; skills and abilities of the external organization cannot be duplicated in the organization in a cost-effective manner; the value of partnerships is widely recognized inside and outside the RD&E organization. |
Characteristics |
Performance Level |
Metrics |
Important Technologies |
Poor |
There are no systematic programs or processes for introducing, managing, or assessing research technologies in the research program. |
|
Adequate |
Base technologies being developed or used in the research program are necessary for fulfilling technological needs but offer little differentiation in product performance from other alternatives; important technologies are recognized, developed, and used, but technology development is not advanced. |
|
Good |
Pacing technologies are being developed or used in the research program; these technologies have the potential to change significantly the nature of the research program, but they are not yet embodied in products; incorporation of pacing technologies results in leap-ahead developments. |
|
Excellent |
RD&E programs are anticipatory; development and incorporation of new technology to support RD&E and product development are planned and adequately funded; new areas of research and technology are appreciated, and researchers understand the implications of particular research programs; new scientific discoveries are frequently translated into pacing technologies within the organization. |
Organizational Climate |
Poor |
The work environment is acknowledged by management and staff to be poor; personnel are preoccupied with furloughs, early retirement, and downsizing initiatives; personnel equate reengineering to organizational instability; initiating risky programs is discouraged; management punishes failure by withholding resources. |
|
Adequate |
The work environment is perceived as professional and collegial; personnel enjoy their work and say it is meaningful; responsibilities are clear, and teamwork and collaborative efforts are evident; managers tolerate innovation and occasionally empower their staffs, teams, and groups; personnel are recognized for their contributions; although anxious about reorganization and downsizing, individuals feel relatively secure about their jobs. |
|
Good |
Work and organizational climate is considered good; bold and innovative thinking is encouraged and rewarded; research personnel are fully empowered to set goals and pursue original and innovative solutions, but they do not fear failure; the organization is recognized as possessing a "can do" attitude. |
|
Excellent |
Management and staff perceive the organizational climate as excellent; there is clarity of purpose and vision; the staff is secure; no hint of fear is present, and rewards and recognition motivate individuals and teams to make excellent contributions; management encourages the development of new work environments that result in increased productivity. |
Characteristics |
Performance Level |
Metrics |
Information Technology |
Poor |
Computer hardware and software are not available at every work station; software and hardware are two generations or more out of date; personnel cannot communicate electronically or transfer data internally or externally; personnel are poorly trained and hesitant to learn new applications; funding for information technology and user training is inadequate. |
|
Adequate |
Information technology is used as a tool by research and support personnel, and it increases productivity and ultimately decreases the organization's overhead; acquisition of new hardware and software is adequately funded; training and technical support are available; personnel are comfortable with the available technology and are electronically connected internally and externally. |
|
Good |
An information technology strategy guides program direction; research, support, and administrative systems are integrated; information technology enhances the effectiveness of the RD&E allowing work to be done in entirely new ways; information technology is credited with recent advances in research programs; hardware and software are state-of-the-art; technical support is abundant; the staff is educated in the use and application of the technology. |
|
Excellent |
Information technology enables rethinking how RD&E is done, and technical breakthroughs, previously thought of as being impossible, are within reach; the products include information-technology components. |
Facilities and Infrastructure |
Poor |
Facilities and equipment are inadequate, poorly maintained, and out-of-date; no new investments in equipment and facilities are forecast;. preventive maintenance is seldom performed; safety and regulatory compliance are rarely addressed. |
|
Adequate |
Facilities are judged adequate to meet the needs of the organization; there is a schedule for periodic maintenance and upgrading of equipment; safety and regulatory compliance policies are in place, but audits, inspections, and training are limited. |
|
Good |
Research and support facilities are clean, spacious, and comfortable; facilities are environmentally controlled year round; equipment is upgraded or replaced routinely; preventive maintenance and service contracts are well funded; relatively new technical capabilities are acquired, and user training is provided; there is evidence that safety and regulatory compliance are important (e.g., statistics are maintained, periodic inspections are made versus appropriate standards, and training is emphasized). |
|
Excellent |
Facilities and equipment are exceptional; there is timely access to equipment and facilities, which aid personnel in many unexpected ways (e.g., the latest technologies allow them to look at problems in new ways; specialized analytical equipment opens new horizons; there is sufficient equipment to meet user needs); critical programs are supported with state-of-the-art equipment; there is pride in the installation's records in safety and regulatory compliance, ample resources are devoted to inspections and training, and employees continually strive for better safety and regulatory compliance. |
Table C-3 Metrics of the Strategic Vision Pillar
Characteristics |
Performance Level |
Metrics |
Alignment of Vision and Mission |
Poor |
Vision and mission statements are not articulated well, nor are they linked; senior management has difficulty communicating vision and mission statements through command briefings, annual plans, or business plans to staff members and customers. |
|
Adequate |
Vision and mission statements are articulated and understood by most employees; a research strategy is developed using these statements as a guide; research programs, resources, and management support are aligned, in general, with the research strategy. |
|
Good |
The organization's strategic vision is inspiring, and the vision and mission are in harmony with each other; the vision and mission provide a "guide to action" for all programs; management support and resources are aligned with the research strategy. |
|
Excellent |
The strategic vision and management's translation of this vision into a research strategy yields superior products and services; the alignment of resources with the research strategy is readily apparent. |
Anticipatory Strategic Planning |
Poor |
No strategic planning process is implemented, or the strategic plan is ineffective. |
|
Adequate |
A strategic planning process is in place, and business plans and annual plans are implemented; senior management enlists research and support staff assistance to draft and implement the strategic plan through the business and annual plans; the plans are communicated to the staff. |
|
Good |
A robust planning process is in place, with broad involvement across the organization; the resulting planning document is used to measure progress throughout the year; contingency or alternative plans are developed to accommodate rapid changes in customer needs, the environment, or resources. |
|
Excellent |
Plans for human resources, information technology facilities, budget, and travel are fully integrated into strategic plans; the planning horizon for the strategic plans is sufficient to anticipate major Army and joint service needs; multiple examples demonstrate a high degree of flexibility within the organization, which has reacted rapidly to either major opportunities or critical customer needs. |
Characteristics |
Performance Level |
Metrics |
Stakeholder Buy-In |
Poor |
The strategic vision and research plan either have not been communicated to the RD&E stakeholders or have not been articulated well and are misunderstood; stakeholder response to the vision and research plan is either negative or indifferent. |
|
Adequate |
A strategic vision is spelled out and understood by most stakeholders; the vision makes all major initiatives readily understandable. |
|
Good |
The strategic vision ''speaks'' to all stakeholders even if they have not been involved in creating it; customers and disinterested parties understand the research plan and advocate providing adequate resources to implement the plan. |
|
Excellent |
The strategic vision is so clearly articulated that stakeholders lobby Army and DoD planners to implement the research plan fully; stakeholder support for the organization's vision and the research plan are so strong that resources are reprogrammed from other accounts to implement the vision. |
Leadership |
Poor |
Commitment of the senior leadership to the strategic vision or research plan is poorly communicated to the staff; administrative and product development managers are not involved in planning the direction of future research or developing the business plan; personnel are suspicious or do not trust the organization's leadership; stakeholders view the senior leadership as ineffectual and reactive. |
|
Adequate |
The strategic vision and research plan are understood by the staff; resources (i.e., time, personnel, and dollars) are aligned to meet these plans; the staff trusts senior leadership and is receptive to new ideas and re-engineering opportunities. |
|
Good |
Management and staff co-develop plans that are understood and embraced by staff and stakeholders alike; ideas flow freely and in both directions between management and staff. |
|
Excellent |
The leadership has created an air of excitement and commitment throughout the entire laboratory; bold and creative ideas are encouraged and funded; RD&E successes are rapidly exploited, and ideas are rewarded; failure is considered an opportunity to learn. |
Table C-4 Metrics of the Value Creation Pillar
Characteristics |
Performance Level |
Metrics |
Proper Portfolio |
Poor |
Products are developed that do not meet customer needs; products have poor customer acceptance; customers perceive that commercial alternatives are cheaper, perform better, and are more durable. |
|
Adequate |
An analytical process to examine the product portfolio is used to design and field products that have greater value and soldier acceptance; results of the analytical process lead to modifications in product design; major changes may be made after fielding the initial product. |
|
Good |
Portfolio analyses of a program are an integral part of the strategic planning process; there is broad and active customer involvement in the portfolio analysis; programs yield products that have significant customer acceptance, meet or exceed customer requirements, and demonstrate increased value compared to current products or commercial alternatives; minor changes in product design occur after initial fielding. |
|
Excellent |
Portfolio analyses result in RD&E processes that yield products and services with excellent value, performance, and customer acceptance. |
Product Performance |
Poor |
Products do not meet customer requirements (e.g., in terms of weight, volume, function, durability, or maintainability); customers complain that product performance does not meet the developer's claims; products are not suitable for use in certain locations or environmental extremes. |
|
Adequate |
Products meet customer requirements, needs, and expectations. |
|
Good |
Products fully meet or exceed customer requirements; products are perceived as better than the ones they replace. |
|
Excellent |
Products not only exceed customer expectations, but product performance includes some pleasant, unexpected surprises (e.g., reduced maintenance requirements, longer shelf life, longer mean time to failure, resource savings). |
Characteristics |
Performance Level |
Metrics |
Cycle Time and Responsiveness |
Poor |
Cycle time for project completion is longer than anticipated; milestones are routinely missed; program delays result in increased end-item cost; research programs do not anticipate customer needs; management and staff are not flexible to modifications of product requirements. |
|
Adequate |
Elapsed time from project initiation to project completion is measured and can be reliably forecasted; research programs are described as being on-time and on-budget. |
|
Good |
RD&E programs are initiated and completed significantly faster than similar government or commercial programs; research staff is responsive to "quick fixes" for troops, and numerous examples are readily available for major products; senior management ensures that adequate resources are reprogrammed to fulfill quick-fix requests. |
|
Excellent |
RD&E programs are initiated and completed substantially (e.g., one third) quicker than similar government or commercial programs; innovative processes and technical solutions reduce typical quick-fix response times by nearly half; the staff monitors foreign and domestic industrial and academic research for solutions to new and unanticipated technical problems; commanders directly and indirectly express gratitude for responsive quick fixes. |
Value of Work in Progress |
Poor |
No evaluations of historical RD&E programs are available for comparison to current programs; no methodology is in place to assess current RD&E programs; customer perception of prior RD&E programs is predominantly critical and negative, and little or no value is placed upon the current programs by the customers. |
|
Adequate |
A database on select historical RD&E programs and all current programs is available; current RD&E programs are vividly described, and these descriptions are used during peer-review discussions to justify programs and prioritize personnel and budget requests; customer perception of prior RD&E programs is generally positive; customer perception of current RD&E programs is positive (i.e., the products and services will generally meet user requirements and be delivered on time and on budget). |
|
Good |
A database is maintained on all past major projects (e.g., for the last decade) and their primary and secondary impacts; the database is used for comparison with current RD&E programs; leadership creates a scale to rate continuously the potential value of current programs compared with previous programs and show improvements; customers rate RD&E programs as very good (i.e., products are expected to fully meet or exceed customer requirements; products are perceived as likely to be better than the ones they replace). |
|
Excellent |
A complete historical database and evaluation methodology are used to demonstrate the value of the organization's products and services; data are used to justify and defend program expenditures; customers rate products and services as excellent (e.g., product performance exceeds customer expectations); product performance exceeds anything projected to be available from domestic and foreign sources for at least several years. |
Table C-5 Metrics of the Quality Focus Pillar
Characteristics |
Performance Level |
Metrics |
Capacity for Breakthroughs |
Poor |
RD&E programs are routine and unimaginative; there is no evidence of imaginative or innovative solutions being applied to RD&E tasks; resources are directed to meeting specific customer requirements only. |
|
Adequate |
RD&E programs are characterized by steady but incremental improvement; several innovative solutions can be pointed out; minimal funding is available for programs that anticipate future military requirements. |
|
Good |
Although most programs are characterized by incremental improvements in technology, the organization has demonstrated several leap-ahead improvements; the organization encourages and funds opportunities to seek truly innovative, moderate-risk solutions. |
|
Excellent |
Unexpected innovation based on breakthroughs in technology occur fairly regularly among internal and external (cooperative) RD&E programs; moderate- and high-risk research that offers high return receives stable funding; numerous examples of breakthrough research are cited from the previous five to ten years. |
Continuous Improvement |
Poor |
There is no tangible evidence of senior management commitment to continuous improvement; the need and ability to focus on continuous improvement are recognized, but not funded; products and services show incremental changes; innovations are not rewarded; solutions from industry and academia are discounted as "not invented here." |
|
Adequate |
Quality of the work is discussed and several measures of quality are used routinely; innovative solutions are encouraged; staff members frequently make suggestions for improvement; several changes are made (and documented) each month for improving the work and the output of the organization. |
|
Good |
The organization takes steps to improve work processes and RD&E results significantly; quality audits are performed periodically by internal and external review groups; numerous improvements can be pointed out; productivity is an important topic of discussion; report cards are issued annually by senior leadership; senior managers have the resources to enact recommendations. |
|
Excellent |
Greater productivity, enhanced research and product quality, improved customer involvement and satisfaction, and continuing education of the work force are areas of primary interest to senior management; the concepts of continuous improvement and excellent product value are embedded in the goals of each RD&E and support function; there is a systematic analysis of research and support processes to eliminate non-value-added activities; research personnel are renowned for finding innovative solutions to technically difficult problems. |
Characteristics |
Performance Level |
Metrics |
Commitment to Quality |
Poor |
Management espouses a commitment to quality, but no formal process to review and evaluate quality is in place; some quality-related results are managed by exception; the quality of products and services varies between RD&E units in the organization. |
|
Adequate |
Management is investing resources for total quality training and implementation; the variability of products and services is being measured and tracked; personnel are aware of the importance of quality. |
|
Good |
Total quality implementation is a major goal in the organization's strategic plans; a framework and methodology for measuring and assessing total quality is in place; measurable objectives for work-process improvement are established; there are methods (e.g., statistical process controls) to improve effectiveness and product quality with existing resources. |
|
Excellent |
The commitment to total quality is inherent and pervasive throughout the organization; the focus of all measurements is on optimizing the RD&E processes to deliver value; frameworks, such as ISO 9000/2 (international quality standards), Baldrige criteria, or locally developed systems, are used for assessment; recommendations to improve quality are immediately funded and implemented. |
Structured Processes |
Poor |
Work processes and procedures are understood and milestones are established, but there is no system of internal or external review; project management results in products or services that are delivered late and over budget; delays result in termination of projects; disciplined approaches to defining problems and the scientific method are rarely used. |
|
Adequate |
Work processes and procedures are monitored; project costs and milestones are closely tracked; processes are established to improve quality incrementally, contain or reduce RD&E cost, and reduce product cycle time; disciplined approaches and the scientific method are used most of the time. |
|
Good |
Program managers are flexible and adaptive; senior leadership and staff are receptive to innovative ideas for improving work processes and procedures; product quality and customer focus mean continuous improvement; disciplined approaches and the scientific method are used consistently. |
|
Excellent |
The senior leadership strives to identify and incorporate best business practices into the organization; processes are considered flexible and not overly restrictive, prescriptive, or bureaucratic; management is focused on achieving superior performance and product quality; emphasis on cross-project management ensures timeliness and the proper allocation of resources; disciplined approaches to problem solving include an extensive network linked to Army technological resources worldwide; the scientific method is strictly followed. |
Characteristics |
Performance Level |
Metrics |
Learning Environment |
Poor |
Senior leadership is characterized as reactive; little if any learning takes place on an organizational basis; some managers and staff learn from mistakes. |
|
Adequate |
Senior leadership recognizes and communicates the importance of organizational learning; management and staff learn from mistakes and from others; personnel are well networked both inside and outside the organization; teams on one project teach teams assigned to other projects; new skills and techniques are acquired through new hires and continuing professional education. |
|
Good |
Organizational learning is characterized as adaptive; the organizational climate is conducive to learning; personnel are rewarded and encouraged for taking risks and entrepreneurial initiatives despite occasional mistakes; personnel learn from others and by doing; management experiments with new organizational concepts to discover new ways of doing things. |
|
Excellent |
Organizational learning is adaptive and anticipatory; research and technical capabilities continually expand, and management anticipates change; traditional and innovative methodologies are used to measure and evaluate organizational learning. |
Quality of Research |
Poor |
Research and technology programs are not generally aligned with customer requirements and needs; records of research methodology and results are poor; although recorded in technical reports, data are not published in peer-reviewed journals or cited by other scientists in academia or industry; research results cannot be replicated by scientists and engineers outside the organization. |
|
Adequate |
Research and technology programs are aligned with customer requirements and needs; research methodology and results are peer-reviewed and published as both technical reports and journal articles; the research staff is invited to participate in scientific meetings and workshops; research results are easily replicated by other laboratories. |
|
Good |
The research and technology programs are recognized by peers as being of very high caliber; several programs are among the best in the federal government and are described as innovative and original; some patents are awarded. |
|
Excellent |
The quality of the research and technology programs is considered to be among the best in the world; basic research not only fulfills customer needs, but also anticipates future requirements, thus reducing cycle time for new products; research and technology programs are innovative and state-of-the-art; new procedures, processes, and materials are developed by personnel; numerous patents are issued for RD&E innovations. |