Assessment of the U.S. Army Natick Research, Development, and Engineering Center (1997)

This chapter presents the committee's assessment of the three commodity directorates at the Natick RDEC (i.e., the Mobility, Survivability, and Sustain-ability Directorates).¹ The assessment was conducted relative to the five pillars and the associated characteristics and metrics for world-class performance by an Army RD&E organization (refer to Chapter 1 and Appendix C for summaries of the pillars, characteristics, and metrics). Following a discussion of the assessment process, this chapter is organized by pillars and their associated characteristics. Each directorate is discussed in terms of each characteristic. A summary is presented at the end of each pillar, along with an indication of trend (vector).

The committee sent a questionnaire to MobD (with related questions for STD and ASCD) in April 1996. The questions were formulated in light of (a) the five pillars and associated characteristics and metrics for a world-class Army RD&E organization from the committee's phase-one report, and (b) the ten key issues shown in Table 1-1. (See Chapter 1, footnote 3, regarding questions and answers.) The answers were reviewed by the committee in preparation for a visit to MobD, which took place on June 3 to 5, 1996. More than half of the personnel in MobD, including the director and his staff in the two management offices, were interviewed. Staff members of all work teams that composed the main elements of the MobD organization were also interviewed.

The visiting members of the committee split into three interview groups, organization and resources, quality and customer satisfaction, and research and technology. During the on-site visit, these groups covered both questions prepared in advance and questions that arose naturally during the interviews. At times, the interviewers presented the metrics for a particular characteristic and asked the

interviewees to indicate which one best fit MobD. These self-assessments prompted helpful discussions by MobD personnel. Following the on-site visit, the committee synthesized the written answers provided by RDEC management and the results of the on-site interviews to arrive at the judgments reflected in this assessment.

On February 10 to 13, 1997, the committee revisited the Natick RDEC, at which time a brief follow-up assessment of MobD was conducted for the following reasons. First, at the time of the initial visit, the committee's phase-one report had not yet been published and, consequently, the metrics and processes had not been finalized. Second, MobD was the first directorate to be assessed, and the initial visit was partly used to evaluate and refine the interview process. Third, the committee wanted to be updated on major changes that might have occurred since the earlier visit.

On the second visit (February 1997), the committee was able to interview the MobD director, the associate director, the head of the business management office, and a senior scientist. Although the interviews did not produce a thorough reassessment of MobD, they yielded much useful information. The results of the second visit were not used to alter the earlier assessments but to indicate trends since the earlier visit.

Assessment of SurD proceeded in a manner very similar to the process described above. Specifically, the assessment process included formal written questions submitted in June 1996 that were answered before the visit. The site visit took place on September 25 to 27, 1996, and included laboratory tours and interviews with SurD personnel. The director, members of the business management office, and some management and staff of the main elements within the SurD organization were interviewed.

The committee was given short guided tours of the various laboratories in SurD, as well as of the STD laboratories that support SurD. The interviews were again conducted by three groups (i.e., organization and resources, quality and customer satisfaction, and research and technology) in parallel sessions. Following the on-site visit, the committee synthesized the written answers provided by the RDEC management and the results of the on-site interviews.

Assessment of SusD followed the pattern described above. The committee sent a set of questions in September 1996 and reviewed the answers prior to the site visit, which took place on December 3 and 4, 1996. Tours of SusD

laboratories were included. The director and his staff were interviewed, as well as management and staff members of each main element of the SusD organization. After the on-site visit, the committee synthesized the information.

This section is organized into subsections presenting the assessment of each commodity directorate for each of the eight characteristics of the Resources and Capabilities Pillar (see accompanying box). An assessment for the entire pillar appears at the end of the section.

The committee determined that MobD staff members are motivated and committed. MobD personnel were able to highlight several programs where their work had exceeded customer expectations (e.g., the guided parafoil aerial delivery system [light]). MobD offers ample opportunities for personnel to upgrade or acquire new skills, and personnel have good connections with external technical and scientific communities (e.g., the American Institute of Aeronautics and Astronautics, National Parachute Technology Council, and Parachute Industry Association). However, the MobD workforce is aging, and employees with new skills and capabilities are not being brought in. This has serious implications for the future.

Assessment: Good

The committee determined that SurD has an abundance of highly qualified, highly motivated people. Personnel are encouraged to upgrade and acquire new skills and to make effective use of internal and external resources. Some are attending universities full time and working toward advanced degrees. However, many experienced people have been lost through downsizing and cannot be replaced because of the hiring freeze.

Assessment: Good

The committee determined that the SusD workforce is highly experienced and, for the most part, has a very strong work ethic as reflected in their commitment to their respective areas of expertise. Overall, SusD has expertise in a wide range of disciplines, and, because of the organization's willingness to support training and education, there are continual opportunities to upgrade and maintain expertise. SusD personnel were able to highlight several projects where they had exceeded customer expectations (e.g., shelf-stable bread products). However, the workforce is aging, and people who leave or retire are not being replaced by employees with new skills and capabilities. To some extent, SusD has been able to compensate for the lack of new personnel by aggressively promoting training and education for its workforce.

Assessment: Good

The committee determined that the metrics associated with a good performance level best fit the overall personnel quality of the commodity directorates. Personnel within the directorates could cite examples of exceeding customer expectations, although the committee notes that there appears to be little reward in the Army system for consistently exceeding customer expectations. The RD&E personnel have many opportunities to upgrade their skills or acquire new ones. The Army has recognized the capabilities of its personnel through various RD&E awards, and the personnel are well connected to the scientific and technical communities outside Natick. The committee notes that the Natick RDEC has not been

able to hire employees to provide (or replace) critical skills and capabilities, which limits the assessment of the performance level to good.²

Assessment: Good

Budget cuts over the past several years have greatly reduced airdrop R&D programs. The ground mobility budget has been reduced even more severely, re-suiting in a drop from four full-time-person equivalents to about one in 1996. Projections of future budgets for airdrop R&D programs appear to maintain this level. Funding from nontraditional customers (e.g., the Special Operations Command and the U.S. Marine Corps) have eased the pain of budget cuts, but these sources of funding may not be available in the future.

Assessment: Adequate

Overall funding for SurD has increased in the last three years, mostly from the Force 21 Land Warrior Program and customer orders, mainly from the U.S. Marine Corps. Leveraging has significantly added to SurD's capabilities. Examples of leveraging are programs with the Army Research Laboratory, the Bureau of Printing and Engraving, and industry and academic partners.

Assessment: Good

Financial allocations for SusD have remained almost flat, although funds for services from outside organizations have increased the budget slightly. Even with outside funds, however, SusD has not been able to increase its areas of expertise, and the organization cannot keep up with inflation. At best, the present

budget will allow SusD to maintain a slowly eroding position in terms of performance. Some divisions have adequate funding for projects but do not have enough personnel to perform the work. A high percentage of the SusD budget goes to requirements-driven activities rather than to discretionary science and technology programs.

Assessment: Adequate

The committee determined that the metrics associated with an adequate performance level best fit the assessment for the overall budget of the commodity directorates. Budget cuts over the past few years have jeopardized or eliminated some major programs. Natick has worked hard to find ways to do more with less, especially by leveraging resources with other government agencies and the private sector. This positive factor is offset by indications that the RDEC's longer range budget projections are either flat or decreasing. Uncertainties about year-to-year funding contribute to the adequate performance rating of the commodity directorates.

Assessment: Adequate

The capabilities of the individuals in MobD are extremely good. Some members of the MobD staff are recognized as masters of their trade (e.g., two individuals were recognized by the Army for their achievements in R&D, and one was recognized by the American Institute of Aeronautics and Astronautics for a lifetime of achievement in parachute development). Many MobD personnel are active in professional societies, such as the American Institute of Aeronautics and Astronautics and the Parachute Industry Association. However, limited budgets and hiring freezes have jeopardized the RD&E skills of MobD. In many areas, the directorate has only one expert, and his or her expertise will be lost upon retirement (e.g., parachute materials). In the past three years, MobD has lost 15 individuals through retirement, attrition, and transfers to other positions in the RDEC or Soldier Systems Command. None of them has been replaced.

Assessment: Good

SurD is a recognized leader in several core capabilities, such as textile technology, chemical and ballistic protective apparel, and countersurveillance. However, continued downsizing coupled with a hiring freeze have significantly reduced the number of highly skilled personnel. Reductions have been partially offset by bringing in technical interns, using outside contractors, and leveraging via partnerships with industry and academia. Interviews and the committee's experience indicate that one disadvantage of using external resources to help maintain RD&E capabilities is that it shifts the workload of current employees from their own research and technology to contract management. Over time, this tendency increases the bureaucratic nature of the organization and dilutes its technical expertise.

Assessment: Good

The capabilities of the individuals in SusD are extremely good. In addition, most of the people with whom the committee spoke said that they were receiving up to 80 hours of training and development per year (although in some cases they questioned whether the training was particularly relevant to them, e.g., people who were assigned to take a course on U.S. Department of Defense [DOD] acquisition reform). The biggest problem is that the organization is very lean, and there is no one to fill in for anyone who transfers, quits, or retires. Over time, the core capabilities of the enterprise will slowly erode unless the hiring freeze is lifted.

Assessment: Good

The committee determined that the metrics associated with a good performance level best fit the overall RD&E capabilities, skills, and talents of the commodity directorates. In several areas, the members of the research and support staff have been recognized outside Natick for their superb technical skills, and some members of the RDEC staff are recognized masters of their trade. These would indicate an excellent performance rating, but new techniques and skills are not being acquired via new hires, although the Natick RDEC can perhaps maintain its present core capabilities through internal training programs. Personnel are

encouraged to participate in formal continuing education programs and professional societies and to serve on external committees, which are indicators of a good performance level. The committee weighed all of the considerations in this category and determined that the overall performance level was good.

Assessment: Good

MobD has formed many liaisons with external organizations to supplement in-house capabilities. Most of these liaisons involve modeling, primarily with the Army Research Laboratory and the Army Research Office or projects they support. The exception is a cooperative R&D agreement with an outside company to investigate gas-injected air bag technologies. A significant portion of the external work is conducted with the High Performance Computing Research Center at the University of Minnesota, which is supported by the Army Research Laboratory and comes at no cost to MobD. The range of external activities is limited, but it does significantly leverage the financial resources available to MobD. (Even if MobD had additional financial resources, it might choose to work with the same outside groups.) These external activities complement internal efforts to fill in gaps in personnel, capabilities, funding, facilities, and time. However, with the exception of the air bag agreement, none of the current external projects is likely to lead to a major breakthrough in airdrop systems in the foreseeable future.

Assessment: Adequate to Good

SurD has developed a wide range of partnerships so work can be done outside the organization. Examples include (1) a technology program annex with the Army Research Laboratory in science and technology (at no cost to the Natick RDEC); (2) a cooperative R&D agreement on high performance fibers (at no cost to the Natick. RDEC); (3) fiber project contract work with the Bureau of Printing and Engraving; (4) a working group with the textile industry to develop performance specifications; (5) a working arrangement with the Northeastern University Center for Electromagnetic Research; (6) serving as the Natick Textile Technology Center of Excellence with Drexel University, Temple University, the Philadelphia College of Textiles and Science, the University of Massachusetts Center for Industry Research on Polymers, and the National Textile Center; and (7) task

order contracts with small businesses to convert detailed specifications to performance specifications. These partnerships and outside contracts account for about 60 percent of SurD's budget. The amount of time individuals spend on outside contracting varies from 25 to 90 percent.

SurD selects outside partnerships and contractors based on the skills and capabilities of the outside organization and on SurD's assessment of the organization's ability to complete the work. SurD uses measures such as best-value contracting, expected costs, and past performance to evaluate potential partners. In general, SurD considers that the results of these liaisons have been valuable, especially with external sources that provide services at little or no cost to SurD (obviously representing an infinite return on investment). The least valuable have been the task order contracts to convert detailed specifications to performance specifications. In hindsight, these contractors were judged not to have the necessary training to perform the job well. As a consequence, these projects occupied an inordinate amount of the project officer's time and expertise. The last example raises questions about the criteria used by SurD to select outside contractors. Nevertheless, using external resources has led to breakthroughs in the development of materials for combat clothing and individual equipment at little or no cost to the RDEC.

Assessment: Good

SusD does business with a wide range of outside contractors, including business firms, academic institutions, and other government agencies. Sixty percent of the SusD budget is spent on external resources, including joint programs, modeling, and testing. External clients are selectively chosen and provided with carefully developed specifications to ensure high performance. Moreover, the SusD carefully monitors the work of these external organizations. SusD has learned to focus on the outcomes of relationships with contractors and collaborators rather than on the number of relationships.

Assessment: Good to Excellent

The committee determined that the metrics associated with a good performance level best fit the overall use of external resources by the commodity directorates. In general, the Natick RDEC has made good use of external resources. All of the commodity directorates understand the need to be smart buyers of services from other parties, and all have had successful programs with

businesses, universities, and other federal agencies. Some external programs have provided leap-ahead technologies. (The committee notes that recognizing potential breakthroughs by external groups requires residual expertise within the RDEC.) However, not all external programs can be categorized as successes. Sometimes using external resources has been the only option in the face of hiring freezes within Natick, and valuable internal resources have been tied up with managing contracts. The committee's concern with the value of some partnerships limits the assessment to good.

Assessment: Good

The research funds available to MobD are severely limited (no funds for research to develop fundamental new knowledge and very limited funds for concept exploration). Furthermore, MobD receives no support from the STD. Consequently, MobD has little chance of developing pacing technologies. However, the parachute inflation simulation and the advanced precision airborne delivery system have the potential to be leap-ahead developments in parachute technology provided a sustained, reasonable level of funding is available to support them. The ground mobility mission of MobD, which represents a small fraction of MobD's activity and has no research program, was assessed as poor.

Assessment: Adequate

The core capabilities important for SurD include: chemical warfare protection; ballistic, laser, flame, and surveillance countermeasures; textiles; systems integration; clothing design; textile testing and functional finish formulation; color science and evaluation; specification development; electron microscopy; and system integration. These capabilities rely on the disciplines of anthropometry, biomechanics, mathematics and physics modeling, human engineering, mechanical engineering, and materials engineering. Eighty percent of the SurD's core capabilities are utilized in one of five programs: countermeasures to battlefield sensors; integrated headgear and laser eye protection; ballistic protection; percutaneous chemical protection; and multifunctional protective materials and uniforms. The other 20 percent of the core capabilities resides within the two support directorates in three other programs: enhancing warrior performance and endurance; bioengineered materials; and systems analysis. All eight programs are well defined and have specified qualitative or quantitative objectives.

As a rule, SurD considers the science and technology programs as support technologies for the SurD mission rather than as pacing technologies. Exceptions to this rule are aspects of the Force 21 Land Warrior program and the development of thermal infrared and agile-laser eye protection. Technological improvements developed from the science and technology program that have found their way into products include: advanced ballistic materials; lightweight materials for chemical and biological protection; durable static resistance for battle wear; miniaturized components for microclimate cooling; water resistant, fire resistant, camouflage printed, near-infrared signature reduced, antistatic, high durability, and high strength materials; chemical protective underwear; and reflective technology for laser-protective eye wear.

The core capabilities of SurD are threatened, however, by restrictions on funding and hiring that SurD believes make it impossible to hire well known experts or well trained technologists. Retirements and promotions (primarily to administrative functions for reasons of greater responsibility and higher pay) have translated into a loss of critical expertise in some areas (e.g., color science, shade evaluation, and textile testing). In response to these restrictions, SurD has opted to focus on a limited number of technologies related to countersurveillance, fiber and fabric development, and chemical and physical resistance. SurD intends to obtain other needed capabilities externally (an example being the relatively low value, task-order contracts, which were discussed earlier, that were issued for converting detailed specifications to performance specifications).

Assessment: Adequate to Good

Core SusD science and technology programs in the rations and equipment and shelters business areas are: combat ration research; preservation, packaging, and stabilization technologies; performance enhancing and nutritional technologies; and equipment and energy technologies. Currently, SusD has no science and technology programs in the unit and organizational equipment portion of the equipment and systems business area or in the shelters business area. However, recent program breakthroughs have led to some science and technology funding for air beams. Development is focused on group, individual, and special purpose rations; veterinary inspection; systems for field, shipboard, and airborne feeding; and organizational equipment. Illustrative pacing technologies are noted below.

Rations. Pacing technologies in rations include: targeted nutrient delivery; molecular level analysis of food matrices; advanced physical methods for food preservation; intrinsic chemical-marker-validation of process effectiveness;

biopolymers as food preservatives; formulation and processing for extending shelf-life in extreme environments; irradiation; and encapsulation, coating, and enrobing.

Equipment and Systems. Pacing technologies in the equipment and systems area include: diesel and JP8 (standard jet aircraft fuel) combustion; heat transfer; integral power generation; heat-driven refrigeration; exothermic, self-heating rations; self-heating technology; multifuel burners; and catalytic conversion of diesel fuels.

Shelters. Pacing technologies in shelters include: physical, mechanical, and chemical behavior of high performance materials; processing of high performance materials; structural mechanics and modeling; advanced textile manufacturing techniques; smart materials with active response to multiple stimuli; electromagnetic inhibiting materials and manufacturing techniques; and high strength, insulative structural elements.

Each SusD program is focused on a targeted set of operational capabilities for a particular area. All programs are initiated in response to the general requirement to provide rations, feeding systems, and shelters that are lighter, easy to use, rapidly deployable with reduced logistic burden, and that can be used for a broad spectrum of military applications. All of the SusD programs appear to be pacing programs, i.e., they are concentrated on basic R&D that will ultimately provide either new or enhanced capabilities. The science and technology program has resulted in several developments that have been incorporated into products. These include the multifuel burner, Hooah bars (high energy snack bars included in field rations), and pouch bread for meals-ready-to-eat.

Assessment: Good

The committee determined that the metrics associated with a good performance level best fit the important technologies of the commodity directorates. The committee found examples of pacing technologies both being developed and already incorporated into products. The committee also found numerous examples of base support technologies that fulfill a technological need but do not lead to dramatic changes in performance. The mix of pacing and base support technologies seems appropriate for the missions of the commodity directorates.

Assessment: Good

Most MobD personnel noted during interviews that the new, flat organizational structure has encouraged teams to set goals and seek innovative solutions with less intervention from management. But the committee observed that many MobD staff members (both senior and junior) did not approve of the changes in organization. Typical negative reactions were variations of a reluctance to change. Few complained of being saddled with more work, and, in fact, considering the cuts in the workforce, there was surprisingly little talk about having to work extra hours or taking on new responsibilities. The committee noted that most staff members of MobD display a can-do attitude and that they appreciated the work of the director and technical management office. Staff members were concerned about ongoing reorganization and downsizing within the Department of the Army.

Assessment: Good

The recent reorganization and continued downsizing of SurD have resulted in many new assignments, new bosses, and physical moves. These changes, along with increased workloads, less job security, and less opportunity for advancement, have increased stress and, in many cases, lowered morale. In spite of this atmosphere, job satisfaction in many cases was still high. Most interviewees felt that Natick was a good place to work.

Assessment: Adequate

Everyone who was interviewed indicated that they liked their jobs; and turnover is extremely low in SusD. The personnel are dedicated and committed to what they are doing. Their primary focus continues to be on their main customer, the soldier, and they will do whatever it takes to meet his or her needs. However, the SusD personnel are anxious about downsizing and future budget cuts, which are causing concerns about more outsourcing as well as the belief that no one will be promoted because there will be no openings. The personnel are also concerned about the lack of communication with management (despite the fact that a communication system known as Straight Talk is available to everyone), and there is a feeling, especially among personnel in the shelters business area, that

management does not pay attention to their concerns. At the same time, some complained that things were being micromanaged.

Assessment: Adequate

The committee determined that the metrics associated with an adequate performance level best fit the overall organizational climate of the commodity directorates. The committee noted that the vast majority of personnel interviewed perceived the work environment as professional and collegial and felt that their work was meaningful. Teamwork and collaborative efforts abound. Some personnel felt that bold, innovative thinking was encouraged and rewarded, but this feeling was not shared by the majority. In general, most people were extremely anxious about ongoing reorganizations and downsizing, and many felt insecure about their jobs. For these reasons, the committee assigned an adequate rating.

Assessment: Adequate

Nearly all MobD employees had personal computers ranging from a 386 processor to a Pentium processor. All personal computers had software for word processing, spread sheets, databases, time management, internet, and e-mail. MobD also had several workstations, all software was state of the art, and e-mail was the standard mode of communication. Information technology had enhanced MobD effectiveness during the external air-transport certification process; answers could be provided in the field by notebook computers. The Soldier Systems Command maintains an information systems plan that covers the MobD as well as other directorates.

Assessment: Good

Nearly all employees had personal computers with capabilities ranging from a 386 processor to Pentium. Virtually everyone was computer literate, and e-mail is now the standard mode of communication. Improved communication has

reduced the amount of time spent in meetings, and information technology has improved cost effectiveness and created high quality marketing and presentation capabilities. Project planning and management have benefited greatly from information technology. An example is the production line analysis system, which has significantly enhanced communication with SurD customers.

Assessment: Good

Everyone interviewed by the committee noted that they had adequate to superior computer hardware and software. Computers were 486s or Pentiums and had ready access to the internet and e-mail so everyone could gather data easily and keep in contact. Equipment in SusD has been continually updated, and computers are used for making important analyses.

Assessment: Good

The committee determined that the metrics associated with a good performance level best fit the overall information technology of the commodity directorates. The committee also found evidence of an information technology strategy at the Soldier Systems Command level. Most personnel noted that e-mail allowed them to do their jobs more efficiently. Hardware and software were state of the art, and staff members had ample opportunity to attend training programs in the use and application of information technologies. There were indications that information technology had improved effectiveness. Nevertheless, the committee did not find substantial evidence that information technology had reshaped the way RD&E are performed in the commodity directorates (e.g., that directorates use databases to improve their data gathering capabilities or employ rapid prototyping techniques).

Assessment: Good

During the on-site interviews, the staff noted that the facilities were clean and adequate, although they were not all environmentally controlled year round. The lack of air conditioning 24 hours a day made it difficult for employees to

work early, late, or on weekends. The buildings were old, and repairs were sometimes delayed, although some preventive maintenance and service contracts were in place. Safety and regulatory compliance policies were also in place (e.g., frequent notices of asbestos hazards, wheelchair access to all buildings). Overall, the facilities of MobD were judged to be adequate.

Assessment: Adequate

The facilities and much of the equipment at SurD are 20 to 40 years old, and very little has been done to upgrade them in the last several years. For example, the air conditioning system needed updating, especially in the physical testing laboratories where temperature and humidity are critical. Multiyear funding has recently become available so needed improvements are expected.

Assessment: Adequate

The facilities and infrastructure of SusD were sufficient to get things done, but they were not well maintained. Maintenance contracts existed for some equipment that could not be maintained in-house. Other machinery was repaired on an as-needed basis. Many of the staff members interviewed stated that funds budgeted for facility maintenance and renovation were insufficient. The present situation differs sharply from the past when funds were available for preventive maintenance. Most facilities were sufficient and clean, but many repairs had been put on hold.

Assessment: Adequate

The committee determined that the metrics associated with an adequate performance level best fit the overall facilities and infrastructure of the commodity directorates. Most personnel noted that the Natick facilities were adequate, but few characterized the facilities as spacious or comfortable, and many mentioned ongoing problems with the air conditioning system. There was

periodic maintenance for most equipment and facilities, but major repairs and upgrades were often implemented at a snail's pace because they require multiyear funding. Safety and regulatory compliance policies were in place.

Assessment: Adequate

The committee's overall assessment of the resources and capabilities pillar for the three commodity directorates is good (see Table 2-1). However, the committee notes that, in some instances, the vector for the future is pointing down. The most notable examples are in personnel quality and RD&E capabilities, skills, and talents, where the loss of experienced people could have serious long-term implications for the Natick RDEC. Also, facilities could deteriorate if they are not maintained. The commodity directorates are making good use of information technology and, for the most part, of external resources. These can compensate, in part, for the loss of in-house personnel.

The committee was discouraged about strategic vision at the Natick RDEC during the site visits in 1996. The committee decided, therefore, to revisit strategic planning in early 1997, and the results were encouraging (see discussion at the end of this section). The assessment below reflects the situation of the commodity directorates as of 1996, a situation that could continue if the improvements that were initiated by early 1997 are not fully implemented.

The MobD vision is to be a global leader in meeting mobility needs (broadly defined). However, personnel within MobD are not fully supportive of their newly defined mission, which includes working on airdrop systems, terrain traversal, and personal augmentation products and technologies to protect, sustain, and improve the quality of life for soldiers. The focus is now primarily on airdrop technology, with limited attention to terrain traversal and personal augmentation. MobD provides support and technology development for airdrop systems across the DOD. Because of the way the RDEC is structured, however, its programs and resources are not truly supportive of the broader airdrop mission.

Because the Natick RDEC is an Army organization, research programs within airdrop are tailored to meet the perceived needs of the Army and are focused on the requirements for individual soldiers. The requirements of associated forces (e.g., the Air Force, which conducts airdrop operations in

support of foot soldier missions) are not considered germane to the Army and, consequently, are not funded by the Army. These programs must rely on independent financial assistance from the relevant service. Thus, the funding structure for MobD is inconsistent with its mission.

Members of MobD understand the soldier-as-a-system mission of the Soldier Systems Command, but they do not understand how mobility, and specifically airdrop (which, to MobD means delivery of personnel, supplies, and equipment from aircraft) fits into the mission of the command or even how it fits into the mission of the Natick RDEC. This is of great concern to the staff. Senior-level management has held several town meetings to convey the RDEC mission and vision, but several MobD staff members feel the mission statement is incomprehensible and has been written for public consumption.

Assessment: Poor to Adequate

The SurD mission involves RD&E programs for combat and noncombat clothing and individual equipment, and the directorate's vision is to become the world leader in individual survivability technology. Joint teams of management and staff have formulated vision and mission statements with input from the workforce. The vision and mission have been communicated through town meetings and posters. The members of SurD see the individual soldier as the core of their mission and clearly understand their roles in support of their mission. Somewhat problematical, however, is their link with the broader mission and vision of the RDEC regarding the soldier-as-a-system. Interviews revealed a conspicuous lack of support and understanding within SurD for the soldier-as-a-system concept.

Assessment: Poor to Adequate

Within SusD, the committee found that there was a dichotomy of responses to the terms vision and mission. Most people were not only aware of the vision but were also able to recite it. However, they did not know (or at least were unable to verbalize) SusD's mission. At the same time, they were aware that their primary objective was to serve soldiers, and they were dedicated to this mission and determined to do whatever was necessary to ensure that soldiers' needs were met. The personnel admitted, however, that they did not feel there was a strong

link between the SusD vision and their mission. They felt the mission statement was long, incomprehensible, and written for public consumption.

Assessment: Poor

The committee found that few people appeared to understand the broader mission and vision of the Soldier Systems Command or how their directorate's vision and mission were linked to those of the command. Surprising as this might seem, the committee suggests the following explanation.

The soldier-as-a-system vision was part of an advanced concept that migrated to the Soldier Systems Command from the Natick RDEC (and elsewhere) when the command was formed. The creative people behind the idea migrated with it, and the remaining RDEC personnel now have a strong ''us versus them'' attitude about the command. Only a few people are left at the RDEC to convince their colleagues that they are vital to the soldier-as-a-system concept, and managers at the RDEC have not convinced their employees that their work is integral to the concept. As a result, the workforce does not see itself as part of the larger picture called soldier-as-a-system, although workers are very proud of their contributions to the "soldier." In short, the directors have not done a good job of explaining the vision (see later section on Leadership), and the culture of the directorates remains insular and narrow. To correct this situation, leaders must try to communicate vision and coach their workers during every conversation and every meeting.

Assessment: Poor to Adequate

No strategic planning process has been implemented at MobD since the formation of the Soldier Systems Command. MobD strategic planning is related to the funding level. Programs at the concept exploration level are proposed on the basis of needs identified by customers and operational capability requirements from the Army battle laboratories. Programs are reviewed and ranked by the Natick RDEC's planning integration team, senior technical staff, and others for technical merit, probability of success, and return on investment. They are funded, in order of rank, until financial resources are depleted. Programs at the advanced

development level are based on the battle laboratories' requirements and the Army Science and Technology Master Plan, proposed as either advanced technology demonstrations or programs to satisfy science and technology objectives; they are ranked and funded through the battle laboratories.

To a significant extent, therefore, strategic initiatives at the Natick RDEC level are dictated by visions communicated from outside the RDEC. For airdrop technology, for example, there appears to be little or no long-term concept development within the Army. This is a direct consequence of the Army's narrow definition of airdrop as serving only individual soldiers. The Natick RDEC has not been able to influence other airdrop requirements, even in direct support of Army activities. Thus, no broad strategic vision for airdrop systems has been communicated downward to the RDEC; and broader mission or vision statements communicated upward from the RDEC have not been taken seriously.

Assessment: Poor

The planning process for SurD's science and technology programs is based on requirements in the Army's Long-Range Research and Development Plan and the RDEC five-year integrated planning process, both of which have a five-year to six-year planning horizon. Proposals for research by individuals are submitted to the planning integration team for assessment. Individuals are encouraged to base submissions on needs and deficiencies identified by customer focus groups or the battle laboratories. Proposals are subject to three evaluations. The planning integration team assesses proposals for technical approach, probability of success, innovation, resource availability, timetable for return, and value to the customer. Senior scientists conduct technical reviews, and another team evaluates them for return on investment. Proposals are ranked by consensus of all three evaluation panels, and funds are distributed accordingly. Unfunded projects are reconsidered as funds become available, and proposals are usually resubmitted in subsequent funding periods.

Contingency planning is reactive, as needed to solve immediate problems. From a strategic perspective, there is little or no internal planning to support long-term objectives or to justify current science and technology projects. Many interviewees noted that work within SurD is subject to priorities set by Washington, and interviewees repeatedly referred to "firefighting" or "crisis management" as the common mode of operation. They measured progress towards achieving goals by program and budget growth and improvements in productivity and product quality, benchmarking against SurD metrics and metrics from the Government Performance and Results Act.

Assessment: Poor to Adequate

The SusD has a strategic planning process in place and formulates business plans and annual plans. Senior-level management relies on both top management personnel and support staff for assistance. However, most of the personnel who were interviewed felt that the strategic plan had not been communicated to them. In general, the feeling about the nature and scope of these long-range plans was negative. Many of the personnel, especially those in the shelters business area, believe that the R&D expertise of SusD is being allowed to deteriorate. They believe that in the next decade the organization will become a procurer of materials, products, and services and will no longer do any research. Therefore, there is a general lack of trust, even when senior management tries to communicate long-range objectives to personnel. Many employees believe that SusD is going to be eliminated (at least in its current form) and that the strategic plan is not an operational plan but a requirement mandated from above.

Assessment: Adequate

The committee determined that the metrics associated with an adequate performance level best fit strategic planning by the commodity directorates. Some strategic planning processes have recently been implemented for the direction of the research program. Directorate-level business plans, when they exist, have been formulated without broad involvement from the organization. But planning documents are not used to measure progress throughout the year, and contingency or alternative planning is simply reactive.

Assessment: Adequate

MobD's ultimate stakeholders are soldiers, but there are also other stakeholders, including MobD personnel, who recognize their mission only as it relates to airdrop technology and not as it relates to the broader area of ground-mobility requirements. The rationale for incorporating the latter within this directorate is neither understood nor welcomed by many members. Even within the specific focus of research and technology on airdrop systems, not everyone understands how projects support and complement each other. For example,

members of the product development group (in MobD) do not appreciate the potential impact of the modeling groups (in MobD and in ASCD). MobD groups are isolated from each other according to the type of funding they receive.

Assessment: Poor

SurD defines its stakeholders as all internal personnel, the Defense Logistics Agency, Program Manager-Soldier, and all customers. On an ideal level, SurD considers stakeholders' trust and the directorate's continuing relationship with the stakeholders as buy-in to the directorate's vision. On a more pragmatic level, SurD relies upon stakeholders' involvement in the integrated planning process and customer workshops. The committee found no concrete examples of proactive stakeholder buy-in. Interviewees repeatedly said that they wanted to be left alone to perform their research and not be bothered with the bureaucratic and management details of justifying and assimilating their work into the integration team's plans. The committee agrees that implicit trust, ongoing relationships, and ample funding are indicators that stakeholders are supportive of SurD activities but is not convinced that they represent a buy-in to the vision.

Assessment: Poor

SusD personnel recognize that their ultimate stakeholder is the soldier, and feedback from soldiers is monitored and used for making changes in the design, development, and creation of goods and services. Soldier feedback indicates that they have a fairly strong buy-in. Based on the interviews, however, other major stakeholders (i.e., the in-house SusD personnel) have not bought into the organization's vision and mission. Many do not understand the plan; most cannot explain it. More important, they appear to be indifferent to what top management is doing because they are convinced that, as long as they do their jobs, the ultimate stakeholders, the soldiers, will be well served. The committee believes that another reason for the failure of employees to buy in is that they believe their areas are being downsized and that their core competencies will eventually be outsourced to private firms that can do what they do at a lower price.

Assessment: Poor

The committee determined that the metrics associated with a poor performance level best fit the stakeholder buy-in of the commodity directorates. The commodity directorates have both external and internal stakeholders, but the committee was able to interview only the internal stakeholders.⁵ Consequently, their responses weigh heavily in this evaluation. In general, directorate personnel do not understand and, therefore, do not buy into the organization's vision and mission. Their response to the directorates' visions and plans is indifferent, at best.

Assessment: Poor

Management of the Natick RDEC has attempted to encourage staff involvement by establishing a planning integration team. Given current budget restraints and the fact that proposals are initiated on the basis of indicators external to the RDEC, the team has become reactive rather than proactive. Although the broad mission of MobD in air and ground mobility has been convincingly communicated to the staff, some suggestions by the team that worked on reorganizing the directorate were not implemented. The staff trusts the current leaders of MobD but also believes they are overworked and, therefore, constantly having to put out brush fires, leaving them no time for mentoring, planning, or implementing plans. On this issue, both staff and upper management agree. The committee believes that the unavailability of upper management is a serious limitation on the effective functioning of MobD.

Assessment: Poor to Adequate

Although there appears to be a good deal of support for the stand-alone mission of the SurD and trust that leadership is doing its best to fulfill that mission, there appears to be little or no understanding of, or support for, the broader mission of the soldier-as-a-system. Many interviewees noted dwindling resources, deteriorating facilities, and the loss of personnel as impediments to

meeting their goals. Several interviewees were concerned about the increasing amount of contract management they had to do. In the committee's view, these problems reflect back on SurD leadership.

Assessment: Adequate

The strategic vision and plan of SusD are not well understood by the staff, and many individuals who were interviewed felt that resources were not being aligned to meet strategic objectives. In addition, the personnel do not seem to have much confidence that senior-level management will implement the strategic plan. In fact, lack of trust in organizational leadership seems to be growing. In many cases, personnel view actions taken by senior leaders as reactive. For example, budgets are not totally allocated at the beginning of a project or fiscal year. Instead, additional funds are often made available as the year wears on. As a result, personnel may initially have to work on a shoestring budget believing that there may not be enough money to complete the project, or at least not enough to do things correctly. After additional funds are allocated, adjustments are made to complete the project within the predetermined guidelines. This budgetary approach is one reason personnel feel that senior-level management does not provide strong, directive leadership.

Assessment: Poor

The committee determined that the metrics associated with a range of poor to adequate performance best fit the leadership of the commodity directorates. The strategic vision of the directorates is not well understood by staff, especially the links between the vision of a directorate and the vision of Soldier Systems Command (see earlier discussion of alignment of vision and mission). The plans in the directorates appear to be mostly reactive, which limits the alignment of resources with the strategic vision. In some directorates, senior leadership is trusted; in others, there is growing distrust. Senior leadership is viewed as receptive to new ideas in some directorates but not in others. The committee believes that the senior leaders of the commodity directorates have a long way to go before they can "create an air of excitement and commitment through the entire laboratory," which is indicative of excellent leadership and a world-class organization.

Assessment: Poor to Adequate

In February 1997, the committee was informed that the Natick RDEC was developing a new strategic planning process. This process was initiated internally to help position the RDEC for the future and not in response to an Army requirement for planning. It is too early to assess the new planning process, but the committee is optimistic that this is the first step in improving the Natick RDEC's performance level in the Strategic Vision pillar. Perhaps now planning within the directorates will be better aligned with the Natick RDEC and Soldier Systems Command vision, mission, and strategic objectives.

The committee's overall assessment of the Strategic Vision pillar for the commodity directorates is poor to adequate (see Table 2-2). The lack of acceptance of a strategic vision suggests an endemic shortcoming in the management as far as leadership qualities are concerned. The encouraging factor is that, during the February 1997 site visit, the committee learned that strategic planning had been moved from a support directorate to the top levels of the Natick RDEC where it could receive effective management oversight and attention. The committee believes that strategic planning must go well beyond the articulation of vision and mission statements; it must permeate the organization in a participatory way and lead to realistic strategic plans and programs that match the vision and mission of the organization. Leadership must stand firmly behind strategic planning, which is essential for world-class performance. On balance, the committee believes the strategic planning vector is beginning to point upward, but diligence on the part of leadership (i.e., dedicated commitment, which is the foundation of all five pillars [NRC, 1996]) will be needed to keep it pointing that way.

The examples of breakthroughs cited by MobD (the advanced precision airborne delivery system and guided parafoil aerial delivery system [light]) are indeed noteworthy. Still, these airdrop systems use existing technology, parafoils and remote sensors, rather than changing the basic nature of parachutes. The ideas for both systems were picked up by several individuals at the Natick RDEC who are well connected with aeronautical networks, but the ideas originated in the National Aeronautics and Space Administration and probably will not result in patentable properties for the Natick RDEC. Hence, although both programs represent substantial technological improvements, neither is a surprise or a true breakthrough. The management of MobD is trying, however, to create an atmosphere that encourages advances and creates the capacity for breakthroughs. Presently, due in large part to mission requirements, most MobD advances are not revolutionary in character, that is, they build incrementally on existing technologies and capabilities.

Assessment: Adequate

Most programs at SurD are characterized by incremental improvements, but several examples were cited as innovative, including primed Nomex™

(Nomex is the trademark for DuPont's aramid fiber), which has multifunctional capabilities (flame resistance, camouflage, anti-static capabilities, liquid-chemical-agent resistance); Primaloft™ synthetic-down insulation; ChemPak™ LT™, a semipermeable membrane that protects against chemical agents; Kevlar™ KM2 (Kevlar is the trademark of another DuPont aramid fiber, and KM2 is the grade used by the Natick RDEC) and concepts for ballistic protective systems; and nonlinear optic materials to protect eyes from lasers. Most of these breakthroughs were the end products of materials development by outside contractors that were managed by SurD project officers. SurD has little funding for long-term research to develop fundamental, new knowledge that could lead to breakthroughs. In addition, the committee detected an attitude of complacency among the interviewees toward obtaining new and innovative results from research, and they gave few examples of successful, or potentially successful, research. Research was characterized as excellent if the customer was willing to continue or increase funding.

Overall, SurD has been a resourceful integrator of material systems and design concepts to fulfill specific mission requirements. The capacity for breakthroughs has been enhanced through SurD's increased cooperation with industry, academia, and other government agencies.

Assessment: Adequate

SusD personnel identified their most significant breakthroughs as the development of shelf-stable bread products, the catalytic conversion of diesel fuels (leading to multifuel burners), the flameless ration heater, and high-pressure air beams. The committee agrees that these are significant technological breakthroughs, all of which resulted from joint internal and external projects. The directorate promotes teaming and is attempting to change from a culture that avoids risks to one that takes risks to foster innovation. However, a high percentage of the SusD budget necessarily goes to requirements-driven projects, thus limiting the science and technology resources for research that could lead to breakthroughs.

Assessment: Adequate

The committee determined that the metrics associated with an adequate performance level best fit the capacity for breakthroughs of the commodity

directorates. The RD&E programs are characterized by steady but incremental improvements. Several innovative and leap-ahead improvements have been made, and some directorates are attempting to encourage their staffs to seek innovative, moderate-risk solutions. The committee felt that the adequate performance rating reflects the minimal funding available for anticipating future military requirements, much less for funding moderate-risk or high-risk solutions.

Assessment: Adequate

Virtually everyone in MobD understands the importance of quality, but the committee found no hard evidence of quality projects, changing work processes, or metrics for assessing quality. The number of parachute malfunctions has dropped dramatically, however, from more than 800 per year in 1976 to about three per year today. The quality deficiency reports collected three times a year at Fort Lee are rapidly addressed by MobD and have led to continual improvements in Army parachutes. The committee also recognized that project teams are now a way of life at the Natick RDEC and that the integrated planning process allows input from across the RDEC concerning a project. MobD's response to the Government Performance and Results Act also represents an attempt to formalize quality metrics.

Assessment: Adequate to Good

Some quality systems and measures are in place in SurD, and the directorate can point out several improvements (e.g., SurD was the source of ideas for improving body armor). SurD also has its own plan for responding to the Government Performance and Results Act and uses some customer feedback to support continuous improvement. If an employee has a suggestion, that person can go directly to the project officer, and, for the most part, action will be taken. No report cards or measures, however, are in place to encourage workers in SurD to strive for continuous improvement.

Assessment: Good

Everyone in SusD understands the importance of quality. Although the process of continuous improvement has not been formalized throughout all SusD divisions, numerous mechanisms support continuous improvement. The committee found that all directorate personnel are exposed to formal training in total quality management practices and continuous improvement processes (e.g., the Continuous Improvement Ration Program). Teams of interviewers visit soldiers in the field and assess their satisfaction with rations. These assessments, as well as unsolicited customer feedback, have led to several cycles of improvement in meals-ready-to-eat. Finally, the President's Quality Award assessments and Government Performance and Results Act process are being used to encourage continuous improvement (e.g., annual reports on the Act are being used to measure process improvements within SusD). SusD is still refining its use of metrics.

Assessment: Good

The committee determined that the metrics associated with a good performance level best fit continuous improvement in the commodity directorates. The directorates have taken steps to improve work processes, and quality audits are performed periodically by external reviewers. Some directorates have programs in place to encourage continuous improvement. Product and process improvements have been made in every directorate. Formal internal reports, ''report cards,'' are not issued annually by senior leadership, however, and there are few systematic analyses of R&D and support processes to weed out activities that are of little or no value.

Assessment: Good

MobD staff members seem to want to improve quality. An initiative to improve quality is now under way, and some selected metrics are planned. The leadership recognizes the importance of quality but has not had enough time to devote to this initiative. The Government Performance and Results Act, an externally driven process, is used to measure and assess quality.

Assessment: Adequate

Interviewees described many systems and programs that have already been established to improve quality. With the cooperation of the workforce, SurD is developing appropriate metrics for assessing R&D programs. Several individuals have taken quality training, and the directorate hopes to receive the President's Quality Award. Although some employees do not wave a total-quality-management banner, they are committed to doing quality work, based on the belief that the work of SurD determines life or death for the soldier. Senior management of SurD is a driver for quality work and quality systems. When management asks questions concerning quality, workers know that follow-up actions will be taken. But the committee was told that it had been four months since the last town meeting when the issue of quality had been discussed. Several interviewees rated their own areas as excellent because of their extensive use of standards and other criteria for measuring quality.

Assessment: Adequate to Good

SusD obtains feedback from soldiers in the field to improve quality. Interviewees cited specific instances where feedback had resulted in improvements, most notably in the combat rations program. The directorate also maintains a file of unsolicited comments from soldiers in the field and circulates these comments. The committee did not observe any formal, internal mechanisms for measuring quality but learned that SusD is studying the metrics listed in the committee's phase-one report and plans to formalize a self-assessment process. Many of the persons interviewed recognized the importance of their work and stressed that quality would be important to them even in the absence of formal quality improvement programs. When the committee asked SusD personnel to assess their commitment to quality on a 1 to 10 scale, with 1 being a low score, most rated the directorate as a 7. This rating reflects the commitment to quality within SusD. Finally, the directorate has actively pursued the President's Quality Award initiative and plans to continue this effort.

Assessment: Adequate

The committee determined that the metrics associated with an adequate performance level best fit the commitment to quality of the commodity directorates. Total quality implementation is a major goal of all three directorates

and their support staff; management has been willing to invest resources in quality training; and personnel are aware of the importance of quality. However, the committee found few formal internal metrics being used to measure or assess quality at the Natick RDEC. The committee did not find established, measurable objectives for improving work processes.

Assessment: Adequate

Work processes in MobD are well defined, and steps have been taken to simplify development without compromising safety (e.g., simplifying work for Special Operations), but the committee noted a lack of coordination between research, development, and applications. Processes for budgeting and acquiring materials can also be improved. Both the integrated planning process and the planning integration team represent recent attempts to provide structure for the overall planning process, but these programs are too new for the committee to judge their effectiveness. However, they do have the appropriate elements (stakeholder and customer input, team-set objectives, and project monitoring) to be potentially useful, although some fine-tuning may be required after several years of experience. MobD has recently implemented the Government Performance and Results Act, which should help reduce cycle times.

Assessment: Adequate to Good

Work processes at SurD are well documented and highly structured in the development area. The process known as the Total Army Performance Evaluation System is used to monitor individual processes. Work orders are carefully followed and documented, although interviewees suggested that laboratory notebooks may not be as carefully kept as they had been in the past. A structured process has recently been implemented for the submission, review, and funding of proposals. Once funded, projects are reviewed on an annual basis by the planning integration team; costs and schedules are managed by the command finance system and by the Natick RDEC progress and results system (a management-by-objective program). Research and technology projects can be terminated if the goal is determined to be either unachievable or no longer critical to warfighting needs. The termination authority resides with the original approving body. Results of R&D are documented as Natick Technical Reports and through professional

journals, society meetings, and patents. The database of the Defense Technical Information Center provides a vehicle for the broad dissemination of R&D results.

Assessment: Adequate to Good

SusD uses an integrated planning process to define its science and technology programs. Potential programs, based on needs identified by customers or requirements from the battle laboratories, are submitted by individuals and reviewed for technical merit and return on investment, and funds are allocated based on priority rankings. A list of unfunded projects is maintained for contingency funding. Programs are reviewed annually, and progress towards project goals is tracked by the Natick RDEC progress and results system; costs are tracked by organizational cost reports. Projects can be terminated if the goals are deemed unachievable or no longer relevant. Termination authority resides at the original approval level.

Adjustments in programming are made through the integrated planning process, and program results are documented in Natick Technical Reports, refereed journals, and presentations. Reports and publications are available for general distribution through the Defense Technical Information Center database. The directorate has recently implemented annual reports under the Government Performance and Results Act to assist in program management. However, the committee questions the adequacy of the processes used to quantify each product's contribution to the performance of the soldier system.

Assessment: Adequate to Good

The committee determined that the metrics associated with a range of adequate to good performance best fit the structured processes of the commodity directorates. Work processes and procedures are monitored, and project costs and milestones are closely tracked. Several programs have been put in place recently to improve the quality (possibly more than incrementally) of the R&D program as well as to reduce cycle time. Senior leadership is trying to promote a supportive culture in which leaders can be receptive to ideas for improving work processes. There are no established, measurable objectives for determining the continuous improvement of product quality or work processes.

Assessment: Adequate to Good

Since the hiring freeze, individuals have been encouraged to keep up their skills or develop new ones (e.g., computer literacy). The committee found a major commitment had been made to improving the MobD learning environment. Even when expenses had been cut drastically, individuals were able to travel to training courses. The committee is concerned, however, about learning across the organization and about the transfer of information, lessons, and skills from team to team and group to group. Networking and interactions among groups should be improved. The committee is also concerned about how MobD can capture the experience of older employees who will retire in the near future. MobD's new "flat" organizational structure may improve organizational learning, but it is too new for the committee to judge its effectiveness.

Assessment: Adequate to Good

Several people interviewed believe that organizational learning in the SurD is poor. They felt that management is too focused on maintaining a "happy family" atmosphere to risk disturbing the status quo with changes or criticism. However, the committee found that managers encourage training courses, development, and continued education, as well as cross training on the job. This encouragement, which should continue, could become part of a strategy for capturing the knowledge of older employees who are ready to retire. Unfortunately, many individuals appear to be too busy to take advantage of the educational opportunities, and use of the library has dwindled. There was some informal networking (e.g., one project officer described lessons learned from a project that had failed because potential manufacturers had not been involved early in the process). This is an effective way to learn across the organization. Lessons could also be learned by sharing experiences in joint-service activities. The committee was told of overkill in some training areas (e.g., the acquisitions course, which elicited one response of "it would boggle your mind what had to be learned by all!").

In SurD, the team concept has replaced the individual, which means there is not as much control at the individual level. Everyone agreed, however, that teaming has helped them get things done more efficiently. Many of the team leaders are new in their positions following restructuring and could learn leadership skills from others. The committee believes the best example of a good learning environment in SurD is the Warrior Systems Group, which meets often

for team training and is very focused because of its involvement with the DOD Infantry School.

Assessment: Good

On an individual level, SusD workers are encouraged to receive training and develop new skills (e.g., computer literacy, acquisition process training). Even when expenses were being cut, individuals were still able to travel to training courses. When the committee asked SusD staff members to rate the learning environment, most spoke highly of training opportunities. However, the committee is concerned about learning across the organization and the transfer of information, lessons, and learning from team to team and from group to group. During the interview process, many SusD staff members suggested that interactions among groups were weak but improving. Also, many staff members expressed concerns about the ability of the directorate to capture the experience of older employees who will retire in the near future.

For several reasons, the committee is concerned about the adequacy of databases or records of past projects and studies (e.g., laboratory notebooks are maintained in the library for a period of time but are not reduced to microfilm, indexed, or cross-referenced). First, the library staff, including those who had been keeping up the microfilm records, had either been greatly reduced or eliminated; so no help was available to look for information. Second, the project leaders interviewed did not know of any sources, other than themselves or past leaders (if they were still around), who could provide details on the history of projects. Third, project leaders were not making any conscientious attempts to keep records that could be retrieved or comprehended by others. Although the SusD leadership insisted there was a well established policy for project leaders to keep formal records, the interviewees were generally not aware of it.

Assessment: Adequate to Good

The committee determined that the metrics associated with a range of adequate to good performance best fit the learning environment of the commodity directorates. The committee notes that a learning environment can be evaluated on an individual basis or an organizational basis. The commodity directorates could be considered to have good, perhaps even excellent, environments for individual

learning. Individuals have tremendous opportunities to attend courses and seminars. However, at the organizational level, which is the focus of this assessment, no formal processes are in place to ensure that management and staff learn from mistakes or from each other. Team interactions are improving, but teams do not always have a formal way (or sometimes even an informal way) of learning from each other, especially across directorates. The committee notes that management is experimenting with new organizational structures to find ways to promote organizational learning.

Assessment: Adequate to Good

Using the committee's metrics in a general way, several staff members rated the MobD research as adequate to good. But concerns were raised that research was not correlated well with development, although the situation appeared to be improving. The new integrated planning process gives everyone a vote, which means that research can no longer remain in a vacuum. Several interviewees observed that the modeling and simulation at the Natick RDEC may be the best in the Army, but that capability may not be appreciated outside the RDEC. Certain development capabilities, such as heavy-cargo airdrop, are among the best, if not the best, in the world. The staff members have published papers, attended scientific meetings, and received many awards and patents.

Assessment: Adequate to Good

Twenty years ago, the SurD laboratory's research was widely recognized outside of the RDEC. In fact, researchers came to Natick to do quality research. Considerably less research is done today, and many researchers are being encouraged to complete their projects. Still, some quality work is being done. Current research and technology programs have their genesis in needs defined by the customer and, therefore, are generally aligned with customer requirements. SurD assesses the quality of these programs in the planning-integration process, looking for correlations between research and technology, development, and customers. SurD considers approval of a system technology objective by the user community as affirmation of the quality of the research and technology. Other measures of quality include awards (e.g., a Soldier Systems Command Pin for Development

and Engineering, an Army Science Award), patents (two in the last two years), and publications in peer-reviewed journals (six in the past two years and four in press). SurD offered the committee several examples of aspects of its programs that have been recognized externally.

Assessment: Adequate to Good

SusD assesses the quality of research and technology in the program-integration process and through evaluations of technical merit and return on investment. External assessments are obtained through DOD and Army reviews, the publication of papers in peer-reviewed journals (30 in the past two years, and 10 in press), patents (five in the past two years), presentations at scientific meetings, and by technical personnel being asked to serve on scientific and technical committees worldwide. Two examples of external assessments are: recognition by DOD of SusD's High Pressure Air Beam Program as an outstanding science and technology program, and the restoration of funds to the DOD Food Program. The directorate has received several awards, most notable among them being the DuPont Award for Innovation in Food Processing and Packaging Technology in 1996.

During the interview process, the committee asked several staff members to characterize the quality of the research program today as compared to the past. Most staff members answered this question by stating that reduced funding and the hiring freeze had resulted in a reduction in the quality of research. The committee is concerned that the business area of shelters and organizational equipment needs better science and technology support to achieve and sustain high quality R&D. The committee was told that more expertise in composite materials was needed in order to achieve world-class performance.

Assessment: Adequate to Good

The committee determined that the metrics associated with a range of adequate to good performance best fit the quality of research of the commodity directorates. The research and technology programs are aligned with customer requirements and needs; the research results are published in technical reports and journal articles; the research staff is invited to participate in scientific meetings and workshops; peers recognize several research and technology programs as being of very high caliber, if not among the best in the world; some patents have

been awarded. Nevertheless, the erosion of research staff and lack of steady funding has lowered the caliber of some research and technology programs so that overall performance is slightly less than good.

Assessment: Adequate to Good

The committee's overall assessment of the Quality Focus pillar for the commodity directorates is adequate to good (see Table 2-3). In making this assessment, the committee notes that everyone in MobD and SurD who was asked during interviews to rate quality indicated that, on a scale of 0 to 10, with 10 being the best, the quality today rates from 4 to 6, with some improvement in the past several years. Everyone in SusD who was asked to rate quality indicated that, on a scale of 0 to 10, quality today rates 6 to 8. These judgments tend to substantiate the committee's overall assessment for this pillar.

The committee's assessment reflects two perceptions of most interviewees. First, they perceive a decline in the quality of R&D resulting from retirements without replacements, the shift in technical manpower to higher paying, more visible administrative positions, and the requirement that technical people perform more contract management. Second, they perceive a rise in quality as a result of

improvements in the development and procurement process. Taken together, these perceptions reflect a positive feeling about the RDEC's timeliness and handling of short-term issues, underscored by a concern about the capability of addressing longer-term issues.

The committee is concerned about learning across the organization and the transfer of information and lessons from team to team. As the funding declines and levels off, there is no room for repeating mistakes. The committee applauds the support for individual learning, which appears to be an effective strategy for compensating for dwindling human resources. The committee is concerned, however, that the RDEC does not have internally developed quality metrics for tracking improvement in key areas. The committee is persuaded that internally developed metrics that track real improvements should be considered. Externally imposed quality programs are sometimes seen as necessary busy work.

The committee notes that, in contrast to the Strategic Vision pillar, which had several poor marks, the Quality Focus pillar had several good marks. The committee judged the vector for the future to be uncertain to accommodate positive and negative indications for the long-term.

This characteristic is difficult to assess because the list of customers provided by MobD appears to be a mailing list rather than a customer list. The panel notes that MobD was awarded a certificate of achievement for its recent contributions to Operation Provide Promise, indicating customer satisfaction with airdrop products. The guided parafoil aerial delivery system (light) is an example where MobD personnel had reduced product cycle time to the great satisfaction of the customer. The committee observed that soldier involvement and input is

emphasized in MobD. Also, almost by definition, paying customers appear to be satisfied. MobD workers were generally only moderately pleased with the way things are going in science and technology. Their biggest complaint was the lack of understanding of the responsibilities and authority of researchers. There appears to be a good deal of confusion about who the paying customer is for the development programs. The key individual appears to be the officer in charge of the Army's Airborne Airlift Action Office. Turnover in this position has created the appearance, at the MobD level, of a lack of continuity. A strategic plan for the Airborne Airlift Action Office could provide longer-term technical direction, which would assist MobD to develop a strategy for product development and associated research.

Assessment: Adequate to Good

SurD measures customer satisfaction and studies trends. Customers appear to be generally satisfied, except occasionally (e.g., several interviewees indicated there was some dissatisfaction with the body armor used in Somalia and the boots in Desert Storm). After reading a news article concerning soldier dissatisfaction, the committee contacted the U.S. Army Science Advisor for the Southern European Task Force, which was deployed to Bosnia. The science advisor informed the committee that, in concert with Soldier Systems Command/Natick, he had conducted an after-action review of equipment performance. The science advisor stated that results ''indicated a high degree of satisfaction with all of the Army equipment provided.'' The advisor went on to say that the news article "did not convey the full degree of soldier satisfaction with equipment during the deployment." He added that he has "found Natick to be extremely responsive to soldier needs and feedback" (Sprinkle, 1996).

SurD has a process for assessing the satisfaction level of internal customers (i.e., the management and work force of the RDEC). This process uses a rating system (0 to 7) for customer satisfaction, which is normally 5 to 7. Corrective action is taken if there is a rating of 4 or less. If needs are met and customers are pleased, additional funding might be forthcoming, but SurD does not have great incentives for exceeding customers' needs. The Operational Forces Interface Group in the ASCD constantly surveys soldiers and returns feedback on customer satisfaction.⁶ SurD also sends project officers out for customer feedback.

Assessment: Adequate to Good

Everyone in SusD who was interviewed by the committee recognized the importance of customer satisfaction. Most regarded soldiers as their primary customers; however, they also recognized that SusD has many other customers. Project officers frequently go into the field and conduct trials before writing requirements. Several interviewees commented that long development cycle times were a barrier to improving customer satisfaction. The Natick RDEC often shares responsibility for fielding items with other agencies, such as the Defense Personnel Support Center, and delays in procurement elsewhere often cause the customer to have a negative image of the RDEC.

The committee contacted the program manager of Soldier Support at the Soldier Systems Command during the February site visit, who indicated that some organizations within SusD are more responsive than others. The committee notes that program managers are important customers of the RDEC.

Assessment: Adequate to Good

The committee determined that the metrics associated with a range of adequate to good performance best fit the customer satisfaction of the commodity directorates. The committee found that customers generally seemed satisfied with the technical capability and performance of RDEC products, as well as with the technical support for fielded products. Customers were sometimes dissatisfied with product cycle and delivery time, but the committee found that circumstances beyond the RDEC's control often determined cycle time. Because the committee also learned of some indications of dissatisfaction with the products, it would be difficult to conclude that customers were either always very satisfied (required for good performance) or always delighted (required for excellent performance).

Assessment: Adequate to Good

The panel saw evidence of extensive involvement by MobD customers in the later stages of development and of continuing customer input. Soldier field

visits are frequently made, and nearly everyone in MobD feels exceptionally concerned about the soldier. However, some personnel in MobD noted they have never formally queried customers as to their level of satisfaction as participants in the development process.

Assessment: Good to Excellent

SurD customers are involved in the development programs, and customer involvement increases as programs progress. Advanced development work involves a great deal of customer input. The planning integration team now brings customers into the shaping of concept exploration and research to develop fundamental new knowledge (e.g., now that the research and concept exploration work is co-located with work on the end item in the ballistic protection group, greater customer interaction is possible). The ballistic armor team is getting ready to undergo review without the presence of the manager in order to encourage good input from the entire directorate. The Program Manager-Soldier also conducts periodic reviews. SurD sponsors workshops involving all the Armed Forces, the Chemical School, and the Infantry School. A survey of durability is now under way and will elicit responses from about 200 soldiers directly from the field.

Assessment: Good to Excellent

The SusD planning integration team is a good example of customer involvement in the evaluation of proposals for return on investment. SusD also sponsors customer workshops to gain feedback on its programs. The Organizational Equipment Division involves customers through joint working groups. The committee found that customer involvement is much greater during the development phase than the research phase.

Assessment: Good

The committee determined that the metrics associated with a range of good to excellent performance best fit customer involvement of the commodity directorates. Customers are involved in setting program objectives and following their progress. Customers have a major impact on the development phase of

products and services but less impact on the front-end research phase, which limits the committee's assessment to good to excellent.

Assessment: Good to Excellent

MobD's airdrop capabilities have resulted in products for the Army and other uniformed services, as well as for other federal agencies, such as the National Aeronautics and Space Administration. It may be difficult for MobD to diversify more than it has. Areas to investigate could include other customers who use parachutes. Except for the New Ventures group and individuals in special operations, no one the committee interviewed had ideas on how parachute skills and technology could be used to diversify markets. The committee explored the extent to which MobD had ventured beyond familiar markets and familiar technologies and products and tried to imagine how the technologies of MobD could be applied elsewhere (e.g., to transportation or forest service). The barriers to approaching these markets will probably come from the Army, rather than from Natick. Other government laboratories (through cooperative R&D agreements and advanced technology demonstration programs) may provide models for leveraging and extending the market for MobD's products, services, and technologies.

Assessment: Adequate to Good

SurD primarily supports the clothing and individual equipment needs of DOD, as well as of foreign military operations, including operations by NATO countries. However, SurD also supports customers outside the military (e.g., the Federal Bureau of Printing and Engraving for currency authentication; and the U.S. Postal Service as part of the Uniform Quality Control Program for clothing and footwear). These organizations are long-standing customers of the directorate who depend on the unique qualifications of SurD to develop clothing and individual equipment. SurD also tests textiles, which is more of a service than a technology application. Staff members of SurD indicated that they are attempting to identify new markets in the civilian sector. The lag in developing new markets can, to some extent, be blamed on rules that have prohibited SurD from pursuing, until recently, opportunities in the civilian sector.

Assessment: Good

The products of SusD are used not only in the military and other government organizations, both in the United States and abroad, but also in the civilian sector. Diversification is especially strong in the food technology area. The shelters business area has limited opportunities to develop products with dual uses in the civilian sector because of the stringent requirements for military shelters (e.g., shielding against electromagnetic fields and resistance to chemical and biological weapons) and the economics of the civilian tent market. SusD has had some success in marketing its services beyond the U.S. Army, however. The directorate has provided services to NATO, benchmarked its products against those of Great Britain, and been asked to analyze rations from 17 countries for United Nations peacekeeping missions. SusD was recently contracted to redesign shipboard kitchens for the U.S. Navy and is responsible for the DOD Food Program.

Assessment: Good to Excellent

The committee's overall assessment of market diversification is good to excellent for those capabilities that lend themselves to market diversification in the private sector. The committee notes that several capabilities of the Natick RDEC are specific to the military (e.g., a large portion of the airdrop program). It is not clear to the committee that devoting a significant amount of the budget to partnerships with industry and academia (part of the metric associated with a good performance level) to encourage market diversification makes sense in these cases. Other programs (e.g., food technology), however, do lend themselves to market diversification, and the RDEC has done a good job of developing external markets in these areas.

Assessment: Good to Excellent

The committee believes that, overall, customer focus of the commodity directorates is good (see Table 2-4). Directorate personnel are highly conscious of their primary customers, the soldiers, and are very attentive to feedback from the field. The committee notes that an important aspect of this feedback is the work done by the survey teams within ASCD (see Chapter 3). The committee observed

that less attention was paid to market diversification by MobD and SurD than to the other characteristics in this pillar, which is explainable in part because of the focus within the RDEC on the Army rather than on potential customers outside the Army.

The MobD portfolio reflects primarily funding and history rather than potential payoffs. Because MobD has little or no discretionary money and has not

put forth breakthrough-level concepts, it has had little opportunity to develop a product portfolio different from the traditional one. Historically, science and technology programs go on and on; they are never terminated. Fortunately, MobD has an analytical process to examine its product portfolio, the integrated planning process, which is now in its second year at the Natick RDEC. The committee is hopeful that this process will be used by MobD to evaluate priorities annually and to create a viable portfolio.

Assessment: Adequate

SurD has some control over technology-based programs but little control over engineering development and procurement programs, which are strictly focused on user requirements. Hence, SurD really does not have a portfolio of programs. Severe restrictions on discretionary funds also limit SurD's opportunities for developing a diversified portfolio. Interviewees noted that they feel a sense of urgency because everything they do is important, and therefore, everyone must work on everything. The committee is hopeful that the integrated planning process can help SurD evaluate priorities among program alternatives and help to establish a portfolio.

Assessment: Adequate

The committee found that SusD customers are involved in portfolio analysis through the planning integration team and the teams that assess proposals for technical merit and return on investment. To some extent, the SusD program portfolio is driven by funding, and not by the potential for payoffs, because of SusD's lack of discretionary funds for unsolicited R&D. The committee found that the products developed by SusD do enjoy significant customer acceptance, particularly the food and combat rations. Continued funding of the DOD Food Program and recognition by customers outside the U.S. military (the United Nations, NATO, Great Britain, and Australia) tend to support this finding.

Assessment: Good

The committee determined that the metrics associated with an adequate performance level best fit the proper portfolio of the commodity directorates. The

committee found some evidence of analytical processes being used to examine portfolios, but the majority of products are developed to meet customer needs. Portfolio analyses have not been part of the strategic planning process (see section on the Strategic Vision pillar).

Assessment: Adequate

The interviews and answers to questions suggested that MobD products almost always meet major performance requirements. The committee noted the great improvement in parachutes over time, which is evidenced by fewer malfunctions. MobD personnel know how to get their products and services out the door, but they do not know how to speed up the DOD acquisition process. A major complaint is DOD's use of performance standards rather than specifications for products. Many staff members expressed concerns about safety, added complexities, more work, and uncertainties associated with this change. The staff believes that the issues and implications of changes like this are not appreciated by those who make them.

Assessment: Good

SurD is now working on most requirements using cost as an independent variable (i.e., cost is taken into account in determining whether a requirement must be met absolutely or whether some relaxation is permissible to lower cost). Because requirements are sometimes unrealistic, SurD now aims at meeting only the most important requirements. When a new project begins, the Army Training and Doctrine Command goes out to a user, such as the Chemical School, and gets the operational requirements. When milestone decisions are made, all players discuss the results. Sometimes SurD exceeds expectations, but there is not much incentive for the directorate to do so.

In general, indications are that SurD products meet established customer requirements. Meeting customer "expectations" is more complicated. With the growth of the recreational clothing industry, soldiers can now compare Army products with commercial products. Soldiers may want some of the features of commercial items, and commercial products can often be adapted to Army needs (e.g., color, durability, and buttons that can be repaired in the field rather than zippers or noisy Velcro). One key issue is that the final user, the soldier, does not

usually pay for the product; thus weighing cost, performance, and customer expectations can be difficult. The committee notes that SurD interviewees provided two examples of customers who had complained about product performance: boots in Desert Storm and body armor in Bosnia. The committee was also told that in some cases the requirements were not well specified. In both instances, SurD was responsive to customer complaints, fixed the problems, and returned products to the field that were perceived as better than the ones they replaced.

Assessment: Good

The interviews and answers to questions suggested that SusD products almost always meet or exceed customer requirements. A majority of staff members interviewed felt that their products were very good to excellent, particularly at the prototype stage. Some noted that it was more difficult to maintain product quality once an item was turned over to contractors. In recent field tests of improved meals-ready-to-eat, the meals were well accepted by soldiers. Another SusD product, the high-pressure air beam shelter, is considered significantly better than previous shelters.

Assessment: Good to Excellent

The committee determined that the metrics associated with a good performance level best fit the product performance of the commodity directorates. Products appear to meet customer requirements, and the committee did not find any rewards in the Army system for exceeding customer requirements. Most products are perceived as better than the ones they replace. The committee did not find many examples of products that include "some pleasant, unexpected surprises," which is necessary for an excellent rating.

Assessment: Good

MobD is very responsive to the soldier's needs in the field when they are well defined. MobD has also demonstrated that it can respond quickly when

motivated by immediate military involvement (e.g., Desert Storm and Bosnia). MobD personnel have looked for ways to speed up the cycle time but, very wisely, have not yielded to pressures that might compromise soldier safety.

Assessment: Good to Excellent

SurD excels in quick responses, as illustrated in the example of boots for use in Macedonia, the soles of which were changed very rapidly; bureaucratic requirements are waived under such circumstances. In other words, evidence suggests that systems are in place at SurD to allow rapid work on emergency items in times of conflict. The committee found that SurD has discovered no "sure fire" way to cut down the time of delivery under normal circumstances. However, SurD has tried to reduce cycle time through low-rate-initial-production procurements, replacing detailed specifications with performance specifications, and eliminating unnecessary field testing. SurD programs are mostly on time and on budget, but the length of the normal acquisition process remains a problem.

The cycle time for clothing and individual items of equipment is two to seven years; product improvements take from one to three years. The directorate considers these response times too long for customer satisfaction. Once customers identify a need, they want it satisfied immediately. The fastest cycle times are for products where a commercial market has an acceptable solution or product available, thus making the directorate's task one of acquisition rather than RD&E. Long cycle times are typical of products unique to the military that have no marketability outside the DOD establishment (e.g., a new uniform for female officers that was delayed for more than seven years).

Assessment: Good to Excellent

The committee found many instances showing that SusD can respond very rapidly to unforeseen circumstances that affect forces in the field. A prime example is SusD's rapid fielding of sunscreen shelters for Desert Storm. However, some interviewees felt as if they were always "fighting fires." In normal situations, the planning cycle for science and technology is consistent with the five year planning time frame for the RDEC as a whole. Science and technology programs have a slightly shorter cycle time of one to three years. The DOD Food Program is reviewed and renewed on a yearly basis, but most interviewees indicated that they had no incentive to complete a project ahead of schedule. Most

interviewees felt that projects funded for three years should take three years to complete. The committee found no motivation for trying to complete a three-year project in two years.

Assessment: Good to Excellent

The committee determined that the metrics associated with a range of good to excellent performance best fit the cycle time and responsiveness of the commodity directorates. At first glance, one may question this assessment because there are examples of cycle times and responsiveness that appear to be less than good. The committee decided to weigh the ''quick fix'' portions of the metrics more heavily because the RDEC streamlines processes and allows personnel to perform in a less constrained environment in time of need. In those instances, the committee found the staff to be very responsive to the needs of the troops, and senior management ensures that resources are reprogrammed to fulfill quick-fix requests. The committee also observed several examples of commanders directly and indirectly expressing their gratitude for quick fixes.

Assessment: Good to Excellent

MobD is attempting to show the benefits of projects (e.g., from cost savings to lives saved) in concrete terms by participating with ASCD in the development of airdrop battlefield simulations. These simulations reflect efforts by the Army to derive hard numbers to demonstrate the value of various RD&E programs. The evaluation and prioritization phase of the integrated planning process is also used to demonstrate the value of MobD projects.

Assessment: Adequate

SurD relies on the Defense Technical Information Center database and the evaluation and prioritization phase of the integrated planning process to evaluate past and current programs. SurD projects are primarily intended to ensure that U.S. soldiers and marines are the best equipped and best protected ground troops

in the world. A secondary goal is the commercial adaptation of SurD technology. Examples are the use of ballistic protection equipment by law enforcement agencies and the use of cold-weather clothing for outdoor equipment. SurD can describe the value of its work, but only in general terms. SurD is still searching for ways to measure the value of its products, estimating the number of lives saved by ballistic protection, for example. Feedback from soldiers through comments to the Operational Forces Interface Group can help.

Assessment: Adequate

SusD maintains a quarterly database on all projects as part of the RDEC and Soldier Systems Command Planning and Reporting System. In addition, under the Government Performance and Results Act, the directorate recently implemented annual assessments, which document planning, performance accounting, and budgeting for each project. The committee found indications that all divisions of the directorate have developed products with greater value than similar products available in the private sector. Nevertheless, the committee believes that SusD needs to characterize the value of projects in terms of improvements in the performance of soldiers in the field.

Assessment: Adequate

The committee determined that the metrics associated with an adequate performance level best fit value of the work in progress of the commodity directorates. The committee found that customer perception of current RD&E programs appears to be good. Some methodology has been put in place to assess the current RD&E programs with respect to a "value of work in progress." However, across the directorates, there is no database of the primary and secondary effects of past projects that can be used for comparison with current RD&E programs.

Assessment: Adequate

The committee believes that performance of the commodity directorates is good with respect to the Value Creation pillar (see Table 2-5). The directorates are responsive to soldiers' needs, and product performance is usually good. When

deficiencies surface, the directorates appear to do their utmost to correct them quickly. Better ways to measure the true value of products would be useful to the Natick RDEC.

The committee's summary of this portion of its assessment is presented in the form of a spider diagram (NRC, 1996). The spider diagram (see Figure 2-1) shows the assessment results for the five performance pillars and indicates that the collective performance of the commodity directorates at the Natick RDEC has been assessed as good for three pillars (Resources and Capabilities, Value Creation, and Customer Focus), adequate to good for one pillar (Quality Focus), and poor to adequate for the remaining pillar (Strategic Vision). Although the assessment did not show the commodity directorates of the Natick RDEC to be performing at a world-class level (i.e., excellent in all five pillars, which is very difficult to achieve), the committee believes they are performing well. Forty percent of the characteristics were assessed as good or good to excellent across the directorates. The characteristics judged to have the best performance overall are customer involvement and cycle time and responsiveness.

Of the two pillars that were assessed as less than good (i.e., Strategic Vision and Quality Focus), the committee is most concerned about Strategic Vision, which was assessed as poor to adequate. Nevertheless, the committee is encouraged that the vector for the future appears to be turning up as the result of the recent emphasis on real strategic planning and the dedicated commitment of RDEC leadership (observed during the committee's final visit in February 1997).

Figure 2-1 Spider diagram for the commodity directorates.

The committee is concerned that the vector for the future of the Quality Focus pillar, which was assessed as adequate to good, is uncertain.

The committee is troubled that the vector for the future appears to be pointing down for the Resources and Capabilities pillar, largely because of the continuing loss of skilled personnel. Unless the direction of this vector is changed, a future assessment could find resources and capabilities at the Natick RDEC to be less than good.

Finally, the committee was struck that the assessment results were very similar across the directorates in all but one characteristic. The exception is the market diversification characteristic in the Customer Focus pillar. Some organizations were able to diversify more easily than others because of the availability of non-Army markets for their products. Not all of the Natick RDEC's products lend themselves to world-class performance in market diversification.