Management Systems: Lessons Learned Process and the eRoom Tool
LESSONS LEARNED—A MANAGEMENT SYSTEM
An important management system in any manufacturing process with high hazard potential is one in which institutional knowledge and experiences are captured and shared with all affected personnel. Management systems can be composed of both information technology elements and processes whereby human work is performed. The chemical stockpile disposal program makes use of both techniques. There are frequent and focused (usually telephone) meetings on lessons learned. These meetings involve the lessons learned coordinator for each site plus appropriate experts to provide advice/information on any particular topic. The lessons learned database was set up in 2002 and has been continually updated. The current (2010) version enables personnel to query the database to obtain information about procedures and specific topics and to search site spill history and prior closure activities. The database covers multiple years of operation and multiple facilities and would appear to be serving the needs of the program. Many management personnel, both Army and civil service, have access to this database. While similar commercial systems do exist, the lessons learned database, which was developed internally by the Army and its contractors, is able to codify and catalogue, as well as search and retrieve, needed information.
In setting up the database, the program has successfully wrestled with most of the key questions surrounding use of the database; information collection, retrieval, and sharing; worker training; and systems support and management. The database has been in use since early 2002, but the design and operational details were revised in 2006. The newer version is more database, in review mechanisms to ensure the accuracy complete in controlling how information gets into the and usefulness of the data, and in delineating to whom and what type of access will be available. In a teleconference call with members of the committee, the Army asserted that the database program is substantially improved over that in use in 2002 particularly with regard to some shortcomings identified in the earlier National Research Council (NRC) report Evaluation of Chemical Events at Army Chemical Agent Disposal Facilities (NRC, 2002).1 The Army attributed the improvement to a unified ownership of all aspects of the database by its contractor, the URS Corporation, with support from the Army. This includes keeping it up to date. The Army also provided the committee access to the most recent version of the lessons learned program.
Finding 4-1. The current version of the lessons learned database is significantly improved over the 2002 version; it is much easier to access and use the search functions. The Army is to be commended on implementing the changes that made the database more usable.
It is noted that having a good lessons learned system is a widely used operations tool in industry. The development of the current programmatic database took a more bottom-up approach than the previous (2002) version discussed in NRC (2002). With continuing support by the Army, the current version of the database has gone through several iterations to arrive at its present form. The data in the system are almost entirely specific to operational issues arising during weapon destruction at their respective facilities. Nonetheless, the data on closure, while currently limited to the three facilities that have closed thus far—Johnston Atoll Chemical Agent Disposal System, Aberdeen Chemical Agent Disposal Facility, and Newport Chemical Agent Disposal Facility—are likely to be of interest to demolition contractors, for example. The committee strongly supports the concept of continuous improvement of the existing system for programmatic data.
The Importance of Lessons Learned
In any endeavor, the experience that comes from doing offers opportunities for learning and gaining wisdom. This is true whether the outcome of the activity was good and as anticipated—or otherwise. That is to say that we learn from our past, both good and bad. Similarly, the chemical stockpile disposal program has embraced a lessons learned approach. When dealing with chemical agents and other toxic substances, protecting the safety and health of the workforce and the surrounding communities becomes an essential priority. This is true for activities spanning construction and agent disposal operations, and it extends to the closure and dismantlement of the disposal facilities. It becomes an even more critical priority as the majority of the workforce shifts from workers trained in agent disposal operations to those trained in demolition but who are less familiar with chemical agent properties and safe practices in agent issues.
Strong anecdotal evidence indicates that a lessons learned approach has been helpful in the planning, construction, operation, closure, and deconstruction of chemical agent disposal facilities over the course of the chemical stockpile disposal program.2 The committee judges that a continued, formalized, lessons learned process has and continues to significantly benefit the conduct of chemical demilitarization closure activities.
Defining Lessons Learned and the Lessons Learned Process
A lesson learned is derived from knowledge, experience, training, exercises, and actual incidents, and it reflects both positive and negative lessons. The lessons learned process can be divided into four discrete steps (see Figure 4-1):
Identify idea and articulate concepts.
Codify, catalogue, approve, and store.
Search and retrieve.
Integrate into current work activity.
Only by completing all of the above steps is the value of the prior knowledge and experience able to be fully assimilated and be useful in planning and executing a specific task or change. If any of the four steps is not completed, the objective of having a functioning lessons learned process is not fully realized. Likewise, the continuous improvement process applies to all steps. It is also important that any staff member, government, or contractor be able to easily access the data and find any lesson learned that is applicable to a particular issue. Moreover, the committee believes that a mechanism should exist whereby proposed lessons learned that are initially rejected be independently reviewed and potentially reconsidered for inclusion in the database.
The current lessons learned process flow (shown in Figure 4-2) was adapted by the committee from a more detailed flow sheet prepared by the Army. Many of the current documented lessons learned are pertinent to agent processing operations. Some of the information concerning closure lessons learned from the Johnston Atoll Chemical Agent Disposal System and associated procedures apparently exists only in hard copy and is not able to be digitally searched. However, there is a growing body of electronically searchable knowledge and experience relevant to closure, deconstruction, and dismantlement. This latter body of knowledge, the closure-based lessons learned, is the focus of the evaluation done by the committee.
This committee has focused its attention on the programmatic lessons learned process that is managed by the Army and its prime contractor. Useful data reside
in the information system and are reasonably straightforward to use, although a little training is needed. The current programmatic lessons learned program is very much improved over the earlier (2002) versions with regard to searchability, accessibility, and a more formalized process of entering the lessons learned. However, it is incumbent on the Army and its contractors to remain good stewards and to continuously improve the process.
Any staff person who has an account on the system can submit a lesson learned. There is a lessons learned submission form that is available electronically and in hard copy. Subcontractors who do not have accounts must take the extra step of raising a potential lesson learned with a prime contractor representative. This limitation could become an issue during closure activities as some new lessons learned may originate from other than Army or prime contractor sources.
Finding 4-2. Lessons learned over the course of conducting closure operations at chemical agent disposal facilities will be helpful to completing without incident future closure activities within the chemical stockpile disposal program, and they will minimize costs by reducing the time and effort needed for learning curves and training.
Recommendation 4-2. The Army should continue to support the closure lessons learned processes and to encourage the prime contractor for closure operations to strengthen the timeliness and manner in which the lessons learned are shared. In this regard, it is important
that all contractors on-site have access to or knowledge of the lessons learned applicable to their specific site activities.
Each site has a lessons learned coordinator who receives the submitted lessons learned forms and logs them into the system for review. This coordinator ensures that the data are complete and assigns each data set a lessons learned number. At this point, one of three outcomes is possible: the lesson learned can be rejected; it can be accepted and entered into the system; or it can be reworked at the initiating site and then forwarded.
Because each site has a lessons learned coordinator, system variation is introduced by having different people exercising judgment. However, terms and categories for the system are preprogrammed into the software, which reduces variability. The title for a lesson learned is at the discretion of the submitter and is free form. The submitter is also allowed to categorize suggested priority ratings for timing and safety. If site managers determine that a lesson has imminent relevance to safety, an email is generated and uploaded to the database. This determination of a high priority requiring immediate notification of the other sites is made by the lessons learned coordinator and subject expert at the site.
The site coordinator and other appropriate personnel verify the uniqueness of the proposed lesson learned and, if appropriate, recommends inclusion in the database. An email accompanies any lesson learned that is forwarded beyond its originating facility. The proposed lesson learned is subject to further review, and appropriate actions (if any) are conducted.
Once the lesson learned is forwarded from the site, it is subject to external review by a program-wide subject matter expert who reviews and approves, or reviews and issues, the lesson learned for information only. A lesson learned issued only for information indicates that no specific action is required by any facility. This review process at the program level allows for greater consistency within the specified subject area. Coordination among the subject matter experts is vital to ensure consistency in lessons learned treatment across subject areas. The subject matter expert is the person who can revise or alter a lesson learned entry during the review process, as well as being qualified to make a determination that the lesson learned is only for informational purposes.
There is no documented appeal process in place if the submitter disagrees with the decision at the site level by the lessons learned coordinator or site subject matter expert to reject the lesson learned. The individual(s) who submitted the lesson learned may disagree with the disposition decision and should have an opportunity to document this position and make a case for inclusion. There does not appear to be an opportunity to question this rejection. Rejected submissions should be reviewed independently from the initial review to ascertain that in fact the lesson might not be useful to, for example, a demolition contractor. While the committee has not seen evidence of serious problems with this part of the process, the inability to capture what information was discarded at a given point in time is not optimal.
The search mechanism for the lessons learned database is significantly improved over the 2002 version. However, the novice or inexperienced user may not have adequate ability to conduct a search without help or extra training. It was not apparent to the committee that novice users are sufficiently familiar with the search functions. This may become a more serious issue during closure when most of the activities are not typical of the more standardized activities that take place during agent disposal operations.
Finding 4-3. There is no system in place to review a determination to reject a proposed lesson learned. Rejection may become a more important issue during closure than it is during operations because the review system is not geared to closure. The current system depends in part upon knowledge of how the search mechanism is constructed and upon use of the appropriate search words or terms.
Recommendation 4-3a. The Army should require a mechanism to validate the decision to reject a lesson learned.
Recommendation 4-3b. The Army should require implementation of a means to familiarize people with those paper-based lessons learned from the experiences at the Johnston Atoll Chemical Agent Disposal System that are not accessible through the electronic lessons learned database.
Recommendation 4-3c. The Army should consider developing a real-time user support tool to help novice users search the lessons learned database.
In addition to the searchable electronic database, the Army and its contractors participate in weekly teleconferences and conduct quarterly meetings on lessons learned relevant to closure preparations. These activities serve to make the preparatory efforts directed toward closure lively and current, and attendance has been high. The ability to query an expert in real time is another excellent way to ensure that lessons learned deliver the benefits intended.
Access to the Lessons Learned Database
There are two levels of access to the database, one of which is widely available, and another that in addition to access allows data input/changes, but is not widely available. It appears to the committee that the current system operates in a manner that could inhibit a potential user (particularly during closure activities) from correctly locating an applicable lesson (assuming it is present). The Army should consider how the closure lessons learned information could be made available to all potential subcontractors during the bidding phase for particular closure tasks. Such information may be pertinent to all bidders, only one of whom will be selected.
During actual closure operations, a different set of contractors will be on-site and a very different set of problems may arise. There appears to be no current means of ensuring that the lessons learned will be accessible during closure and/or the knowledge contained in the database will be made available to appropriate subcontractor personnel. The information in lessons learned documents can provide a firm foundation to facilitate safe, fast, and cost-effective closure operations, but the information must be readily available.
Currently, although a detailed database does exist, it may be somewhat difficult for an inexperienced user (such as any closure contractors) to access or obtain the pertinent information contained therein. This has not seemed to be an issue during chemical agent destruction operations, but as indicated above, it might be more problematic during closure. At present, management holds periodic meetings and phone calls to share lessons; however, during closure, that approach could suffer from unfortunate timing, and it may require participants to have good memories. A user who is looking for closure information to, for example, prepare a work plan for occluded space surveys might have difficulty finding any appropriate information in the database as currently configured. The Army should consider implementing a more proactive system by which information is immediately pushed out to users who are specifically notified when a lesson learned is approved in their area or for their facility. This will become increasingly more critical as closure activities accelerate. Such a system could have subject matter experts taking a greater leadership role in the process, for example, by checking the type of data a user is seeking and ensuring that the user is able to find all pertinent information. One of the key aspects of lessons learned is their value as an appropriate “just in time” tool. A lesson learned too early can be lost and forgotten, and a lesson learned too late may be disastrous for individuals and the program.
Finding 4-4. Since the number and type of contractors on-site will differ during closure and agent disposal operations, the use of the lessons learned database and its applicability may be different during closure operations from what has been the case previously.
Finding 4-5. The lessons learned database is searchable, but the search mechanism is relatively difficult to use by the novice user.
Recommendation 4-5. Rather than relying completely on the current means of searching the lessons learned database system, the Army should develop a proactive mechanism that assists new or novice users, particularly dismantling subcontractors to find, or be made aware of, the data in the lessons learned that would apply to a particular problem.
The contractor maintains an eRoom, an electronic repository of documents related to the chemical stockpile disposal program that includes closure-related documents, permit-related documents, and documents relating to operational matters (see Box 4-1). However, there does not appear to be a very strong in-place training system to familiarize all appropriate employees (including those primarily involved with closure) with the use and benefits of the eRoom. It is potentially a very strong management tool. Other companies that have a similar system typically find it necessary to devote a considerable amount of time and resources to ensuring that it is used to the fullest possible extent. The Army and its contractor might evaluate whether the current training is adequate, and whether the use of the eRoom could be strengthened to benefit closure activi-
Description of the CMA eRoom
An eRoom is an electronic space established by project management to enable members of a team selected by project management to collaborate and share information pertaining to work-in-progress. This is accomplished by making project information available to the team members for reviewing, copying, commenting upon, and possible editing irrespective of organizational affiliation or geographic location.
Project management selects a member of the team to establish, implement, and coordinate eRoom activities. This team member is referred to as the coordinator. There may be more than one coordinator for an eRoom.
A coordinator, with IT support, adds other team members to the eRoom membership (list) at the request of project management and, with management guidance, assigns one of three possible roles to each member.
Team member roles are that of observer, participant, and coordinator. Each role provides different levels of functionality within the eRoom. An observer may view and copy contents located within the eRoom; a participant may view, copy, add, and modify contents; a coordinator may view, copy, add, modify, and delete any content. A coordinator may also modify roles and access permissions to content for team members. The coordinator monitors the eRoom for usage and periodically consults with management as to whether access to the eRoom by any individual should be maintained or terminated.
The eRoom content primarily consists of files, folders, and objects, including audio and video. A team member who has been assigned a role which would enable them to add content may either drag and drop or upload content from another location. The team member who adds the content becomes the “owner” and may specify at the time of content addition which other team members may view the content, are identified as co-owners of the content, and may edit the content. The team member may also send an email alert to individual or multiple team members to advise them of the content availability.
The eRoom incorporates additional functionality which may be deployed by the implementing organization. The Closure eRoom was established to disseminate programmatic information to stakeholders and to exchange information from the various sites for enhancement in the development of site-specific documents.
SOURCE: Rafael J. Gramatges, Specialty Group Manager, URS Corporation, March 29, 2010.
ties. As an example, the wording of specific documents such as sampling strategies and permit language at different sites could be reviewed for internal consistency before submission to a regulatory agency.
An individual must ask for access to the eRoom. If an individual has not used the eRoom within 60 days the access is canceled. An individual may be granted access to portions of the eRoom (for example, closure-related topics only or monitoring-related topics only). The committee was told that this security protocol is needed because the eRoom is URS company-wide and not restricted to Chemical Materials Agency activities only. For this reason access is limited; a change would require action at a high URS corporate level.
Typical screens for the eRoom show who has access to the room, what role the individuals have in the organization, and where their offices are physically located. From the screens the committee reviewed, it is apparent that a relatively small number of individuals actively use the eRoom. In order for the eRoom closure lessons learned to be an effective management tool, the room would typically need to be used by a wide variety of people, and during closure. For example, selected portions might need to be made available to subcontractors since they will be a key part of the closure process.
One eRoom screen shows the details of how each topic area within the eRoom is addressed. For example, a screen on the topic of coordination and collaboration outlines when the eRoom was created, who the main contact is, key news and information items for the users, and a legal disclaimer about privileged information. This is followed by a listing of all the files, documents, and training materials deemed pertinent to the subject area of closure.
Another eRoom screen outlines the history of document retention and change. This is extremely helpful when determining the age and relevance of information. The records retention page lists the document’s name, the date it was modified, the name of the document owner, and the document’s size.
The eRoom is a potentially very powerful tool both for coordination purposes during draft markups or for someone looking for information on a particular topic (for example, a Resource Conservation and Recovery Act permit and its contents). The use of the eRoom appears to be relatively widespread, but relatively few
individuals are heavy users. The committee noted that the closure managers at their June meeting consistently referred to the eRoom and often asked for certain specific documents to be uploaded so that all sites could have access to them. As with the lessons learned, use of the eRoom is for the most part from the bottom up in terms of personnel. There appears to be less indication of a proactive use of the eRoom as a design tool or by the Chemical Materials Agency management as a means of promoting consistent sets of information among similar documents. The eRoom would perhaps be more useful if the contractor had a system that was more aggressive in “pushing” information in the documents to users. The concept of timely access to lessons learned was described above, and this same concept also needs to be considered in any use of the eRoom during closure, particularly by subcontractors active in closure activities.
Finding 4-6. The eRoom is a very powerful information sharing and management tool, both for developing new documents and for allowing users to find information that is pertinent to a particular issue or problem.
Recommendation 4-6a. The committee strongly supports the concept of the eRoom and encourages its use as often as possible.
Recommendation 4-6b. The committee suggests that the Army and its contractor examine current eRoom usage and, if appropriate, develop procedures to increase its usage, including the development of new documents and determining who should have access during closure and dismantling activities.
NRC (National Research Council). 2002. Evaluation of Chemical Events at Army Chemical Agent Disposal Facilities. Washington, D.C.: National Academy Press.