Information Management Systems
Information management is an important aspect of managing obsolescence at stockpile disposal sites because it facilitates tracking the history of changes in the facilities and personnel over the lifetime of operations. Data of this sort can aid in identifying areas of potential obsolescence, facilitate planning, provide a permanent record that documents operations, and track abnormal incidents and trends. Information management facilitates communications throughout the site, among the sites, and programmatically. However, since information management systems too may age, become outdated, or become no longer supportable or accessible, they are also subject to obsolescence.
Information is created during plant operations in a number of ways. Human resources systems, process control records, inventory and transaction management activities, and other sources of data collection are all based on computer software, or applications, that generate digital electronic information as a product. Specific sources include word processors, laboratory analytical software, spreadsheet software, electronic inventory archives, and related databases. The generated information must be maintained for periods of time dependent on the nature of the content and on regulatory requirements, program stipulations, or prudent management practices. For this to happen the information that is generated must be identified uniquely (so that it can be recognized at some point in the future), be collected at its point of origin, then be transported to data storage facilities and stored in an appropriate medium, and it must be accessible at some arbitrary or predetermined point in the future.
The nature of information management, and the volatile characteristics of the industry that supports the software, transmission facilities, computational systems, and storage media used in information management systems lead to challenges related to obsolescence when the operating programs that generate the information continues over a protracted period of time (see Appendix B).
The committee conducted a thorough review of the information management systems at the incineration facility sites. This is an area that has not been thoroughly reviewed as a whole in the past by an NRC committee and includes physical facilities (servers, desktop units, storage, and distribution), software (operating systems and applications), and system development processes. The committee’s major concerns in the information management area relate to how it is being managed inconsistently within and between sites. These issues are discussed in the following section. Later sections address specific observations concerning the information management and technology systems currently in use.
PLANNING AND DECISION MAKING
The process control systems at chemical agent stockpile incineration facilities employ information technologies that have of necessity been managed as part of the integrated processing system. However, a history of plant operations is stored in a process data acquisition and recording system (PDARS), which is part of the control system for the plant but also of the larger information management mission of each site. In the course of the committee’s data gathering, some of the experts in the control system technology at the sites were consulted to provide general advice on the general elements of the information management systems.
There is a disparity in the way planning and decision making are approached at the various facilities. This is a consequence of the way information management processes are implemented. At ANCDF all aspects of information management governance are now coordinated centrally or are moving in that direction. At the other sites examined, information management governance is split into several areas, and coordinated in a loose collegial manner. At TOCDF, for example, the management of the new control subsystem for mustard agent processing will be assigned to the plant information systems department. The management of the exist-
ing control system will remain in the control engineering department.
At most sites personnel responsible for the office, laboratory, and technical functions, and in some cases human resources and other functions, make decisions on information technology and information management systems independently, although meetings are held to discuss needs and requirements on what is a locally determined basis. ANCDF is distinct from the other sites in the degree of central control exercised on information management decision making; the ANCDF site also has a level of autonomy on information technology and management decisions that involves little consultation with or guidance from the other sites or CMA.
This variability in governance is not in itself necessarily a threat to continuing operability, except to the extent that a lack of concerted purpose can lead to increased costs and decreased interoperability. However, it does raise questions. For example, ANCDF personnel made the decision to move toward a single-link fiber optic facility network backbone in place of the original double-loop wired system, based on their interpretation of needs and opportunities. The cited rationale for this was the greater reliability of fiber optics, which was interpreted to mean that a single fiber optic link was more reliable than a double-loop wired system. This is true from one perspective, but not necessarily so when considered from the perspective of a mechanical failure—as might be associated with an unplanned ordnance event. Whether that scenario is likely and whether the decision to change the network topology was correct is not argued here. What is noted as significant is that no perceived need or interest in consulting at a higher level on this decision was evident; the decision was entirely locally determined.
Another example of the variability in problem-solving approaches is the move to different software in the control room operating consoles. There are now two major systems in place among the sites in this regard rather than one, and there are local variations in how those systems are enabled. This is a questionable practice from a performance and reliability perspective. Again, it is not the end result that is being questioned here, it is the practice of devolving decision making on issues that affect uniformity and therefore inevitably reliability and interoperability. In the long term in the context of stressed resources and an extended operating period, such practices will tend to gravitate toward less effective programmatic results than when standards are regulated in some central way. This issue is dealt with in more detail in Chapter 5 of this report.
Finding 12. The committee found no central or unified approach to identifying information technology solutions and implementing information technology changes at chemical agent stockpile incineration facility sites. This lack can lead to an erosion of compatibility, increased costs, a reduced potential for interoperability, and other challenges to continued operation.
Recommendation 12. The Chemical Materials Agency should implement a mechanism to coordinate and formally demand consensus in areas of information management where joint operations between the chemical agent stockpile incineration facilities are appropriate. Such mechanisms should be developed, implemented, and reinforced for the remaining life span of the chemical agent stockpile disposal program. (Tier 2)
DETAILED OBSOLESCENCE ASSESSMENT OF DISPOSAL SITE INFORMATION MANAGEMENT
This section provides more detailed information primarily relevant to specialists implementing information management programs at the disposal sites.
The scope of information management in this discussion is restricted to those elements that function in a supporting role to the actual processing operations. Other aspects of information handling, such as the instrumentation and computer control systems that are directly associated with plant operations, are not covered by this definition and are dealt with in Chapter 2. Also not included is information that may enter the plant from external sources, such as the Internet, e-mail, or physical storage media that could be transported onto the property, excepting updates to applications and other elements of information management discussed in this section. This latter group may have security significance or be relevant to operations in other ways, but are not fundamentally related to the issue of continuing operability. Similarly, the fate of information transported off the facility sites and out of facility control may be of significance, but is considered separate from the focus here on continuing operability.
Readers should note that many of the issue areas that are identified are presently managed at a local or site level in the facilities investigated, but would more properly be dealt with at a CMA level as discussed in an earlier section. The judgments made in this investigation are based on the premise that reliability is a governing requirement for the facilities being addressed. While it has not been possible to exhaustively review and eliminate all possible failure modes, this evaluation does recommend steps to eliminate substantial process, equipment, and software shortcomings that in the context of a continuing operation could or will threaten secure and reliable system performance.
Overall, the facilities visited varied in form and detail, but all were effective in their ability to deliver services at this time.
At a minimum the observed server hardware was rela-
tively modern, and interviews with personnel indicated that hardware at this level was being replaced as needed and was not constrained by budgets or other limitations. Uninterruptible power supplies and redundant storage were in place in all cases, and although Halon or other active nondamaging fire suppressant apparatus was not in place, the facilities themselves were secure and fire resistant.
Finding 13. The server systems at chemical agent stockpile incineration facilities in their present physical state do not constitute a threat to the continuing operability of the facilities as long as budgets and management procedures enable the progressive updating and replacement of systems as needed.
Recommendation 13. Continued vigilant monitoring and maintenance of servers, based on adequate funding and management of core capabilities, is a mandatory element of the continued operability of chemical agent stockpile incineration facilities and should be ensured across sites under guidance from the Chemical Materials Agency. (Tier 3)
User PCs varied widely in their degree of capability. Some units were current, but some were very old (in PC terms). The cause for this disparity seems to be related to intended functionality. As reported during interviews, hardware for completing common office tasks could be updated when needed. In contrast, hardware associated with technical activity was in some cases less easy to update because of its association with software or requirements that are static. The latter situation has resulted in older (in some cases much older) hardware still in service than is commonly encountered in professional operations today.
PC hardware that is not current does not constitute a direct threat to continuing operability as long as it is still functioning within operational tolerances since it can be readily modernized provided that external funding or procedural constraints to upgrades are eliminated. Nevertheless, the cost of maintenance will increase and support will become more difficult as older hardware ages even more. The problem is that if older hardware is being maintained because of the need to continue using an older version of an operating system, and that hardware then fails, it may be difficult to find a platform on which the legacy operating system will run. Rather than being approached as a balance between preferred PC replacement practices and other needs, PC maintenance is in some cases subordinated to peripheral requirements. In summary, the committee believes some inconsistency exists in how PCs are maintained.
Finding 14. While personal computers (PCs) dedicated to typical office applications are generally kept relatively modern and up-to-date at chemical agent stockpile incineration facility sites, certain other PCs in use in laboratories, because of their linkages to dated analytical facilities, are out-of-date to the point that their continued use is problematic.
Recommendation 14. The Chemical Materials Agency (CMA) should devise and implement a life-cycle replacement program for all PCs. Consideration should be given to setting a maximum life span for PCs (e.g., three years) and replacing older machines with current hardware according to a predetermined service cycle. The CMA should specify that when a PC has been retained beyond a reasonable lifecycle expectation because it was required to support dated peripheral devices, software, or other features that are themselves substantially dated, alternatives to those peripheral items should be identified and if possible acquired so that the overall system can be updated to current standards. (Tier 3)
Storage and Backup Facilities
Details regarding storage and backup of data varied among facilities. However, there was general consistency in that (1) local storage was on digital media secured in a fireproof area outside or apart from the server room, (2) long-term storage was in a secure off-site location (electronically and physically), and (3) the transaction process was adequate in terms of methods and record keeping.
The digital format in which information is stored was not verified directly, but the committee was informed that data are stored in a format that is recoverable in the long term. The available space in those facilities was reported to be substantial in terms of foreseeable requirements. Therefore, the issues of continuing operability that may arise from the physical facilities are not expected to be affected by the protracted life span of the chemical stockpile disposal program unless at some future point the available space is exhausted or requirements change.
The physical storage facilities for electronic information media do not constitute an immediate threat to continuing operability. However, it would be prudent for CMA to conduct periodic review of and continued maintenance of storage facilities to ensure that they remain effective.
The electronic links between buildings within the facilities examined were found to be varied, and have been changing over the life span of the facilities. For example, at ANCDF, the older twinned facilitywide communications backbone loop has been replaced by a single fiber optic cable, whereas at TOCDF, the original system remains in place.
Significant issues related to continuing operability are not presently apparent from this aspect of the physical sys-
tems, but the current situation could eventually indirectly reflect on the robustness of continuing operability.
Commercially Available Software
Over 100 different commercially available applications were identified as being managed on computer systems in the facilities visited. It is convenient to consider these in several basic categories, namely,
electronic reference software;
computer maintenance and management software; and
program planning and management software.
Although there are differences in detail among the members of each category, the various individual applications have general commonalities in terms of maintenance and continuing operational requirements. The sections below discuss each of these categories in that light.
The operating system defines the backbone functional environment of each computer, and constitutes the envelope within which each user operates each software application on each piece of computer hardware. Examples of the operating systems used include Microsoft Windows (98, NT 4, 2000, 2000 Server, 2003 Server, and XP) and Linux (Red Hat and Slackware).
Continuing Operability Issues. The range of variability in operating systems is not ideal. Not all versions being used are current, and the heterogeneity found at the sites examined increases the complexity of support and maintenance. Operating systems inherently define system security, stability, and maintenance requirements, and in some cases, the suite of applications that users can run. They are seldom chosen or fully exploited by end users, but have profound consequences for the managers and users of information management systems alike. It is known that efforts are being made to eliminate older versions of operating systems, but the process of upgrading these systems will be an unending one as it is in any such operation. Of particular note are impending releases of Microsoft products in two dimensions. A substantial change in operating system was slated for release late in 2006, and a major initiative to simplify and facilitate system management is targeted for 2007. The continued development of Linux as an accepted and secure alternative for servers and desktop systems may also have implications for operating system management over the remaining life span of the facilities. A counter pressure to the continual updating of operating systems is the presence of special technical software that requires a specific version of an operating system (e.g., Windows 98) that would otherwise be retired.
Finding 15. A wide range of operating systems exist in the chemical agent stockpile incineration facilities, and this variability could pose problems for effective long-term continued operability. At the least, costs and maintenance are complicated by this diversity and apparent lack of integrated planning.
Recommendation 15. The Chemical Materials Agency should conduct an overall evaluation of security requirements, maintenance implications, and impending evolutionary changes in the basic computer operating systems (Windows and Linux) used at chemical agent disposal facilities. A migration path that drives toward a minimally heterogeneous and maximally robust environment should be identified and considered for implementation. (Tier 3)
In general, office applications are used by workers at all levels, and have little special technical significance. They do provide workers with the basic capability to develop electronic documents and therefore communicate effectively. They are also the applications in which most document-level communication (memos, reports, and accompanying illustrations) are developed.
A wide range of office applications exist at the facilities. These include:
Adobe (6.0 Pro, 6.0 Standard, 7.0 Pro, 7.0 Standard);
Roxio Easy Media Creator 7;
Word Perfect Office (11, 12, 2002); and
Microsoft Office (97, XP, 2003 Pro).
Continuing Operability Issues. In general, the heterogeneous deployment (both in vendor and version) of office applications has only moderate consequences from a continuing operability perspective. Vendors will typically limit support to legacy systems, and interoperability may suffer somewhat, but these can readily be surmounted by upgrading to new versions. Some software variability is caused by specific needs of the facilities (e.g., Word Perfect maintained because of a document limit inherent in Microsoft Word). Learning curves are minimal and commonly are collegially communicated if staff members do not already possess competency. It is considered unlikely that the variability of office software will cause more than nuisance issues over the life span of the chemical stockpile disposal program. However, vendor data formats are not always perfectly inter-
changeable, and in the long term, it may be that records stored in a proprietary vendor format will be unrecoverable.
There is also no consistent mechanism to evaluate the implications of changing software applications, or the impacts that might propagate through the system as a result of those changes. When any office application is retired, conducting a systematic review of content developed using that application by any site affected by this change is among steps that can be taken to ensure that such content is in a format that can be dependably recovered in the future.
Finding 16. A variety of data formats are used in different contexts in the chemical agent stockpile incineration facilities, and the prospect of long-term records retention and recovery is complicated by the resulting variability of native data formats.
Recommendation 16. The Chemical Materials Agency (CMA) should consider formally requiring that each copy of an electronic document requiring long-term availability be preserved in an agreed permanent or semipermanent form defined by the CMA (e.g., ASCII or portable document format). (Tier 3)
Technical software is used by professional staff to conduct various job-related functions. In contrast with office applications, this software is typically specialized to the extent that only a subset of the staff will use it or know how to use it. Some examples of the wide range of technical software solutions used include:
laboratory support software (Deltek GCS Premier, LabView, Honeywell Loveland DocuMint, Omega ME LIMS, ChemStation);
computational software (MathCad 2000 and 2000i, SAS, Crystal Ball);
Computer-aided design and drafting software (Alibre, AutoCad 2000 and LT 2000, Microstation, SmartSketch);
programming software (Microsoft VB6, Visual Studio.NET 2003, 2005, Visual Source Safe);
database environments (Oracle 8i and 10g, Microsoft SQL Server 2000 and 2005); and
numerous other products covering topics such as human resources support, pipeline design, and thermochemical analysis.
Continuing Operability Issues. These applications inherently tend to be user preference driven, in part because of the learning curve associated with their effective application. As such, a drive to a uniform deployment model is complicated and possibly negated by user competencies. Hence, although there are benefits in principle with enterprise approaches to technical software, the status quo may already constitute an effective solution. Nevertheless, there are data interchange issues that emerge in the long term when multiple applications are used as solutions for the same technical purpose. Not all computer-aided design and drafting packages, for instance, are equally adept at importing competitor data formats. Also, users of one database environment may be uncomfortable or incapable when they encounter a competitive product. Therefore, measures that reduce the range of alternative products with common competencies are desirable.
A wide range of technical software that supports engineering, scientific, human resources, and other users exists in the facilities examined, and this could complicate operations in the long term if version control is not exercised. Lack of central control provides an environment conducive to this problem, and it appears to be emerging now.
The CMA might consider conducting a comprehensive inventory of technical software on a programwide basis, including the location and nature of each product. Thereafter, the CMA could require that before any facility elects to change the version of a product or introduce a new product, a programwide impact analysis be conducted to determine (1) whether an existing alternative can serve the purpose of the intended purchase, and in the case of a version change (2) whether all versions in the system should be upgraded to a common version.
Electronic Reference Software
Electronic reference software applications provide source data used in technical operations. They are typically externally developed and accepted as authoritative by the purchasing firm. Where employed, they are important resources for specialized technical users. Software in this group included the 2000 International Building Code, Plumbing/Mechanical, Akton Psychrometric Chart for Windows, and AHFS First. The first two are self-explanatory; the third is a pharmaceutical database.
Continuing Operability Issues. Content can typically change over time, rendering old versions of vendor supplied data obsolete. Since users commonly apply content extracted without verifying that the source is up-to-date, these references must be maintained in current and up-to-date states in a formal and managed program. It is noted that the cited version of the International Building Code is out of date (2003 is current, and 2006 is pending).
Some reference databases exist within the facilities examined, and they may not all be current. This can increase costs and threaten collaborative interaction and quality control, all of which are problematic in the context of continued operability. A strategy to mitigate this involves implementing a mechanism that will determine acceptable practices in technical software management. Included in this are the following steps. First, an exhaustive inventory of all site tech-
nical reference databases is conducted. Second, opportunities for economies of scale are evaluated by considering acquisition systemwide. Third, maintenance and management requirements for each reference system are determined and appropriate programs implemented to avoid long-term erosion of capability.
Computer Maintenance and Management Software
Computer maintenance and management software includes a wide range of facilitating products that have been developed to help information technology system managers maintain their facilities in a secure and stable state. Examples include applications such as BackupExec, DriveImage, LANDesk, Restorer 2000, Trend Office Scan, True Image, WebSense, and WhatsUp Gold.
Continuing Operability Issues. In general, the point of this group of applications is to maintain systems and ensure that they are up-to-date and stable. It is crucial that this kind of competency be maintained if the information technology and information management systems being evaluated are to achieve the goals set forth for continuing operability. That these kinds of applications were found and are being used speaks well for this element of the program. It may be that the evolving plans of Microsoft and others will affect the preferred suite of support applications, but until that happens the practice noted in this evaluation is effective.
A variety of applications are used to facilitate information system management. This is a positive factor, but one that is not uniformly pursued across all site systems. The variability implies costs and behaviors that are undesirable in a context of continuing operability. The continued use of the maintenance and support software encountered during this evaluation is encouraged.
Finding 17. Service and support capabilities in the information management sector are continually improving.
Recommendation 17. Annually or biennially, the Chemical Materials Agency should survey current information management maintenance options, determine whether costs and benefits in the systems under consideration are consistent with current best practices, and require changes in practice programwide where improvements in reliability or reductions in cost are identified that can secure continued operability. (Tier 3)
Program Planning and Management Software
This group of applications is used in planning and managing enterprises or projects at various levels. Included in this group are project management software (Microsoft Project and Project Server, Primavera P3/EC, and Suretrak), document management software (Papervision), asset tracking software (Intellitrack, MAXIMO, TRACS), and facility maintenance software (ValveLink, PHA-Pro).
Continuing Operability Issues. For the most part the issues related to continuing operability in this set of tools are related more to whether they are used than to their inherent characteristics. Comments on these are therefore for the most part reserved for other elements of this report. It is noted, however, that the existence of these tools implies the competences to use them and that this is a positive indicator where continuing operability is concerned.
Software development is a key aspect of long-term operability, because it relates to some underpinnings of record keeping and process management. An example that is being effectively pursued but is nonetheless faced with substantial challenges can be found in the software used for parts warehousing at ANCDF. This problem is discussed earlier in Chapter 2 and is the basis for Finding and Recommendation 3.
A Microsoft Access product developed by local staff demonstrates an exemplary attempt to manage parts inventory to best effect. The product developed is substantial, and a cursory exploration of its structure suggests that it is technically well founded if not perfect. The product has an inventory of over 5,000 parts and has recorded over 7,000 transactions accumulated since 2001. Certainly, this type of product will help manage scarce resources to best effect, and will be a major asset going forward. However, there are some aspects of this that are not clearly defined. These include the following:
No universal parts classification exists, which means that a method of reconciling parts identification schemes used at different facilities needs to be developed. A photo-image approach, an index mapping scheme, or other methods can resolve this, but such has yet to be defined.
Data entry is manual, which means that errors can creep into record keeping. Bar codes, radio frequency identification (RFID), or other methods can address this, but these have yet to be evaluated and implemented.
The software in question has no outside review, which means that it is likely effective for the immediate purposes of its developers, but may be missing opportunities to serve wider needs with efficiency.
In short, there has been an effective attempt to provide a solution that will assist in delivering a pivotal requirement for continued operability, namely, the effective location and management of parts inventories, but at present this is forced to work in a local and isolated way. This situation does not
guarantee that applications developed in this way will have the staying power to support operations in the longer term. Although the foregoing text focuses on a specific case for illustrative purposes, other examples of this syndrome were detected. The significance of this is that it suggests there may be aspects to software maintenance and development that could benefit from a wider review of requirements and solutions.
There are useful efforts and capabilities to adapt or develop software at the various sites, and it is capably implemented at a local level, but there is no universal needs assessment or quality control mechanism in place for this function. This has the potential to engender results that are not effective in the long term.
A mechanism to provide some assurance that software development in the long term will not lead to operating difficulties as a result of uncoordinated standards and solutions could be beneficial. Options to approach this might be a more centralized management system, an agreed approach to standards setting, or other alternatives. Similarly, it could be beneficial if a formal as opposed to collegial approach to software development were implemented.