Evaluating Cultural Resource Significance: Implementation Tools (2005)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

14 CHAPTER 3 DESIGN AND TESTING OF PROTOTYPES INTRODUCTION As noted in Chapter 2, the focus group and the NCHRP panel recommended advancing the development of two pro- totypes: the HPST and ECREL. In order to finish the design of the prototypes, it was important to understand the ways in which the prototypes might be used within the existing resource evaluation process. The Business Case document (Appendix A), originally developed for the focus group, pro- vided background information on this process. The process that CRM professionals normally use (or should be using, based on the Secretary of the Interior’s Standards for Historic Preservation Planning and National Register guidance) was analyzed in the Business Case. The goal of the two proto- types is to help CRM professionals determine whether or not a property is eligible for listing in the National Register. By focusing on how the two tools might be used to streamline and improve the existing evaluation process, the design team was more likely to create usable products. Table 2 describes the current process and shows how the two prototypes might be used to facilitate and improve the evaluation process. It is important to note that no attempt was made to re-engineer or improve the current business process. The reports created from the HPST are not intended to replace National Register nomination forms. The HPST reports are to be used to communicate the results of an evaluation among consultants, DOTs, SHPOs, and other Section 106 consult- ing parties. OVERVIEW OF DESIGN AND TESTING PROCESS The general approach used to create the two prototypes was design, develop, verify, and validate. The steps for each are listed below. Design 1. Identify a state that will participate in the design process. Work with URS CRM professionals, IT professionals, and a state CRM professional to design each prototype. 2. Identify the state(s) that will evaluate the prototype, and secure their agreement to participate. 3. Develop the design specifications, expanding the basic requirements collected at focus group and NCHRP panel meetings. Develop 4. Develop the evaluation criteria and test plans (i.e., the surveys or other instruments that will be used to evalu- ate the prototype). 5. Develop the prototype, and test the prototype using URS team members (i.e., URS and SRI Foundation personnel). Verify 6. Develop a draft user guide, and have the state CRM pro- fessional from the design team test the system (i.e., acceptance testing). When the CRM professionals (from URS and the state) have signed off on the prototype, the verification process is complete. The verification process certifies that the design requirements have been met. (A verification and validation plan was developed as part of the design process [see Appendix B].) Validate 7. Deliver the final products to the test state(s), providing an evaluation form. 8. Have the test states evaluate the prototype and send their responses back to URS. The specifics of how each step was executed for each of the prototypes are described below. During the design and validation stages, it was difficult to obtain the hoped for level of state involvement. Several focus group members provided some input into the ECREL design process, but the HPST design was completed only with URS staff. In large part, this occurred because ECREL is web based, allowing the design team to display interim products over the web for review. The HPST is a desktop system and requires that the design- ers be in the same place to work with the developers. In order

15 to keep the project moving forward, URS relied solely on in- house CRM professionals for the initial design and verifica- tion stages of the HPST. ECREL Design The basic requirements for ECREL were defined by the focus group, and additional requirements were added by the NCHRP panel. The index fields, valid values, other functional requirements, and business rules had to be defined. The soft- ware and process for converting and loading documents also had to be selected. The URS design team developed a complete list of require- ments and technical specifications. The ability to search doc- ument contents for a word or phrase was determined to be the most critical element of this application. Most of the source documents existed only in hard copy and needed to be scanned into a searchable format. Moreover, many documents that had already been scanned existed only in image format (i.e., they are not full-text searchable). Therefore, the design team had to (1) develop a method to convert all documents into a searchable format and (2) select an implementing technology that supported full-text searching. To accomplish the above two tasks, URS chose to scan the documents into PDF searchable-image compact format. This method retains the original look of the document, but reduces the electronic size as much as possible for faster response TABLE 2 Possible uses for the HPST and ECREL Current Process Used by CRM Professionals A property (site, building, structure, object, or district) is to be evaluated for National Register (NR) eligibility. Find a historic context that contains predefined property types that best represent the resource to be evaluated. Contexts should have predefined property types that have already been linked to the National Register criteria. For example, property type X represents a significant property under NR criterion D. If one cannot find a relevant historic context, a context would need to be developed. Compare the resource to the property types included in the historic context and ask the following question: Does the resource meet the required criteria to be considered a representation of this property type? If the answer to the above question is YES, you have a property that is eligible for listing in the National Register of Historic Places. How the HPST and ECREL Can Help Enter information about the property in the HPST. Information focuses on attributes that are relevant to determining eligibility. 1. Look for a relevant historic context in the HPST. 2. Use ECREL to search for relevant historic contexts if not found in the HPST. 1. Use ECREL to search for similar contexts that could be used as a model for creating a new context or for relevant material that might be incorporated in the new context. 2. Use the HPST to develop a new historic context, storing key components in the database. Using the HPST ensures that contexts are being used in a consistent manner and that all necessary elements are included. 1. Use the Criteria Evaluation form in the HPST to record decisions about the NR criteria under which the property is eligible. 2. Use the Integrity Evaluation wizard in the HPST to record decisions about the property’s integrity (i.e., using the requirements of the National Register’s seven aspects of integrity). One uses the Criteria Evaluation and Integrity Evaluation functions to document the process and the reasons for making a determination of National Register eligibility. 1. Use the HPST to print an Evaluation Summary report as part of the documentation for the evaluation. 2. Use the HPST to print a new historic context report and submit to ECREL for inclusion in the library.

times on a network. This approach was tested by scanning several paper documents and rescanning several documents that existed in PDF image-only format. The process chosen was to first scan the hard-copy documents to a Tagged Image File Format (TIFF) and then rescan the TIFF to PDF. A cover sheet was created to capture metadata2 for each document. The cover sheet is designed to be read by the capture software when the document is scanned; the values entered in each box on the sheet correspond to a field in the database. The capture software reads the value and writes it into the database field. A quality control step allows the oper- ator to review the data entry and make corrections if needed. For hard-copy scanning, the cover sheet is printed, is placed on top of the document, and is the first page scanned into the system. The cover sheet is not used for electronic documents; instead, the metadata on these documents are entered by the operator. As noted in Chapter 2, ECREL is to be a web-based sys- tem. URS used Cold Fusion to develop the web pages because Cold Fusion includes a license for the Verity search engine. Verity is one of the more widely used search engines and therefore allows evaluators and testers easy access to the website containing ECREL. All ECREL design artifacts (i.e., elements and documents) are included in Appendix C. The artifacts include the com- plete requirements list (Appendix C.1) and technical specifi- cations (Appendix C.2), which include use cases and archi- tecture (Appendix C.2.1), entity-relationship diagram (ERD) (C.2.2), data dictionary (C.2.3), and the cover sheets used for document scanning (C.2.4). Develop Development of the ECREL prototype consisted of four components: • Development of the web pages and database; • Collection, scanning, and indexing of the documents; • Loading of the documents into the database and running of the indexer; and • Development of on-line help files. The database was developed using SQL Server 2000 and the web pages using Cold Fusion v.6 and HTML. Documents were collected in several different ways. The NPS has scanned multiple property submission documents and posted them on its National Register website. Twenty- three of these PDF image files were downloaded, rescanned into a searchable format, and loaded onto the ECREL site. In 16 addition, some states have begun to post historic contexts, National Register nominations, and similar documents in PDF format on the Internet. Some of these documents were also downloaded and rescanned. Most of the documents, however, were collected in hard-copy format from SHPOs in Vermont, New Hampshire, Minnesota, and New Mexico. A project team member spent 1 to 2 days onsite at each office, copying documents identified by the SHPO staff. With some guidance from SHPO staff, the cover sheets for each docu- ment were completed. The paper files were then shipped to URS’s Raleigh-Durham office for scanning. A complete list of the documents included in the prototype (and the docu- ment sources) are provided in Appendix C.3. After the documents were scanned, the metadata values were written to the ECREL Structured Query Language (SQL) Server database and documents were indexed for full- text searching by Verity. New documents had to be loaded manually to the prototype system, and Verity re-indexed the entire document. The final step before testing began was to develop an on-line help file and evaluation forms. RoboHelp was used to create the ECREL user’s guide (Appendix C.4). The ECREL evaluation form (Appendix C.5) and test procedures (Appendix C.6) were then posted on the website. Verify and Validate The initial (verification) testing was completed by mem- bers of the design team, and the primary focus was on veri- fying that all requirements had been met and that the evalu- ation form was designed to capture information that could be used to validate the prototype. The testers’ comments were incorporated into the final ECREL prototype. To test ECREL, URS solicited members of the American Cultural Resource Association (ACRA) and the National Con- ference of State Historic Preservation Officers (NCSHPO). ACRA is a national professional association of CRM profes- sionals and includes over 138 member firms. These firms pro- vide historic preservation, archaeological, historic architec- tural, anthropological, and landscape architectural services. A request to test ECREL was posted on ACRA’s “Members Only” section of the association’s website. The NCSHPO acts as a communications vehicle among SHPOs and their staffs and represents most SHPOs in developing national agreements and protocols with federal agencies and national preservation organizations. The request to test ECREL was posted on the NCSHPO’s listserv. The original requests to ACRA and the NCSHPO were sent on December 1, 2003, and the deadline for sending in evalua- tions was January 30, 2004. Because of the low number of responses received, the deadline was extended to February 28, 2004, and a reminder was sent to the NCSHPO and ACRA members. The URS project team also presented a poster on the two prototypes at the TRB 2004 Annual Meeting. During the 2 “Metadata” (which is literally “data about data”) within ECREL consist of a type of index field. The index values are defined by the user and manually entered into the data- base. These values include author, title, areas of significance, etc. To prepare the doc- ument for full-text searching, an indexer is run to create files of every unique word in the document.

poster session, the team passed out reduced-sized copies of the poster as well as copies of the evaluation form and encouraged conference participants to provide an evaluation of ECREL and the HPST. In addition, an opportunity to test ECREL and the HPST and to complete a short evaluation form was pro- vided to participants at a historic preservation and transporta- tion working conference held in Santa Fe, New Mexico, in late February 2004. The results of these evaluations are discussed in a later section of this chapter. HPST Design The URS development team worked with URS CRM pro- fessionals to complete the requirements list for the HPST (Appendix D.1) and develop the detailed design. The HPST design was driven by National Register guidance for eval- uating National Register eligibility. In order to complete a resource evaluation, the system must store data about proper- ties and key historic context elements (e.g., theme, time period, and geographic limits, property types, and appropriate National Register eligibility criteria and integrity requirements). Existing historic contexts generally include a lot of descrip- tive text. Whether or not such long narratives are really nec- essary for creating a viable historic context will not be exam- ined here. While these types of narratives may be relevant to some historic context creators and users, such extensive descriptive text is not strictly needed in the ECREL database. Entering large amounts of text in a database field is not user- friendly, and formatting (fonts, bolding, italics, etc.) is not supported. Nevertheless, several methods for including text were built into the HPST design. A historic context creator could type the narrative directly into an MS Word file that would be saved and linked to the database. Alternatively, the context creator could link an existing file in any format (PDF, for example) to the database. The context creator might also leave this section of the HPST blank. In order to use the HPST evaluation functions, however, certain key components (i.e., text) are required for the historic context. These are • A historic context, • A description of the property to be evaluated, • National Register evaluation criteria (i.e., A, B, C, and/ or D) to be applied to the property (as stipulated in the associated historic context), and • The integrity of the property (again, as required by the associated historic context) (see the Property Types form in Section 6.1.1 of the HPST User Guide, Appendix D.4). The CRM specialists on the design team emphasized the need to make the interface as intuitive as possible. Therefore, a familiar “switchboard” listing all program components is displayed when the system opens; from there, the user may select from the four components listed above. When a com- 17 ponent is selected, a form or set of forms is displayed. If more than one form is used within a module, tabs are used to facil- itate navigation. The general layout, functions, and naviga- tion for each form was designed in an MS Excel spreadsheet and reviewed by CRM staff before development began. MS Access was selected as the platform because the final application can be compiled and distributed as a “run-time” program. Therefore, end users do not need a copy of MS Access in order to use the prototype. All HPST technical spec- ifications are in Appendix D.2, including use cases (D.2.1), architecture (D.2.2), entity-relationship diagram (D.2.3), and data dictionary (D.2.4). The HPST would be placed onto a CD for distribution and testing. The CD would also provide the user guidance on installing the HPST onto their computers. Develop When the design was approved, the URS program devel- oper began working on the prototype. The tables and fields (i.e., columns) in each table were developed in MS Access. The data entry forms were developed in Visual Basic for Applications (VBA). The evaluation version of the HPST prototype included several example historic contexts and properties. Appendix D.3 includes sample reports from the HPST showing some of the data entered as examples. A user’s guide (Appendix D.4) and evaluation form (Appendix D.5) were also devel- oped and included on the installation CD sent to prospective testers. Verify and Validate The HPST was verified by a URS archaeologist and archi- tectural historian. Each was asked to enter a property and a historic context to the tool, and to evaluate the property. Each was also asked to verify that the requested functionality was included and to provide comments on bugs or usage issues. Finally, each was asked to complete the evaluation survey form. Comments were addressed before the final HPST pro- totype installation CD was created. The contexts and proper- ties that the archaeologist and architectural historian entered were included in the version of the HPST that was distributed to outside validation testers. Potential outside validation testers were identified in sev- eral ways. First, the availability of the HPST was advertised in the e-mail letter sent to ECREL testers. Anyone interested in testing the HPST was asked to contact one of the research team members. Requests were received from two people as a result of this notification; however, none of them returned an evaluation form after receiving the HPST CD. An additional 30 HPST CDs were distributed at URS’s poster session at the TRB 2004 Annual Meeting. A few peo- ple spent some time working with the HPST during the poster session. None of these people, however, submitted an

evaluation form. Some verbal comments were received. Gen- erally, people felt that some sort of tool like the HPST was needed, but it would be difficult to get SHPOs to use this type of tool. Some state employees believed that they already had tools that provide the same or a similar function. As described above, the HPST was also tested by some of the participants at the 2004 Santa Fe historic preservation and transportation working conference. A few of the participants requested a CD and, upon returning to their offices, asked their staff to evaluate the tool. EVALUATION RESULTS Appendix E contains the complete set of received evalua- tions, a list of individuals who provided comments on either tool, the responses from the evaluation forms, and the Santa Fe conference participants’ responses. A summary of the evalua- tion results is presented below. Evaluation Forms Relatively few people completed the evaluation forms. URS received nine evaluations of ECREL and two of the HPST (and the latter were both completed by URS CRM staff). Responses were received from the following states: Arkansas, Illinois, Minnesota, Montana, Nevada, North Carolina, Ohio, Washington, and Wisconsin. In general, the respondents were enthusiastic about ECREL and thought the HPST had merit. All respondents felt ECREL should be completed (i.e., more documents should be added) and maintained on a permanent basis. While other websites contain cultural resource docu- ments, ECREL is unique in that it facilitates the search for a wide variety of documents in one place, presents document metadata in a useful format, and provides several different search options. Most reviewers did not think that ECREL would increase the number of historic contexts produced; however, they did feel that this tool would result in better his- toric contexts when they were created. The HPST’s future was more problematic. The testers felt that the idea had merit, but wondered if anyone would use it unless forced. Implementing the HPST will require changes to existing processes, maybe even replacement of current procedures and tools. ECREL, on the other hand, augments existing pro- cedures and tools. Poster Presentation at the 2004 TRB Annual Meeting As noted previously, a large, colored poster that described the two prototype IT tools was available for viewing at the TRB Annual Meeting held in Washington, D.C., in January 2004. Conference attendees were able to use the HPST on lap- 18 tops at the poster session and discuss the project concept with project personnel. URS distributed 30 copies of the HPST CDs as well as handouts with the ECREL website URL informa- tion. No evaluation forms were returned as a result of this poster session. Informal comments at the session were gen- erally positive. Most people felt that tools like these were strongly needed, but that more support (in the form of fund- ing and training) would also be needed to get the SHPOs to use them. Demonstration/Review of the HPST and ECREL at 2004 Santa Fe Historic Preservation and Transportation Conference Conference attendees were given the opportunity to use both prototypes and were asked to complete a very abbrevi- ated evaluation form. Nine people completed the forms. Reviewers represented the following states or tribal groups: Missouri, New Mexico, New York, Texas, and White Moun- tain Apache Tribe. In general, the response was that the tools “showed promise” and could streamline the review process. The caveat was that getting people to use them would be a challenge. After the conference, several individuals worked very hard to try to get additional feedback from their respective staffs. The following highlights some of the key observations made by these staff. Minnesota DOT Beth Hobbs (an IT focus group member) worked with two other CRM staff members in her office to review the HPST. While they did not complete the evaluation form, they did pro- vide written comments on potential enhancements of the tool and stated that they were “very excited about its possibilities.” California Department of Transportation (Caltrans) Margaret Buss (NCHRP panel member) worked with URS to present ECREL and the HPST to members of her staff. After they had reviewed and commented on the prototypes, she invited SHPO staff to her office for a similar presentation and discussion. Again the response was largely positive. Ms Buss noted that In the group session [of Caltrans staff] . . . most of the com- ments on ECREL revolved around concerns for 1) security of confidential information, and, more important to the group, 2) concern about ensuring the context statements were vetted by SHPOs, that is, that they were considered good, sound research and represented an official view of some kind. The tool itself was easy enough to use; there were no sub- stantive comments about ways to improve the tool itself, only concern about who would be arbiter/manager/approver of the

contexts posted. Aside from that concern, everyone was enthu- siastic about the concept of being able to share contexts and research. Regarding the HPST, Buss noted that Caltrans management “is reluctant to impose it if our SHPO doesn’t require it . . .” However, Caltrans is already developing electronic tools and may be able to integrate the HPST into these tools, so Cal- trans could begin to compile contexts in the HPST format. In addition, Buss said that two Caltrans staff demonstrated both ECREL and HPST to about seven SHPO staff members plus one senior archaeologist at State Parks. They all thought both were great ideas but had no specific feedback on how it worked. The SHPO staff were going to 19 send HPST to a couple of CLGs [Certified Local Govern- ments] who were sophisticated on computers to see if they could use it in building their local surveys, and the State Parks archaeologist was very enthusiastic about using it for their current inventory effort within all the State Parks. I con- tacted all of them after a couple of weeks to see if they had tried it and had feedback. The response from SHPO was that they hadn’t played around with it and probably wouldn’t do much with it unless directed by their bosses, who didn’t attend the presentation, and the State Parks staff archaeolo- gist didn’t return my call. The general consensus from these reviewers was that both prototypes are worth pursuing, but, without specific direction from their managers, SHPO and DOT staffs are not going to invest any time to adequately test these prototypes.