Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
51 The proposed evaluation process described in this chapter is appropriate for high level evaluation of AGVT for airside applications and may be analogous to the evaluation conducted during the alternatives analysis phase of the airport master planning process (FAA, AC 150/ 5070-6B, 2015; shown in Appendix F). Once a technology application is selected, additional project planning would be required prior to deployment. The intention of this evaluation process is to develop a framework for discussion and support transparency in decisions regarding the selection of technology applications that are appropriate for further development. Due to the wide range of characteristics at individual airports (e.g., geographic layouts, aeronautical activities, financial situations, cultures and priorities, airport goals and objectives) and due to the rapid changes in technology, this evaluation process is not intended to identify a technology application for a specific category of airports (e.g., large-hub airports) or a specific airport. However, it can be used to identify airport characteristics that may be compatible with an AGVT project. This evaluation process sets forth evaluation criteria and supporting assess- ment areas. Airports may select the assessment areas that are most relevant for their airport and the proposed technology application. The extensive list of assessment areas may be used to guide discussion and the selection of appropriate assessment areas will allow a tailored approach to evaluation. The proposed approach is intended to provide a structured framework for the discussion of impacts and support documentation for decisions. Overview of Evaluation Process The first step in the evaluation process is to identify the stakeholders that should be involved in the evaluation process as part of the study team. Early involvement in planning and evalua- tion will assure that relevant input is provided before major decisions are made. The study team should include airport stakeholders who will be affected by the project (e.g., operations, funding, financing, regulatory approval, and/or administrative oversight) as well as professionals with technical expertise in the candidate technologies. A sample list of stakeholders that may be involved is included in Appendix G. It may be helpful to clearly define the key issues from each stakeholderâs perspective early on. Key issues may relate to cost, safety, operational goals, labor, environmental considerations, and/or airport goals. A clear understanding of each stakeholderâs key issues will be helpful in the definition of appropriate candidate projects, as well as support the evaluation and refine- ment of projects. The basic overview for evaluation is shown in Figure 19. Candidate projects are identified and refined in the evaluation process which considers risk as well as the following evaluation C H A P T E R 5 Evaluation Process
52 Advanced Ground Vehicle Technologies for Airside Operations criteria: ease of adoption, stakeholder acceptance, technical feasibility, infrastructure impacts, operational impacts, benefits, and human factors. The evaluation is described in greater detail in the following sections. Identification of Candidate Projects The first step is to identify the goals of automation, as well as key issues for stakeholders. It is possible that the goals of automation may align with or conflict with key issues for some of the stakeholders. For example, a proposed AGVT application of reduce reliance on labor would align with a reduction in personnel costs but would conflict with worker job security. Candidate projects are defined by the application and the technology used. Multiple tech- nologies may be considered for a single application. For example, perimeter security could be accomplished with either of the following AGVT alternatives: automated without a driver or remote operation. The description of the candidate technology should define equipment components, identify where the technology components are located (e.g., vehicle-based versus Human Figure 19. Overview of evaluation process.
Evaluation Process 53 centrally based), and identify the level of automation (when appropriate). The description should include the operational domain (e.g., ramp, runway, taxiway, and/or remote portions of the airfield) and relevant domain designations (e.g., movement area or non-movement area). Each candidate project should be defined with respect to one of the following implementa- tion concepts. These categories reflect the permanence of the project and the impact on airport operations. ⢠Full deployment: Implementation of technology application on a permanent basis. ⢠Demonstration: Implementation of technology application on a trial basis. Trial is for a limited period of time and with a defined scope. The trial may be designed to have minimal disruption to ongoing airside operations. ⢠Investigation: Collection of data to support proposed technology application in the future. Data collection may include characteristics of current operations and/or capabilities of the technology application in the airport environment. The candidate projects can be compared to one another and to the âdo nothingâ scenario. The do nothing scenario reflects the current operational framework and provides a baseline to compare the evaluation of new technologies; it represents a situation in which no new techno- logies are implemented or investigated. In some cases, the do nothing scenario may be the best recommendation in the near term. A preliminary concept of operations for the candidate project(s) should be developed. The concept of operations builds on the candidate technology application definition and imple- mentation concept. It includes a description of the project objective, the supporting operational goals, and the operational scenario. The concept of operations should state the capabilities and limitations of technology, tasks that the technology will accomplish, and tasks required of the humans who interact with the technology. The concept of operations should describe inter- connections with existing processes and procedures in addition to the operational environment. The concept of operations can include sample scenarios including operation under normal conditions and operation when faced with minor to major problems. Evaluation of Candidate Projects Evaluation of the candidate projects considers both the safety of the candidate projects and an evaluation of the candidate projects using the following criteria: ease of adoption, stake- holder acceptance, technical feasibility, infrastructure impacts, operational impacts, benefits, and human factors. Risk Assessment Safety risk assessment is an important part of safety risk management which is one of the four components of a safety management system (SMS) (FAA, 2016a). If an airport has an SMS program, that program can be used for risk assessment. If an airport does not have an SMS, then guidance from the Office of Airport Safety Management Systems (SMS) Desk Reference can be used (FAA, 2012a). Additional information regarding risk assessment is provided in Appendix H. The 5M model may be used to analyze the proposed AGVT application. The 5M model is shown in Figure 20a and considers impacts for the following five elements: ⢠Mission: the proposed AGVT operation, ⢠Hu(man)/Person: the human operators, ⢠Machine: the AGVT equipment, including hardware, software, firmware, HMI, ⢠Management: the procedures and policies that govern the AGVT, and ⢠Media: the airport environment where the system will operate.
54 Advanced Ground Vehicle Technologies for Airside Operations Safety Risk Management includes five steps for hazard assessment: 1. Describe the system, 2. Identify the hazards, 3. Analyze the risks associated with hazards, 4. Assess the risks associated with hazards, and 5. Mitigate the risks identified when necessary. (a) (b) Figure 20. Risk assessment concepts: (a) 5M Model and (b) risk matrix. Image: FAA, 2016a; FAA, 2012a.
Evaluation Process 55 Risks are categorized by severity and likelihood, as shown in Figure 20b. If the risks asso- ciated with the candidate project are categorized as low or medium risks, the candidate project should be considered for further evaluation and may be recommended for further development. If the risks associated with a candidate AGVT project are considered high based on the risk matrix, then appropriate risk mitigation must be identified. Risk mitigation strategies would vary depending on the risks and the proposed project, but may include reducing the scope of the project or changing the proposed application. Mitigation measures may also include changing the proposed technology (e.g., adding a safety driver), changing the operational domain (e.g., deploying an automated mower in a remote area of the airfield), or changing the proposed procedures (e.g., a snowplow platoon with a driver in the lead shall operate only between the hours of midnight and 3 am when there are no scheduled commercial flights). One way to mitigate risk may be to change the implementation concept from full deployment to demonstra- tion, which would allow additional data regarding the hazards and risks to be collected, or from a demonstration project to an investigation. If appropriate mitigation measures cannot be identified or if they do not reduce the risks to medium or low, there is no need for further evaluation since the project cannot be implemented. If the candidate AGVT project(s) have a low or medium risk, then additional evaluation will be conducted considering the following criteria: ease of adoption and stakeholder acceptance, technical feasibility, infrastructure and operational impacts, benefits, and human factors. Although this discussion describes risk assessment and evaluation separately for simplicity, risk assessment may also be considered during the evaluation process as many of the concepts that are evaluated have implications for risk assessment. For this reason, it is likely that the candidate technologies may be revised during the evaluation process, as shown in Figure 19. Evaluation Criteria Evaluation is based on the following evaluation criteria: ease of adoption, stakeholder accep- tance, technological feasibility, infrastructure impacts, operational impacts, benefits, and human factors. Each criteria is described briefly: ⢠Ease of adoption. Ease of adoption refers to how easily the technology can be deployed, considering physical and financial requirements of the proposed deployment, as well as regulatory, institutional, and political considerations. ⢠Stakeholder acceptance. Stakeholder acceptance reflects the organizational and individual support or concerns with the proposed deployment. Affected stakeholders potentially include all organizations and individuals who would be affected by the proposed technology deployment, which would include the people and organizations that use the technology (e.g., operations), interface with the technology, pay for the technology, or are affected by its deployment in any other way. Stakeholder acceptance would include both labor and management perspectives. ⢠Technical feasibility. Technical feasibility refers to the practicality and maturity of the proposed technology deployment, including the equipment, machinery, computers, or auto- mation in the context of the airside environment in which it will operate. ⢠Infrastructure impacts. Infrastructure impacts include the changes to airport infrastructure that would be needed. This could include new centralized or distributed hardware and soft- ware as well as changes to the airfield infrastructure, such as new requirements for airport signs or markings, or new beacons or towers to mount equipment. ⢠Operational impacts. Operational impacts include the changes to current operations that would occur as a result of technology deployment. This may include changes in procedure, equipment, processes, or systems. ⢠Benefits. Benefits includes the benefits and costs associated with the proposed technology application. Benefits potentially include increased safety, efficiency, security, service, and reduced costs. Costs include both capital costs and operating costs for the new technology.
56 Advanced Ground Vehicle Technologies for Airside Operations ⢠Human factors. Human factors reflect the interaction between people and the proposed technology deployment. Ideally, technology deployments will leverage technology capabilities while allowing people to do what they do best. Assessment areas are identified for each of these criteria. For some assessment areas, sup- porting principles and examples are also provided. In tables 11 to 16, supporting principles and examples are bulleted under the relevant assessment areas. The assessment areas and the supporting principles and examples provide guidance and a framework to consider the poten- tial impacts of the technology. Not all assessment areas are relevant for every project. Only relevant assessment areas and supporting principles should be included; assessment areas that are not relevant to the candidate project should be excluded from consideration. Additional assessment areas can be added as needed to tailor the evaluation to the candidate project. For some projects, it may also be appropriate to âpromoteâ a supporting principle so that it becomes a standalone assessment area. A list of assessment areas (and supporting principles and examples) and how they could potentially relate to evaluation criteria is shown in Appendix I. In many cases, there may be overlap between evaluation criteria (e.g., a reduction in labor due to automation would provide benefits and would affect stakeholder acceptance). For simplicity, each assessment area is typically mapped to a single criterion; exceptions are indicated with a footnote. Ease of Adoption and Stakeholder Acceptance Ease of adoption and stakeholder acceptance are closely related and share many common assessment areas as substantiated by scholarly literature. One of the most broadly used general frameworks of user acceptance is the Technology Acceptance Model (TAM) (Davis, 1989). This model states that perceived usefulness (i.e., the degree that the individual user believes the tech- nology will help carry out a task) and perceived ease of use (i.e., the amount of effort the indi- vidual will exert to properly use the technology) are the main determinants for an individualâs use of a technology. The unified theory of acceptance and use of technology (Venkatesh, et al., 2003) provides additional context for the technology and recognizes social influence and orga- nizational factors such as facilitating conditions and whether the use is voluntary. Ghazizadeh, Lee, and Boyle (2012) extended the TAM to evaluate automation and suggest that trust and task/technology compatibility are important as effective interaction between automation and the human user is crucial to assure its safe and effective use. Ease of adoption and stakeholder acceptance would encompass characteristics of both indi- vidual users (e.g., ramp workers) as well as affected organizations (e.g., unions). Relevant considerations may include the attitude of individual users, compatibility of the proposed technology with the task, and organizational influences. Considerations that affect stakeholder acceptance and ease of implementation often overlap with considerations relevant for other criteria, including technical feasibility, benefits, and human factors. For example, the assessment area âTechnology is reliable and trusted by usersâ could be included in benefits or in the evalua- tion with respect to technical feasibility. An airport may choose to include an assessment area in whichever category is most appropriate, or may include it in multiple categories, recognizing the that impact will be âdouble countedâ when it is included in two categories (double counting may be appropriate due to its importance). The assessment areas for ease of adoption and stake- holder acceptance are shown in Table 11. Technical Feasibility The technical feasibility of an alternative reflects the maturity of the proposed AGVT and whether it has been demonstrated in the airside environment. The maturity of a technology is often measured by the technology readiness level (TRL), which has been widely used in the
Evaluation Process 57 defense industry and is determined using a technology readiness assessment (TRA). TRA examines the following areas: (general) technology readiness, safety concerns, risk criteria, and sustain- ability. The resulting TRL ranges from 1 to 9 with 1 representing the lowest level of readiness and 9 representing a technology that has been successfully deployed, as shown in Figure 21. Assessment areas for technical feasibility are shown in Table 12. If the TRL does not align with the proposed project, then it may be appropriate to shift the implementation concept from a full deployment to a demonstration project, or from a demonstration project to an inves- tigation, or to delay the project until the technology has advanced. Infrastructure Impacts Infrastructure impacts reflect whether the candidate project utilizes and leverages existing airfield infrastructure or requires new infrastructure. If new infrastructure is needed, then consideration must be given as to whether the proposed infrastructure is compatible with existing and future airport needs, in terms of both systems used (e.g., ASDE-X) and physical space required for new infrastructure. The cost component associated with new infrastructure overlaps with the cost considerations included in Benefits. A list of potential assessment areas for infrastructure impacts is shown in Table 13. Operational Impacts The evaluation of operational impacts is based on the concepts outlined by the FAA for master planning, which include capacity, capability, and efficiency. Assessment areas for operational impacts are shown in Table 14. ⢠Capacity. Does the candidate project provide the same capacity currently provided or increase the capacity of the airport? Does the candidate project allow the airport to meet current and/ or future capacity requirements? 1 Compatible with existing culture and supports positive culture (addresses organizational and sociological environment in which change will occur) 2 Considers previous organizational experience with technology or other related initiatives 3 Compatible with existing workforce constraints and considerations (e.g., union rules) ⢠Individuals are likely to use the technology as intended 4 Appropriate allocation of function allows technology to perform appropriate tasks while removing burdensome tasks from people and ensuring humans can maintain situational awareness1 5 Considers funding and financial implications ⢠Considers capital costs and eligibility of potential funding sources ⢠Operating costs can be recovered with appropriate revenue stream 6 Considers need for regulatory compliance and approval (including airport certification, etc.) ⢠Requires approval from FAA, TSA, OSHA, CPB, etc. 7 Project provides expected cost savings (positive B/C, benefits outweigh cost)2 ⢠Benefits accrue to entities that incur a cost (in $ or inconvenience) 8 Technology is reliable and trusted by users3 ⢠Human or technical backup can take over if needed 9 Technology can be implemented incrementally ⢠Automation has viable interim phases (e.g., local monitoring followed by remote monitoring or implementation in non-movement area, then movement area) versus âall-or-nothingâ 1 Could also be considered in human factors. 2 Could also be considered in benefits. 3 Could also be considered in technical feasibility. Table 11. Assessment areas for ease of adoption and stakeholder acceptance.
58 Advanced Ground Vehicle Technologies for Airside Operations ⢠Capability. Does the candidate project meet the current capabilities of operation or would it increase the airportâs ability to meet specific functional objectives, such as the ability to accommodate a specific aircraft or a variety of aircraft (e.g., for a candidate project focused on jet bridge automation), or provide a higher level of security (e.g., for a candidate project focused on perimeter security) than currently exists? ⢠Efficiency. Does the candidate project work well with other system components to meet effi- ciency objectives such as aircraft turn time (e.g., for automated baggage carts) or the ability to conduct operations with fewer pieces of equipment or fewer people (e.g., for automated mowing). In some cases, operational impacts are closely related to and overlap with assessment areas included in benefits (e.g., efficiency) and human factors. Benefits Expected benefits may be a catalyst for AGVT in the airside environment and potential benefits include increased safety, efficiency, security, service, and reduced costs. These benefits are intuitive and have been substantiated by the critical success factors of efficiency, money, Figure 21. Technology readiness levels. Image: U.S. GAO, 2016c.
Evaluation Process 59 1 Overall estimated TRL is compatible with candidate project ⢠Software readiness level, hardware readiness level, integration readiness level and system readiness level are compatible with proposed application 2 Technology Adaptability and candidate technology can successfully be adapted to airside use ⢠System adaptability, including procedure changes and training requirements ⢠Need for additional equipment to manage AV in proximity of aircraft ⢠Need for additional procedures for compliance with ATC and other airside protocol ⢠Need for additional resources due to airside environment (this may include onboard power, signal connection, airline and aircraft profiles, airport profile, ATC communication phraseology database if using synthesized voice on board autonomous vehicles, etc.) 3 Technology safety and system redundancy are adequate in the event of technology failure or unexpected operating conditions. Technology response in emergency is predictable, adequate and appropriate ⢠Hardware redundancy and fault-tolerant software are well proven ⢠Appropriate fail-safe state exists (e.g., move off runway in the event of fatal failure and report position if incapacitated) ⢠Safety testing has been successful in all conditions (all weather, high and low temperatures, proximity testing, loss of communication testing, fail-safe state testing) ⢠Safety evaluation is compatible and proven for designated level of autonomy associated with candidate project ⢠Human or technical backup can take over if needed 4 Technology risks including security and cybersecurity have been identified and successfully mitigated 5 Technology sustainability is adequate and commensurate with investment, includes the expected life of technology before replacement, assessment of the long-term benefits and the needed support for the technology 6 Confidence in key information is compatible with proposed application 7 Quality and reliability of available data are compatible with proposed application 8 Process to be automated is repetitive and predictable (something a computer can understand as well or better than a human)1 9 Operation is in isolated environment not subject to unpredictable crossing traffic 1 Could also be considered in human factors allocation of function. Table 12. Assessment areas for technical feasibility. 1 Minimal changes to the airport infrastructure are required 2 Requirements for new infrastructure will provide benefits commensurate with costs ⢠Equipment such as charge stations, signal towers, and positioning beacons can be located without disrupting current or future aeronautical activities and locations are consistent with ALP ⢠Data center can manage reference queries from AV ⢠Central control infrastructure and location are appropriate 3 Requirements for new infrastructure will integrate with and leverage existing infrastructure systems (e.g., ATC, ground control, ASDE-X, or other airport infrastructure) and benefits that offset costs 4 Project is compatible with future infrastructure plans and needs ⢠Capacity of proposed infrastructure can meet needs identified in the master plan, or be expanded to meet future needs 5 Maintenance needs for AV and AGVT can be accommodated on the airfield or arrangements can be made for contract service off-site Table 13. Assessment areas for infrastructure impacts.
60 Advanced Ground Vehicle Technologies for Airside Operations safety, and support of institutional goals, as defined in technology forecasting (Srivastav and Misra, 2014). These considerations are reflected in the assessment areas shown in Table 15. While expected benefits may include cost savings, it is important to consider the value of these savings relative to the cost of the technology. This may be expressed as return on investment or benefit-cost analysis. Additional information regarding benefit-cost analysis including a procedure based on guidance from the FAA is shown in Appendix J. Human Factors Human factors evaluation reflects how the proposed technology can be incorporated into airside activities in the context of working effectively with the people in the system. The evalu- ation of human factors is based on the 24 categories defined by FAA (2012b) and described in greater detail in Appendix K. These assessment areas are shown in Table 16 and reflect the following components: ⢠Equipment, ⢠Systems, ⢠Jobs, ⢠Environments, ⢠Software, ⢠Facilities, ⢠Procedures, ⢠Training, ⢠Staffing, and ⢠Personnel management. 1 Supports institutional goals 2 Increase safety and reduce human exposure to health and safety hazards ⢠Reduce worker fatigue ⢠Reduce worker exposure to hazards ⢠Reduce worker injuries (severity and incidence) and fatalities ⢠Reduce runway incursions 3 Compatible with staffing needs and requirements 4 Provides useful data to enhance current or future operations ⢠Project will result in data or information to support future deployment ⢠Improve airport situational monitoring through provision of metrics or real-time data ⢠Provides increased oversight or better documentation of activities and events 5 Project provides expected cost savings (positive B/C, benefits outweigh cost) ⢠Reduces aircraft damage and property damage (incidence and severity) ⢠Reduce personnel costs (including overtime) 6 Increases security or efficiency of security (e.g., fewer breaches or false alarms) 7 Reduces emissions or other environmental benefits Table 15. Assessment areas for benefits. 1 Allows current infrastructure to operate at higher capacity (more demand) without new construction 2 Meets current capabilities with the same level of service 3 Increases efficiency ⢠Improves equipment utilization ⢠Improves efficiency of operations by reducing reliance on human capital ⢠Improves efficiency of operations by providing reliable time for task completion ⢠Reduce aircraft delay at airport ⢠Reduce aircraft delay in system 4 Airport characteristics are compatible with proposed AGVT deployment including regulatory considerations, number of operations, types of aircraft, and physical characteristics of the airport Table 14. Assessment areas for operational impacts.
Evaluation Process 61 1 Reflects ergonomic principles and accommodates physical attributes of users (anthropometrics and biomechanics) 2 Good computer-human interaction supports ease of use (dialogues, interfaces, and procedures across functions) 3 Functional design is compatible with operations and maintenance requirements 4 Compatible with existing or proposed procedures (operating and maintenance procedures are simple, consistent, and easy to use) 5 Appropriate allocation of function allows technology to perform appropriate tasks while removing burdensome tasks from people and ensuring humans can maintain situational awareness 6 Provides needed communications and supports teamwork 7 Compatible with environment including extremes and reflects impact of environment on human- system performance; ideally reduces need for humans to perform in harsh environment 8 Operational suitability ensures system supports user to perform job and system provides interoperability and consistency with other system elements 9 Compatible with staffing needs and requirements 10 Enhance situational awareness including operator awareness and airport situational awareness 11 Supports work space requirements including adequate work space for personnel, equipment, and tools, including movements under normal, adverse, and emergency conditions 12 Compatible with workload requirements including physical, cognitive, and decision-making resources 13 System reduces human error and considers supervisory and organizational influences as causal factors, considers error tolerance, prevention and correction/recovery Table 16. Assessment areas for human factors. There is extensive overlap between human factors considerations, and considerations integral to other criteria, such as stakeholder acceptance and ease of adoption, benefits, etc. In some cases, the human factors assessment areas are categorized as supporting principles and examples, as shown in the example in Table 17. It may be appropriate to âpromoteâ supporting principles and consider them as individual assessment areas, depending on the candidate project and airport priorities. One important component of human factor is the interface between human and machine, often referred to as HMI. This includes visual displays, touchscreens or other interfaces through which people operate machines, and data visualization tools that provide information regarding system operation and key metrics regarding performance. Human systems integration (HSI) encompasses HMI as well as other components that support the smooth integration of tech- nology and people in the airside environment including manpower, personnel, training, human factors, engineering, safety, and occupational health (Phillips, 2010). In some cases, the assessment areas reflect a degree of specificity that is neither available nor appropriate for investigation, demonstration project, or planning-level evaluation. For example, reference to ergonomic principles, good computer-human interaction, and functional design all suggest the evaluation of a specific technology rather than a conceptual technology application. In planning-level evaluation, items that are not appropriate for consideration can be excluded from the analysis. Valuable resources when evaluating the human factors considerations include the Human Factors Design Standard (Ahlstrom and Longo, 2016). This document discusses considerations related to automation in Section 5.1 (including evaluation in 5.1.2 and levels of automation in 5.5.11) and HMI in Sections 5.3 through 5.7 (including information on displays, controls and visual indicators, alarms, computer-human interface, and keyboards and input devices). Quantifying the Results of the Evaluation into a Score Airports may choose to summarize the results of their evaluation in a qualitative descrip- tion. In other cases, airports may wish to have a quantitative method for evaluation. Due to the
62 Advanced Ground Vehicle Technologies for Airside Operations conceptual nature of some criteria components, the proposed evaluation is a mix of quanti- tative and qualitative information. For some AGVT applications, technologies are still evolving and there is a degree of uncertainty associated with their operation, which increases the importance of qualitative information and assessment. The evaluation process described below uses the total quality management quality function deployment (QFD) method to translate qualitative judge- ment to a quantitative decision. QFD is a multi-attribute utility theory that has been used in the auto and aerospace industries for decades by companies such as Ford, Boeing, and McDonnel Douglass (Vance et al., 2018). In this evaluation, which uses QFD, a composite score for each candidate AGVT project is based on scores of 0, 1, 3, or 9 relative to the do nothing scenario or to another candidate AGVT deployment. A score of 0 indicates that there is no difference or preference, a score of 1 indicates a marginal or weak preference, a score of 3 indicates a measurable or medium preference, and a score of 9 indicates clear superiority or a strong preference. This evaluation uses equal weighting for all criteria; however, it would be appropriate for an airport to provide weighting factors that reflect their local priorities. The use of equal weighting factors is consistent with the online survey results (average scores are shown in Appendix D). The weight of stakeholder acceptance and ease of use combined is equivalent to other criteria; however, some airports may wish to increase the weight of this combined criteria to reflect the fact that it encompasses both stakeholder acceptance and ease of use. Project Refinement During the evaluation process, projects may be refined or modified. For example, the imple- mentation concept may change from full deployment to demonstration to reduce the asso ciated 1 Good computer-human interaction supports ease of use (dialogues, interfaces, and procedures across functions) ⢠Displays and controls design and arrangement is consistent with tasks and actions ⢠Information presentation uses consistent labels, symbols, colors, terms, formats, and data fields ⢠Information requirements are met (needed information is available and useable) ⢠Input/Output (I/O) devices allow critical tasks to be performed quickly and accurately ⢠Visual/Auditory alerts enhance safety 2 Provides needed communications and supports teamwork ⢠Communication with ATC, ramp control, airport operations, and ramp workers, including the ability to signal intent with people 3 Compatible with workload requirements including physical, cognitive, and decision-making resources ⢠Considers operator, other ramp personnel, ATC, management, airport, and other stakeholders 4 Operational suitability ensures system supports user to perform job and system provides interoperability and consistency with other system elements ⢠Compatibility with airside practices and procedures (including stakeholder intention to use) 5 Compatible with staffing needs and requirements ⢠Compatible user training needed for success including knowledge and skills to interface with system and design system to support learning and retention ⢠Considers needs for new knowledge, skills, and abilities to perform tasks as well as selection requirements for users (existing or new personnel or contractors) ⢠Considers the need for special skills and tools 6 Compatible with existing or proposed procedures (operating and maintenance procedures are simple, consistent and easy to use) ⢠Appropriate documentation can be provided in suitable format Table 17. Supporting principles and examples for selected human factor assessment areas.
Evaluation Process 63 risk. Such a change is likely to increase the evaluation score for criteria such as technical fea- sibility, ease of implementation, and stakeholder acceptance. Recommendations Once the candidate projects have been evaluated, refined, and scored, the final step is to recommend proposed projects for further development in the near term, as well as identify projects for consideration in the longer term, if they are of interest but not currently recom- mended. Projects that align with airport objectives should be considered for evaluation again in the future as technologies mature. Example Evaluation Using QFD An example illustrating the evaluation procedure using QFD is provided for a candidate project to implement automated perimeter security. This example is provided to illustrate the pro- cess of QFD for a specific airport with a specific technology rather than to provide the evaluation of automated perimeter security. The initial candidate projects included automated perimeter security either with full deployment or as a demonstration project, as described in the concept of operations in Figure 22. In this example, during the initial risk assessment, full deployment Candidate Technology and Application: An automated vehicle without a safety driver will be used for perimeter security. Project Goals: The project goal is to provide automated surveillance of the perimeter fence and gates; this will reduce the reliance on personnel in a tight job market and will potentially increase the amount of surveillance that can be provided. A secondary goal is to learn more about the potential for airside automation. Operational Objectives: The first operational objective is to demonstrate the viability of automated perimeter security so the airport can reduce the reliance on personnel to conduct this task, which would allow increased airport security. A secondary operational objective is to collect data using an automated system to learn more about the benefits and limitations of automation. These operational goals align with the mission of the airport: to provide the highest quality of service for the community and for aviation customers (including safety and security), to be an employer of choice and to be a leader in innovation. The candidate project would enhance safety by removing personnel from perimeter security in harsh environmental conditions (e.g., heat and cold) and would enhance airport security by providing the potential for continuous surveillance during the day and night. Operational Scenario: The AV will patrol the perimeter of the airport inside the perimeter fence and check for holes in the fences, locked gates, and people along the perimeter of the airport outside the fence who may be trying to enter the airfield. When any of the above conditions exist and are recognized, or when an unrecognized situation exists, the AV will send an alert to a human supervisor who can view the situation remotely and provide a course of action for the AV, typically it would be one of two actions: stay at the location and provide video recording and a live feed until an airport operations vehicle arrives, or continue the perimeter check. Humans will be required to respond to alerts, charge the AV, and download any data or video the airport wishes to keep for documentation. Domain: The AV will operate in remote areas of the airfield, and is not expected to operate in the movement area. In terms of security designations, the AV will operate or not operate in the secure area or the air operations area. Potential Problems: Minor problems may include unrecognized situations and false alarms for unlocked gates or holes in the fence. No major problems are expected, but a problem such as a cybersecurity threat would represent a major problem. The fail-safe condition for the AV is to stop operation, send an alert, and stay in place until directed to do otherwise. Implementation Scenario: Candidate projects include full deployment (implementation on a permanent basis) and implementation as a demonstration project (implementation for 6 weeks on a trial basis under close supervision). Figure 22. Example of concept of operations for automated perimeter security.
64 Advanced Ground Vehicle Technologies for Airside Operations was considered medium risk because the airport had no experience with this technology and the technology was relatively immature, which were balanced by the facts that deployment was exclusively in remote areas far from the runways and the AV was geofenced and would stop as a fail-safe condition. Demonstration was considered low risk as the operation was exclusively in remote areas, the AV was geofenced, the AV would stop as a fail-safe condition, and the AV could be closely monitored since it was a trial of limited duration. Evaluation with respect to stakeholder acceptance and ease of adoption was conducted and is shown in Table 18. The next step was evaluation with respect to technical feasibility. During this step of the evaluation process at this airport, concerns were raised regarding cybersecurity. Consideration of the cybersecurity risks associated with full deployment resulted in a modi- fication of the risk assessment to reflect uncertainties associated with the relatively immature technology, specifically with respect to the threat of a hacker reprogramming the AV path, changing the geofence boundaries, and allowing the AV to enter the air operations area. The modified risk assessment for full deployment was determined to be high risk, which precluded further evaluation unless adequate mitigation measures could be identified. The risk assessment for a demonstration project remained low risk because of the close oversight and the limited duration of the demonstration. Table 19 illustrates the calculation of the score for benefits for the remaining candidate project compared to the do nothing alternative. Table 20 illustrates the calculation of the overall score Do Nothing Automated perimeter security demonstration Automated perimeter security deployment Ease of Adoption and Stakeholder Acceptance 0 0 0 Compatible with existing culture 0 3 1 Considers previous organizational experience with technology or other related initiatives 0 1 1 Compatible with existing workforce constraints and considerations (e.g., union rules) ⢠Individuals are likely to use the technology as intended 0 1 1 Appropriate allocation of function 1 3 0 Considers funding and financial implications ⢠Considers capital costs and eligibility of potential funding sources ⢠Operating costs can be recovered with appropriate revenue stream 3 1 0 Considers need for regulatory compliance and approval (including airport certification, etc.) ⢠Requires approval from FAA, TSA, OSHA, CPB, etc. 0 3 3 Project provides expected cost savings (positive B/C, benefits outweigh cost) ⢠Benefits accrue to entities that incur a cost (in $ or inconvenience) 3 1 0 Technology is reliable and trusted by users ⢠Human or technical backup can take over if needed 0 0 0 Technology can be implemented incrementally ⢠Human or technical backup can take over if needed 7 13 6 Score 0.54 1.00 0.46 Normalized Score Table 18. Sample score calculation for ease of adoption and stakeholder acceptance.
Evaluation Process 65 Do Nothing Automated perimeter security demonstration Automated perimeter security deployment Benefits 0 3 Supports institutional goals 0 3 Increase safety and reduce human exposure to health and safety hazards ⢠Reduce worker fatigue ⢠Reduce worker exposure to hazards ⢠Reduce worker injuries (severity and incidence) and fatalities 0 3 N/A Compatible with staffing needs and requirements 0 9 Alternative Provides useful data to enhance current or future operations No Longer ⢠Project will result in data or information to support future deployment Under ⢠Improve airport situational monitoring through provision of metrics or real-time data Consideration ⢠Provides increased oversight or better documentation of activities and events 0 0 Project provides expected cost savings (positive B/C, benefits outweigh cost) ⢠Reduce personnel costs (including overtime) 0 3 Increases security or efficiency of security (e.g., fewer breaches or false alarms) 0 0 Reduces emissions or other environmental benefits 0 21 Score 0 1.00 Normalized Score Table 19. Sample score calculation for benefits. Evaluation Criteria Do Nothing Automated Perimeter Security Weight Demonstration Deployment Ease of Adoption and Stakeholder Acceptance 0.54 1.00 Does not pass Risk Assessment because of Uncertainties 0.20 Technical Feasibility 1 0.14 0.20 Infrastructure Impacts N/A N/A N/A Operational Impacts 0 0 0.20 Benefits 0 1 0.20 Human Factors 0.67 1 0.20 Overall Score 0.44 0.63 1.0 Table 20. Sample score calculation for candidate projects.
66 Advanced Ground Vehicle Technologies for Airside Operations for the candidate project. Normalized scores can be found by dividing the total score for each alternative by the highest score of any of the alternatives. Normalizing the values will result in the best alternative having a score of 1.0 and the other alternatives having a score between 0 and 1.0, with scores closer to 1.0 indicative of better ratings. As can be seen in Table 20, although the demonstration project has a lower score for technical feasibility, it provides greater benefits as shown in Table 19. Benefits include a reduced need for labor, reduced exposure of workers to hazards, and increased security as the AV will provide continuous surveillance. One of the greatest benefits of the trial implementation of AV for perimeter security is the useful data that will be generated by the proposed demonstration project. It would also be appropriate to score using other rating systems. One other option is the analytical hierarchy process (AHP). AHP often uses incremental scores from 1 to 10, which may be useful if information about each alternative is detailed and well understood. A second option for scoring is to use AHP and put the alternatives in ranked order for each criterion, with scores assigned for each alternative based on the rank. A third option for scoring would be to simply score using -1, 0, and +1 to represent negative, neutral, and positive for each criteria attribute and calculate the total score as the sum for that criteria for each alternative. In some cases, it may be more appropriate to focus on a qualitative evaluation, rather than a score. In these situations, the assessment areas and supporting principles and examples may be used to guide the qualitative assessment. This section has presented an evaluation procedure that is appropriate for airside applica- tions of AGVT. The procedure is flexible in that the assessment areas can be included, excluded, or added as appropriate based on airport objectives and on the characteristics and capabilities of the proposed candidate projects. Candidate projects can be compared to the do nothing scenario, which provides a baseline reflecting the current conditions, or to each other. Compatible Airport Characteristics The previous section focused on how an airport can evaluate AV or AGVT. It is also possible and appropriate to identify airport characteristics that are most compatible with a proposed AGVT deployment. At this point, many AV and AGVT are still maturing, which makes tradi- tional evaluation challenging. Since equipment is not available âoff-the-shelf,â it is often chal- lenging to know the full cost associated with implementation of a technology. Since operating characteristics are not well established, it is challenging to conduct risk assessment with confidence. The Silicon Valleyâs mantra of âFail Fast, Fail Oftenâ may not be entirely accurate; however, it does illustrate the inherent challenges and uncertainty associated with the deploy- ment of new and advancing technologies. In any case, it is constructive to identify airport characteristics that would be compatible with different applications and technologies. For example: ⢠An AGVT system that relies on central control may require a larger investment for infrastruc- ture; this may be of less interest to an airport that prides itself on having low costs for airlines. ⢠An AGVT application that has reduced emissions as a primary benefit would be of greatest value to airports in non-attainment areas and airports that are interested in sustainability. Example airport operating characteristics that may be considered when evaluating the compatibility with technology are shown in Table 21.
Evaluation Process 67 Category Characteristics Type of Service Does the airport primarily serve commercial or GA? Is the airport certificated through Part 139? Airport Operating Characteristics How many aircraft operations and/or enplanements are there? Is airport at capacity in terms of runways and/or terminal? What constraints or delays are associated with airside activities? Is the ramp area constrained and/or congested? Is there a single signatory airline, multiple signatory airlines or does the airport serve a mix of airlines without a single airline dominating? Size and Geometry What is the airport acreage? What is the number and configuration of runways? Do taxiways cross runways? Is expansion possible? Airport Is the city a non-attainment area? Are reduced emissions a priority to the airport? Labor Are current activities constrained by the available labor? Are there union constraints and work rules that will affect operation? Affected Agencies Is the FAA involved? Does the TSA, OSHA, and/or union need to be involved? Other Airlines? Customs and Border Patrol? Operating Environment of AGVT Is any of the operating area under ATC control? Is the operating area designated AOA, secure, tenant security area or other TSA designation? Is the operating area under ramp control? Does the operating environment potentially include interaction with people? Does the operating environment potentially include interaction with aircraft? Will the path of the vehicle vary or always be the same? Will the vehicle operate when aircraft are active? Table 21. Sample characteristics for consideration during evaluation.
