On day three of the workshop, participants were placed into small groups and given the opportunity to revisit previous scenarios for a second round of analysis. Loosely following the DARPA Grand Challenge competition, each group developed a 10-year research proposal on a topic of their own choosing, using two of the earlier scenario discussions as a starting point: Hospital Service Robotics and Preparing For and Managing a Major Disaster. In addition to receiving an unlimited research budget, each group was obligated to rely on the actual expertise of its members. For example, if a group did not possess a natural language expert, its delivered system could not employ sophisticated or innovative natural language.
Group 1: Disaster Management System for a Collapsed Urban Hotel
Moderator: Alex Morison
Group Members: Paul Maglio, Alex Morison, Don Mottaz, Gopal Ramchurn
The moderator, Alex Morison, spoke on behalf of the group. Based on the large-scale volcanic eruption scenario, he discussed the group’s development of a Disaster Management System for search and rescue efforts following the collapse of an urban hotel. As a result of the collapse, people are believed to be trapped in the rubble within contained cavities that are not navigable by humans or dogs. The group’s system would make effective use of robots to map cavities within the rubble (for size, location, interconnectedness) and coordinate the exploration. Key technological challenges include: mobility, structural stability, communications, environmental awareness, multi-robot coordination, and “big data” sense making. The system’s design considerations would very likely include both staged rescue scenarios and real-world rescue efforts with actual rescue personnel.
To provide the mobility necessary for such a system, the group proposed the design of a crawling robot composed of multiple modular sensor units. This “slug-like” robot would consist of a series of sensor arrays; for example, one module might be an antenna system to improve communication capabilities.
In addition to navigating confined and unstable spaces, multiple robots—each with different perspectives—would provide improved spatial awareness; new reasoning functions could provide 3-D mapping capabilities.
Morison noted that the system’s success will depend on how well multiple robots can work together as a team. In fact, the group identified multi-robot teamwork as the group’s most significant challenge, citing the current lack of breakthroughs in communications protocols and multi-agent coordination. Effective multi-robot coordination becomes especially critical in post-disaster environments that are often resource limited and unpredictable. Under some circumstances, humans would assume a larger or primary role in coordination efforts—for example, under system failure or when human expertise is required. In cases where humans and robots share responsibilities, automated reasoning would be combined with human reasoning.
Lastly, the group observed that as robots develop increased autonomous capabilities, there may be a push for increased autonomous decision making. The group questioned what if anything might limit such autonomy. For example, what ethical considerations exist for human robot rescue teams (with varying various levels of autonomous capabilities) that triage lost or injured individuals?
Group 2: Team Clean
Moderator: Michael Beetz
Group Member: Michael Beetz, Andreas Hofmann, Mark Neerincx, Liz Sonenberg
Michael Beetz, the moderator, provided a summary of the group’s discussions. Beetz indicated that the group focused its efforts on designing a home robotic cleaning team, “Team Clean,” composed of multiple machines (e.g., humanoid robot, vacuum cleaner, small UAV to “map” the environment), and potentially a human director. The team would be capable of accomplishing a number of tasks with varying degrees of difficulty, from cleaning bathrooms to washing dishes, vacuuming, and doing the laundry.
To do this, a number of research challenges would be addressed, including: practical task manipulation (e.g., picking up fragile objects), smooth locomotion and navigation in a dynamic environment (e.g., going up stairs and opening doors), safety (e.g., not getting in the way of residents or pets), human-robot communication, and social robotics. In addition, machines would have to be able to learn and recover from mistakes and possess sufficient knowledge intensiveness (e.g., to go from an abstract task “to clean up” to understanding how clean is “clean enough”).
Some tasks, Beetz acknowledged, would require varying degrees of interaction between machines and residents. In some cases, a robot may request feedback from the resident. For example, a robot might ask whether a dirty glass
situated near the resident is currently being used or in need of washing. In other cases, the system should be adaptable if it is re-tasked by the resident. This could occur if the resident’s cleaning expectations differ from those of the robot. The robot may also need to resolve conflicting resident demands—for example, balancing a parental request to “pick things up off the floor” and a teenager’s request to “leave the bedroom as it is.” To deal with this situation, the group proposed a system with one “chief,” as well as an organizational structure to deal with conflicting goals.
Lastly, the group identified performance evaluation as a significant element of the system. For example, how many tasks were accomplished and in what time frame? How well did the team function? As the team evolved, the system would be scalable to accomplish a wider range of tasks.
Group 3: Biped Hospital Companion Robot
Moderator: Candy Sidner
Group Members: Robert Hoffman, Lakmal Seneviratne, Candy Sidner, Rong Xiong
The moderator, Candy Sidner, provided a description of the group’s proposal for creating a biped hospital companion robot (based on an earlier discussion of hospital service robotics). This robot would undertake personal care activities (e.g., dressing and bathing patients and picking up laundry) and provide mobility/balance support by preventing mobility-related accidents and catching patients who are falling. In addition to physical manipulation requirements, some basis for human-robot communication is required and humans need to be comfortable receiving robotic assistance. For this reason, human-robot trust is an important systems requirement.
To accomplish these tasks, the proposed biped robot would be designed with articulated, touch-sensitive hands and somewhat soft bodies with suitable, nonaversive “skins.” In addition, visual recognition would be integrated with touch and task-manipulation capabilities. Communication between the robot and patient would be computer-controlled and employ simple dialogue—for example, questions that can be answered with a “yes” or “no” or with a very short statement.
Sidner added that algorithms, such as those used to predict the movements of rapidly traveling Ping-Pong balls, would be tuned and applied to predict when a human is falling and to respond appropriately. This would require the robot to distinguish not only between types of falling (e.g., falling while conscious or unconscious), but also between similar actions (e.g., falling versus bending over to pick something up). For some frail individuals, the group noted, the line between falling and bending over is thin. By merging robot companion and robot assistant technologies, the robot could also act as an instructor or coach for patients. For example, a robot that has learned to balance itself could
not only help feeble patients cross hospital floors but could also act as a physical therapy coach.
As a part of its design, the group would also develop a number of testbeds to assess both communication and trust issues, as well as appropriate and safe interactions between robots and patients. Questions addressed would include: How does the nature of human-robot communication and interaction change when robots are working with patients who may be sick, feeble, or physically or cognitively impaired? How anthropomorphic should a robot companion be, and should it be more or less anthropomorphic if it is engaging in a conversation or dressing/undressing a patient? How might robot companions work with other robot companions in this environment?
Group 4: The Robotic Patient Advocate
Moderator: Michael Freed
Group Members: Michael Freed, Yukie Nagai, Jean Scholtz, Satoshi Tadokoro, Manuela Veloso
The moderator, Michael Freed, spoke on behalf of the group. Using the medical service robots as a starting point, Freed described the group’s proposal for creating a robotic patient advocate that would work either as an intermediary between the patient and hospital staff or directly with patients. The robotic patient advocate would keep nurses up-to-date (e.g.., monitor and report changes in patient physical or emotional states), provide continuity when nurses change shifts or when patients are assigned new doctors, and communicate with nurses when patients are asleep or unable to effectively communicate. In addition, the advocate would directly provide information to confused or forgetful patients (e.g., asking “Why am I being wheeled to Room 108?” or “Have I taken my medication already?”). The advocate would also support medical staff when the patient required encouragement. Lastly, the advocate would run interference with visitors and people who stay too long or get in the way of medical staff.
As Freed explained, the advocate would leverage the group’s collective experience in autonomous systems, communications and dialogue, human emotion, machine-human interaction, and performance evaluation of both robots and humans. Although some of these capabilities were possible using conventional technologies, six key breakthroughs would be required. (1) Dialog: The advocate should be capable of high-level discussions with people possessing different knowledge, motives, and cultures. (2) Multimodal Sensing: The advocate should be able to tap into—via many and complex sensors—a hospital’s data-rich environments to access a patient’s medical records, real-time physiological conditions, test results, and schedules. (3) Strategic Planning: The advocate should take action by balancing a patient’s immediate goals and requests with longterm patient support that considers legal and safety issues. (4) Safe Navigation: The advocate should navigate a complex environment of constantly changing
people, carts, beds, and equipment. (5) Social Understanding: This might require the advocate to know when and how to “shoo” away visitors who are unwanted or have stayed too long. (6) Multi-Persona Negotiation: The advocate should be a middleman or “middlerobot” among a floating team consisting of patients, family members, doctors, nurses, and others.
The group acknowledged that evaluation of the system was critical; thus, the advocate would be developed first with limited capabilities that would be expanded on the basis of experience and learning.
Group 5: Providing Post-Disaster Basic Services
Moderator: Lin Padgham
Group Members: Tal Oron-Gilad, Lin Padgham, Dirk Schulz, Holly Yanco
Lin Padgham, the moderator, provided the description of the group’s proposal. As one component of the volcanic eruption disaster-management scenario, Padgham described the group’s design of an information management system to provide basic services, such as communications, food, power, and water, in the first week following a major disaster. The system would not provide total coordination across the entire disaster management value chain, but rather would provide on-the-ground individuals with decision support.
Such a decision-support system would require data inputs from numerous sources, including cell phones, sensors, weather reports, and UAVs. The system would take in data in a variety of formats and then organize and share those data with a range of specialized users. For example, data inputs from UAVs that show downed power lines could be used to coordinate prompt robot deliveries of electrical and other power sources to neighborhoods lacking electricity. Effectively distributing and acting on this information will require simple yet specialized human-machine interfaces.
Ongoing access to massive amounts of parallel data would allow management officials to better prioritize their attention and efforts—for example, whether to immediately evacuate a neighborhood or to first restore basic infrastructure. Padgham added that the system could be used as a simulation tool in advance of a disaster to improve emergency management response. By assessing the efficacy of different communications protocols and of evacuation routes under different environmental and social circumstances, authorities can identify where critical post-disaster response failures are likely to occur.
The system would also make “individualized” information available to both specialized users (e.g., UAV operators with specific data needs to survey for downed power lines) and to untrained users who are stranded in their homes with limited food and water. Although acknowledging that such a system would provide complex decision support, the group noted that human judgment will always remain key.
