Smart Prosthetics: Exploring Assistive Devices for the Body and Mind: Task Group Summaries (2007)

Even though prosthetics have come a long way since they have emerged, functional limitations and challenges of directly and reliably controlling them still makes artificial prostheses less than optimal for the satisfactory performance of everyday tasks of amputees. The challenges are mainly due to the range of motions required for satisfactory performance, which calls for a highly complex prosthetic mechanism to impart movements and a complex control interface to communicate with the prosthetic device.

Research performed over the past decade has led to the refinement of prosthetic materials. Materials that are capable of withstanding the physical and mechanical demands of the prostheses to a great extent are now available. Also, recent advances in bioengineering are greatly helping to develop robotic systems that could mechanically mimic many of the functions of the extremities.

However, to efficiently use a functional prosthetic much more control over prostheses is needed. Even though myoelectric prostheses have better control over body-powered prostheses, these devices involve a steep learning curve for the patients to gain conscious control over the weak electric signals. The finest approach to achieve full control of the prosthetic by the

patient is by turning the thought process in the brain into actual physical movements of the prostheses using direct neural interfaces.

Several challenges remain unresolved to develop a practical interactive hybrid brain machine interface (HBMI) to control a prosthetic in real time.

• To control prostheses using HBMI real-time sampling and processing of large-scale brain activity is needed. This calls for the development of novel methods for measuring large-scale brain activity, learning how to sample and decode motor signals and how to feed them into prostheses to mimic the required movement, new techniques for microstimulating neuronal tissue, developments in microchip design, nano- and microfabrication techniques, and further developments in robotics.

• Lack of sensory feedback is another key limitation that seriously hinders the ability of the prosthesis to respond to external environment. The sensory feedback is highly essential to establish a closed control loop between brain and artificial prostheses and is also a great tool to help the patient to learn how to use HBMI’s. However, this needs understanding of where and how to stimulate the sensory nervous system to reproduce the signals that an organ sends to sensory cortex.

• Materials integration is also needed. Interface implants need to be designed that can integrate with host tissue to obviate the need for frequent replacement.

Abbot, A. 2006. Neuroprosthetics: In search of the sixth sense. Nature 442:125-127.

Biddiss, E., and T. Chau. 2006. Electroactive polymeric sensors in hand prostheses: Bending response of an ionic polymer metal composite. Medical Engineering & Physics 28: 568-578.

Chapin, J. K., and K. A. Moxon, eds. 2000. Neural Prostheses for Restoration of Sensory and Motor Function. Boca Raton: CRC.

Nicolelis, M. A. L. 2001. Actions from thoughts. Nature 409:403-407.

Summary written by:

Wendi Zongker, Graduate Student, Grady College of Journalism and Mass Communication Department, The University of Georgia

Task group members:

• James Abbas, Co-Director, Center for Adaptive Neural Systems, The Biodesign Institute

• Farid Amirouche, CEO and President, Ortho Sensing Technologies

• Bradford Bennett, Assistant Professor of Research, Orthopaedic Surgery; and Research Director, Motion Analysis and Motor Performance Laboratory, University of Virginia

• Nancy Byl, Professor and Chair, Physical Therapy and Rehabilitation Science Department, University of California, San Francisco

• Jose Luis Contreras-Vidal, Associate Professor of Kinesiology, Bioengineering and Neuroscience Program, University of Maryland

• Michael Dorman, Professor, Speech and Hearing Science, Arizona State University

• Gary K. Fedder, Howard M. Wilkoff Professor of Electrical and Computer Engineering and Robotics; Director, Institute for Complex Engineered Systems, Carnegie Mellon University

• Brent Gillespie, Assistant Professor, Mechanical Engineering, University of Michigan

• Anne Heberger, Research Associate, National Academies Keck Futures Initiative

• Hod Lipson, Assistant Professor, Mechanical and Aerospace Engineering, Cornell University

• Yoky Matsuoka, Associate Professor, Computer Science and Engineering, University of Washington

• Michael Merzenich, Francis Sooy Professor of Otolaryngology, Keck Center for Integrative Neurosciences, University of California, San Francisco, School of Medicine

• Santa Ono, Vice Provost and Deputy Provost, Emory University

• Kevin Otto, Assistant Professor, Weldon School of Biomedical Engineering and Biological Sciences, Purdue University

• Blake Wilson, Senior Fellow, RTI International

• Wendi Zongker, Graduate Student, Grady College of Journalism and Mass Communication Department, The University of Georgia

Doctors, scientists, researchers, engineers, CEOs, and even a provost traveled from all over the United States to put their different disciplines aside and work together for a common cause. These men and women traveled to the National Academies Keck Futures Initiative Conference held November 8-11, 2006, in Irvine, California, to discuss the future of prosthetics research. Attendees were divided into 11 groups, each with a different task to tackle and only three days and eight hours in which to accomplish that objective. Task group 3 was faced with the challenge of developing a “smart” prosthetic that can learn better and or faster.

It depends on what you mean by “learn.”

That’s what members of this task group identified as the starting point for three days of discussion about developing a smart prosthetic that can learn better and or faster. After a lengthy discussion, members of our group settled on one definition with two components. The first definition postulates that a device has a built-in predictive model that generates output depending on what information is fed into it. This is possible due to a feedback loop, which allows the model to act differently depending on what its previous actions have accomplished. The second definition defines learning as the act of organizing, or reorganizing, neural circuits so one can successfully interpret information and/or external signals and respond appropriately.

After much discussion about the purpose of a smarter prosthetic device, and about what it should and should not be designed to do, our group listed challenges that must be conquered to make it learn faster and better. These include developing a device that can remember what it’s done and analyze how well its actions worked, which implies having recording capacity. The patient’s brain is an integral part of the feedback loop that will help

the device learn, and much less is known about this than about high-tech materials and robotics for prosthetic devices.

As the first day of discussion drew to a close, our group was focused on this question: What strategies best promote learning in both the patient and the prosthetic device? Participants agreed that motivation, repetition, progression, surprise, feedback, reward, and attention all promote learning, and they resolved to figure out which of these can be automated to maximize learning by client and machine.

On the second day of the conference, our group had only two densely packed hours to solidify our definition of learning, consider how brain and machine should share the learning, begin considering what different customers will require of their prosthetic device, and decide what will be required to accomplish our assigned task.

After much discussion, the group agreed that learning is the method of reorganizing human neural circuits in the prosthetic device to interpret signals and generate cognitive/motor outputs. As circuits are reorganized, internal predictive models are adjusted and the planning and control of actions change. This happens on both the human and machine sides of the transaction, and the two collaborate and interact as they gain experience. The learning is ongoing and can result in bad learning or learning the wrong things.

The discussion quickly turned to the wisdom of trying to develop the smartest possible prosthetic that would shoulder most of the responsibility for translating thought into action. Some participants thought it would be ideal to have a device that became increasingly competent with experience and over time permitted the brain to become relatively “dumb.” Others argued that it would be better to have the prosthetic back off over time, letting the human brain eventually do almost everything. If we can determine a way to connect the prosthetic to the human brain with enough connections, the prosthetic can become like a real limb—or dumb. This preferred option is unlikely to occur in the initial design, therefore, rather than beginning with a dumb prosthetic, smarts will need to be built into the device to interpret the limited transfer of information between the pros-

thetic and the human brain. Abbas summed it up by stating that “we want a prosthetic that is smart enough to do some of the job but not smart enough to do everything.”

The discussions between developing a dumb prosthetic and a smart prosthetic included an idea among members of the group to develop a device that begins with reduced degrees of freedom, enabling simple operation by the user. Over time and with learning, complexity of the device will increase by releasing, or increasing, the degrees of freedom of operation as the user learns to use the prosthetic. This method of operation parallels how a human initially learns to use a motor task. A baby learning to walk looks stiff and unsure, as there are reduced degrees of freedom. As the baby grows and learns the skill of walking involves more degrees of freedom and therefore looks more fluid and smooth.

The group discussed various tasks that the prosthetic device should be able to do, which led them to consider how customer requirements will vary. The prosthetic will only be as good as its capacity to satisfy individual users, they decided, and one person might want to play basketball while another would be more than satisfied to simply walk again.

On day three the group defined learning issues for the client, the prosthetic device, and the training program, which includes hardware and software.

For the client, learning issues include training strategies with neurophysiologic foundations, providing sensory stimuli, and monitoring the learning progress. Learning issues of the prosthetic device include machine learning, sensors, and interface. The learning issues of the training program consist of progressive challenges and performance metrics.

In the wake of traumatic injury or illness that make a prosthetic limb necessary, patients mainly want to overcome pain and disability and regain some measure of independence and mobility. They expect to get better, but adapting to a prosthetic device is a lengthy and difficult process. To succeed, group members agreed that patients need to be motivated and need to be rewarded for progress. The learning will be biochemical and genetic and will involve active exploration and repetition in the form of progressive challenges. For learning to occur in the client there must be a realistic delineation of expectations and the client must also be able to adhere to a training regimen. Exploiting brain plasticity and richness of the interface

between human nervous system and machine are also key issues to explore when dealing with learning occurring in the client.

Knowledge and/or technology gaps exist in regard to learning occurring with the client. These gaps include the know-how to transmit information to and from the brain, specifics regarding sensory and motor representations used in the brain, mechanisms of plasticity, and how to maximally exploit the plasticity.

The key features of the newly developed smart prosthesis should include the basic functions of safety, consistency in performance, reliability of use, and independence of use. The device should also have a bidirectional information flow between the user, or the human brain, and the prosthesis. There must also be an ability to provide rich sensory information, such as force, posture, velocity, temperature, vibration, time, and direction, and the device should also integrate the sensory information with motor outputs. The device must be dynamically adaptive to the client and incorporate contextual information flow and should employ multiple motor outputs to enable a rich repertoire of tasks. This involves gradually enabling the degrees of freedom, as discussed earlier, and the complexity, as well as enabling posture balance, stability, and movement. A vital element of the device is a high-fidelity interface that can imply dumb prosthesis when operating with an enabled brain. The device will incorporate the ability of “amazondotcomification,” or the capability to learn what the user wants and/or needs. High bandwidth and versatility is another key feature. Finally, this device must be able to anticipate and inhibit decrements in the interface, for example, scar formation.

The group identified knowledge and/or technology gaps that inhibit the development of these key features in the prosthetic device. For instance, there is a need for biomorphic sensor/actuators to ensure compatibility with neural representations. Also, to create this device we must know how to maintain and improve performance based on interface. The group decided that task groups 4 and 6 would address this need for further research (see the Brain Interfacing with Materials and Structural Tissue Interfaces sections for more information on these groups). Research is necessary to find out how to detect and communicate user intent and motor commands and to establish machine learning techniques that are appropriate for real time adaptation. Our group also decided that there is a need for research on redundancy of learning and how to exploit it for versatility and efficiency.

The training programs employed must be fun, exciting, engaging, and easy to use to help promote learning by the user. They should have a design

that is based on principles of cognitive neuroscience. Repetition is needed while also incorporating progressive challenges. Also at issue is the quality of information involved with these training programs and the immediate feedback in regard to reward and errors. These programs must be task specific and client specific and include the ability to be modified by the trainer. The method of training should also include practice spaced over time and should be available and accessible to the client.

Knowledge and/or technology gaps in creating the ideal training program include the current inability to identify intermediate performance and neurophysiological milestones. There is also no way to determine the individuality of minimal detectable differences or the know-how to customize the prosthetic device for a specific user group that may have certain needs based on age, gender, or culture.

After laying out key features and issues and determining the necessary knowledge and existing technology gaps to make these ideas a reality, our group moved on to prioritize the research needs and lay out a research agenda for the future. These research priorities were divided into the client, prosthetic device, and training program groups.

Research is necessary to determine how the brain learns to handle rich sensory inputs as well as how the brain translates user intent to motor action for increasingly sophisticated function. To develop this prosthetic device, research must be completed to determine a way to access user intent and then utilize that access effectively. The idea of machine learning in a co-adaptive setting must also be explored. In regard to the training programs, research should and must be conducted to learn how to maximize progressive learning.