Smart Prosthetics: Exploring Assistive Devices for the Body and Mind: Task Group Summaries (2007)

Neural control of limb movement relies extensively upon the interactions between sensory feedback and motor activation in order to execute functional movement. Sensory receptors are pervasive, and are located in muscles, joints, and skin. These receptors supply information regarding muscle force, length and velocity of movement, joint position, and skin sensation, such as touch, pressure, and temperature. Various forms of paralysis interrupt the motor and sensory tracts, causing not only loss of movement but also loss of perception. Clearly, a sensory loss also occurs with limb loss due to amputation.

A smart prosthetic might be expected to restore both motor and sensory function, giving the user the ability to both perceive limb position and contact, but also integrating this subconscious information into the actual control of the prosthesis (or neural prosthesis). Current limb prostheses and neural prostheses have initially focused on the motor elements, either through the powered component of the prosthetic or the electrically stimulated muscles, or the control aspects, for example via myoelectric control or physical movement that is sensed. However, there is considerably less attention paid to the sensory aspects. To restore complex functions will require delineation of the specific information that is required, determining how

such information will be acquired, and determining how it will be utilized both in providing internal feedback in the control of the prosthesis and in providing perception to the user.

• What kind of information should be acquired in order to provide enhanced performance of the smart prosthetic? Smart prosthetics for various levels of dysfunction might require different degrees of feedback, and upper extremity applications might be considerably different than those in the lower extremity. What sensory information is necessary and sufficient for each clinical application?

• How do different prosthetic systems alter the type of information that is required? For example, are there fundamentally different control needs in an artificial limb prosthesis (i.e., attached to the person directly) than an unattached neural prosthesis (i.e., a robotic limb) for restoration of movement?

• What are promising sources of the necessary sensory signals, and how might they be obtained? What technologies will be required to acquire these signals? What are the practical challenges in introducing this into a wearable prosthesis, and how will these challenges be met?

• What type of information will be most useful to the user in improving his/her performance in using the prosthetic/neural prosthetic? How little information will the user require?

• How will sensory information be provided to the user, and how will he/she not be overtaxed by interpretation of the information to provide true sensory-motor integration in the control functions?

Abboudi, R., C. Glass, N. A. Newby, and W. Craelius. 1999. A biomimetic controller for a dexterous hand. IEEE Transactions on Rehabilitation Engineering 7:121-129.

Riso, R. R., A. R. Ignagni, and M. W. Keith. 1991. Cognitive feedback for use with FES upper extremity neuroprostheses. IEEE Transactions on Biomedical Engineering 38: 29-38.

Sabolich, J. A., and G. M. Ortega. 1994. Sense of feel for lower-limb amputees: A phase-one study. Journal of Orthotics and Prosthetics 6(2):36-41.

Scott, R. N., R. H. Brittain, R. R. Caldwell, A. B. Cameron, and V. A. Dunfield. 1980. Sensory-feed back system compatible with myoelectric control. Medical and Biological Engineering and Computing 18:65-69.

Shannon, G. F. 1979. A myoelectrically controlled prosthesis with sensory feedback. Medical and Biological Engineering and Computing 17:73-80.

Sinkjaer, T., M. Haugland, A. Inmann, M. Hansen, and K. D. Nielsen. 2003. Biopotentials as command and feedback signals in functional electrical stimulation systems. Medical Engineering & Physics 25(1):29-40.

Van Doren, C. L., R. R. Riso, and K. Milchus. 1991. Sensory feedback for enhancing upper-extremity neuromuscular prostheses. Journal of Neurological Rehabilitation 5:63-74.

Summary written by:

Tom Zimmerman, Graduate Student, Grady College of Journalism and Mass Communication, University of Georgia

Task group members:

• Eric Altschuler, Instructor, Physical Medicine and Rehabilitation, University of Medicine and Dentistry of New Jersey

• Gary Berke, President, American Academy of Orthotists and Prosthetists

• Daofen Chen, Program Director, Systems and Cognitive Neuro-science, National Institute of Neurological Disorders and Stroke

• Joseph Francis, Assistant Professor, Physiology and Pharmacology, State University of New York Downstate Medical Center

• Brian Hafner, Research Director, Prosthetics Research Study

• Stuart Harshbarger, System Integrator, The Applied Physics Laboratory, Johns Hopkins University

• Jack Kotovsky, Engineer, Meso Micro and NanoTechnology, Lawrence Livermore National Laboratory

• Cato Laurencin, University Professor, Department of Orthopedic Surgery, University of Virginia Health System

• Gerald Loeb, Professor, Biomedical Engineering, University of Southern California

• Mahesh Mohanty, Project Manager, Advanced Technology, Stryker Orthopaedics

• Marcia O’Malley, Assistant Professor, Mechanical Engineering and Materials Science, Rice University

• Steve Potter, Assistant Professor, Laboratory for Neuroengineering, Georgia Institute of Technology

• Joseph Schulman, President and Chief Scientist, Alfred Mann Foundation

• Dennis Turner, Professor of Neurosurgery, Neurobiology, and Neuroengineering, Duke University Medical Center

• Tom Zimmerman, Graduate Student, Grady College of Journalism and Mass Communication, University of Georgia

The loss of a limb and all its functions is the devastating and inevitable first consequence of amputation. In recent decades, however, science, medicine, and technology have become increasingly good at crudely replacing the physical limb itself. And while a new prosthetic hand and arm make it possible to pick up a cup of coffee, what’s missing is the sense of where the artificial limb is in relation to the body, whether the cup is hot or not, and whether one’s grip is coming perilously close to slipping or shattering the cup.

Our group was charged with identifying ways that prosthetic devices might be improved, particularly in restoring sensations that were once generated by signals transmitted from sensory receptors in muscles, joints, and skin. If such perceptions could be mimicked and coupled with systems for controlling movement, it would be possible to create prostheses that deserve to be called “smart.”

Our task group brought diverse talents to bear on this challenge, as it included experts on physical rehabilitation, neuroscience, physics, orthopedics, biomedical engineering, mechanical engineering, and materials science—to name only some of the disciplines represented.

On the first day, hours of spirited discussion yielded a concise statement of the group’s task: “designing/developing/implementing a prosthetic/ robotic limb with a sufficient level of sensory restoration/feedback to achieve functional manual control.” The group defined sensation broadly, including all “somesthetic” modalities, such as vibration, posture, movement, touch, and temperature sensing. This sensory feedback should provide input for restoring manual control, which will be considered successful and fully functional when sensation, motor control, and perception are integrated.

Although there are many types of amputations and catastrophic injuries that disrupt sensation, the group decided to focus on three groups of patients needing upper extremity restoration: amputees who need replace-

ment limbs, stroke patients and others who need exoskeleton structures to restore lost sensory and motor function, and those who need augmentation of function, such as an industrial application.

With these types of upper extremity problems in mind, the group drew a time line for developing new sensory restoration technology. The group identified three milestones on the path to their ultimate goal: creating systems that are better than current prosthetics, as good as normal limbs, and, finally, even better than unaugmented normal limbs. Users should experience an increase in functionality rather than perceived complexity over time; as Antoine de St. Exupery wrote in 1940, “Technology progresses from primitive to complex to simple.”

Armed with a definition of the problem and milestones for development, the group laid out a structure for its solution. The status of research was summarized in three vital areas: sensory acquisition, sensory perception/feedback, and control. Within each the members of the group focused on identifying current technologies that will enable near-term goals, the evolutionary path of prosthetics, and areas of research that should be explored to reach long-term goals.

They began with strategies for acquiring peripheral sensory signals. Tactile sensors may be used to measure parameters of a contact between the prosthetic device and another object. These sensors may be biomimetic, capacitive, Micro-Electro-Mechanical Systems (MEMS), or quantum tunneling composites (QTC). Proprioceptive sensors, such as shaft encoders on the robotic limb or MEMS gyros and accelerometers, are essential for letting the brain know where the artificial limb is in relation to the body. Finally, there should be some means of communication, whether wired or wireless telemetry, to transmit sensory information to the system that controls movement of the prosthesis and to the operator.

Second, the group considered various strategies for presenting sensory information to the brain, either at the level of the central or the peripheral nervous system. Implantable electrode arrays were considered, but members were concerned that these might be mechanically unstable and might cause fibrosis. Targeted reinnervation was also discussed, but the group was divided about whether experimental surgeries were within the boundaries of the task we had set ourselves. The concept of sensory substitution was discussed, in which sensory data can be redirected from a damaged sensory site or modality to an intact one. There was some enthusiasm for the possibility that patients would obtain improved sensory feedback from residual

sensation in the stump if the fixation of the prosthesis to the stump were improved (e.g., by mechanical integration into the bone).

Finally, different ways of controlling movement of the prosthesis were considered. One is a subconscious local control loop in the prosthesis itself, which is fast and reflexive but offers limited function and little perception. Putting the human directly in the loop, either instead of or in addition to the local loop, ought to make the prosthesis more adaptable, more easily customized, and thus more likely to be functional for patients and be accepted by them.

Having determined a structure for solving the task, the group turned to three gaps in current scientific knowledge and technology that need to be addressed. First, research is needed to learn which sensor technologies are best suited to prosthetic devices, which need cutaneous sensing of variations in pressure, temperature, and texture, as well as proprioceptive capacity. Group members agreed that there was a desire for implantable, low-power, high-bandwidth sensors that acquire simultaneous data in multiple modes.

A significant challenge is that providing conscious sensation is likely to feel invasive to users, depending on the type and location of the interface. If the sensory units are invasive, patients are more likely to accept them if they are also highly functional, long lasting, and reliable.

The second gap in scientific knowledge is that researchers are not certain as to ideal techniques to deliver sensory data from the prosthesis to the body to achieve optimal control of movement. Here the group was excited about numerous avenues of research. These included using sensory translation and sensory substitution to improve cutaneous representation. Intervention at the central nervous system level, such as brain implants with more sophisticated coding methods, is also of great interest. And whether the interface is in the brain or in the peripheral nervous system, mechanical interfacing that accommodates motion relative to the recording site is a challenge. As one group member stated, directly stimulating the central nervous system is like “throwing big boulders into the mainframe of a computer and trying to control it.”

The group developed the idea of muscular proprioception, in which muscle power would be utilized mechanically and taking advantage of the built-in proprioception of muscles to provide feedback to the wearer. Further, the group realized the need for further psychophysical experiments to understand which types of feedback required conscious perception.

In terms of managing controls at all levels, the group posited utiliza-

tion of the unimpaired limb as a controller as in teleoperation. Also, use of transmissions tied to muscles instead of relying on electromyography (EMG) signals would create better resolution. Finally, supraspinal interfaces for sensory input to the brain could be used, integrating it with existing methods for motor output from the brain. Group members then constructed a taxonomy of key control loops.

• Local smart—reflexive adjustment of the actuators in the prosthesis;

• Fast human loop—reflexive/spinal cord (brain is able to modulate the loop);

• Cerebral loop—for dexterous manipulation;

• Slow human loop—supervisory control, internal model adaptation; and

• Customization loop—based on user wearability, patient acceptance, training.

From this taxonomy the group identified several questions. First, what degree of local smarts is needed or beneficial to reduce computational and bandwidth load upstream? Second, how does one remain aware that they are grasping something in their hand via feedback? Also, how do you couple feed-forward and feedback control of prosthesis with adaptation to accommodate external system dynamics?

The group considered how much technology could be included in a prosthetic without making the device horribly complex and unfriendly to the user. First, the device must be easy to attach and align, especially for bilateral amputees. The interface and attachment materials will undoubtedly be very sensitive, which will lead to the potential for damage. The long-term stability of sensory input must also be examined. What cognitive demand will the smart prosthesis cause? Group members have debated just how smart is smart, and what is still required of the user? Finally, considering that many people have a low tolerance for learning to use their new VCR or cell phone to its fullest capacity, just how high will the tolerance be with this new “gadget”?

With all emerging technologies and scientific breakthroughs, there are certainly ethical and socioeconomic implications. Developing a smart prosthetic limb is no different. Who will fund the research and development of these new prostheses, particularly if the market size is small? As new technologies emerge, will they be available to all users, no matter their socio-

economic status? Will increased function require increasingly invasive interfaces, and will this lead to an increase in “designer surgeries?” Finally, if prosthetics one day do prove to be better than the average limb, will users with no medical need opt for voluntary amputation in order to seek increased function? Will the ability to interact telerobotically with a manipulandum that is remote from the operator provide desirable functionality?

The idea of a voluntary amputation is one that many group and audience members found unnatural, but compelling for argument’s sake. It is representative of a best-case scenario as defined by the group to allow a hypothetical teasing-out of ideas. This task group and the Keck Futures Initiative allowed experts from many disciplines to convene for eight hours of spirited discussion and occasional sparring, thinking big, and taking risks in solving a common problem. For now, however, attention will turn back to the first milestone on the timeline for solving the problem: constructing prosthetic devices that incorporate more sensory perception than those available at the present.