Following his introductory remarks, John Paul Strachan, Hewlett Packard Laboratories, moderated a discussion among the workshop speakers along with Ivan Schuller, University of California, San Diego. The discussion proceeded with Strachan posing questions to the panel that he proposed and that were received from audience members, and then one or more panelists taking turns providing answers.
Q1. Dr. Strachan: What is the role of memristors within the spectrum of current and emerging technologies?
Strachan began the panel discussion by asking panelists what they believe to be the role of memristors within the spectrum of current and emerging technologies, such as complementary metal-oxide-semiconductor (CMOS) switches, flash, nanotubes, graphene, photonic sensors, and so on. Williams responded that memristors are a different level than any other devices. Memristors are best conceptualized as a mathematical model providing guidance. The guidance is high level, and it points to the need to devise devices or circuit elements that enable nonlinear dynamics and then use those for computation.
It also points to a materials issue. On the one hand, memristors can be made out of many materials, including materials on the presented list. But whether the best material is carbon nanotubes or graphene or something mechanical is yet to be determined. Complexity, nonlinear dynamics, and chaotic response all must reside on some sort of platform. Remaining agnostic initially regarding materials or physical implementation prevents prematurely selecting the wrong platform.
Schuller noted that the development of the digital computer involved a choice at some stage regarding transistors, their use, choice of materials for solid-state transistors, and fabrication methods. Settling those issues shifted the development paradigm. The settling probably happened by accident, at least in part, amid many potential options. At this stage of memristor development, the community should avoid restrictions on investigating any particular path. Schuller suggested that we let a thousand flowers bloom and see what happens.
Q2. Audience, Bill Carter, Defense Advanced Research Projects Agency: Beyond cognition and learning applications, what other neuromorphic functionality could memristors support considering that neurons support many functions in nature?
A. Alec Talin, Sandia National Laboratories, responded first, commenting that allowing for brain-computer interfacing where machines and biological systems can communicate is leading to an emergent paradigm of how humans interact with biology. While this may have a restricted range of useful materials and devices, it opens the possibility of understanding how biological systems compute and also how to enhance and sometimes correct issues—clinical issues, for example.
R. Stanley Williams, Texas A&M University, observed that memristors make ideal sensors, especially for temperature, pH, and nervous system activity. They are well-suited to environmental sensing because of their ability to transduce, extraordinary sensitivity, nonlinear characteristics, and very large dynamic range.
Schuller added that sensors may allow for a shift in computing whereby sensory input will teach the machine somehow, perhaps in a way analogous to how humans learn. Psychology suggests that human intellectual development is contingent on sensory input. Analogously, sensing, local computing, and global computing may be very important for machine intelligence.
Strachan opined that sensing may be replacing programming in the emerging paradigm.
Q3. Strachan: What are the priority challenges that R&D programs need to address from the materials physics and the device perspective?
Marek Skowronski, Carnegie Mellon University, commented on the importance of Strukov’s point regarding device variability. At present, devices vary greatly, are frequently out of specs, and evolve over time, so they can behave in unpredictable ways. While most devices have long retention times, the fact that some evolve faster than others makes it difficult to build a platform with established standards.
The first goal, then, would be to establish standard fabrication and testing procedures. This can enable fabricating devices with reliable standards for predictable behavior.
Next, memristors reduce the need to move data as much as transistors, but they exhibit about two orders of magnitude higher energy dissipation per operation. Memristors, then, can help overcome the current bottleneck of data movement. However, that will likely bring the industry to face the energy-per-operation bottleneck.
This is true at least for two-terminal devices. Skowronski invited Talin to comment on three-terminal devices. Talin finds the ability to change bulk properties and not rely on high-current, high-voltage activation encouraging.
The capability to investigate phenomena in the context of carrying out computing is also important, since very often devices are tested out of context. That leads to speculation regarding how properties are actually useful for computing. Expanding the ability for developers to test primitives in the context of at least simple computations will help with understanding which computations make sense.
Doing that will help elucidate how neural networks work in general, and also hopefully allow developing new algorithms that can utilize this myriad of interesting phenomena that right now have no useful application.
Q4. Audience, Wei Yi, HRL Laboratories: What is the equivalent of the Turing machine for neuromorphic computing? Is the lack of a machine model a major bottleneck to overcome? If yes, how can the bottleneck be overcome?
Catherine Schuman, Oak Ridge National Laboratory, does not believe that there is a good equivalent nor should one be developed at this point. Taking a canonical approach would be dangerous, since it would reduce innovation with potential algorithms. Materials and models are analogous. The best algorithms, materials, and models still remain unclear. Rather than selecting ones now and constraining future development, there should be opportunity to keep experimenting and researching to discover.
Schuller then asked whether Schuman knew of any attempts to map high-level mathematical models for computing, such as hyperdimensional sparse coding, onto an actual machine. Schuman did not know of any such efforts.
Q5. Strachan: If memory, especially binary program memory, is driving commercial production, how can memristors for computing be developed? Can they piggyback or must there be entirely new optimizations?
Dmitri Strukov, University of California, Santa Barbara, does not believe that selectors are needed in neuromorphic accelerators, so there are probably important differences. But what can drive all approaches is learning how to improve yield and reduce variations, and investing in infrastructure that allows easily comparing new materials of similar composition. This would allow researchers simply to adjust the recipe rather than needing to integrate completely new materials and steps in the process.
Talin mentioned that Intel and Micron Technology produced a memristor-based memory, XPoint memory.
Financial investment remains a significant factor in developing any new technology in a mature industry. Memory manufacturers may not take on a new, promising technology because of investments they have already made. Thus, even promising technologies may not make it to market.
Q6. Audience, Hans Cho, Naval Research Laboratory: Building on Talin’s statement, how much do financial considerations of commercialization drive device development? Does that mean that remaining agnostic regarding materials may be unwise?
Strachan added to the question, asking whether constraints such as materials properties or fabrication facilities’ capabilities may further constrain commercially available devices so that they may not be able to empower the applications discussed during the workshop, or if algorithm modification could accommodate any of those constraints.
Williams agreed with Cho. Private industry will undertake any scaled production of devices. That will present new challenges that only industry, not academic or government lab researchers, will be able to solve. One reason is that only industry has foundries.
The ecosystem approach explained by Clayton Christianson seems correct. Attempting to introduce a new technology and compete at the very top end of any product ecosystem is guaranteed to fail. New technologies can best compete first in niche areas, where they can solve an important problem that has not been solvable with currently available technology. Probably a clever group of researchers will observe the possibility, start a company, and fabricate new devices just for that problem. From there, an ecosystem can begin to grow.
This tends to start with one to three niche applications that other technologies cannot run, or not efficiently. The history of transistors developed along this path. In the 1960s, vacuum tubes dominated because of their high quality, and applications remained suited to tubes. When integrated circuits came of age, they provided a solution to building hearing aids, something vacuum tubes could not be used for. Hearing aids became the niche application that opened the way for transistors to solve a valuable problem that vacuum tubes could not.
Talin agreed that investigating in niche applications is crucial. One example now is satellite communication where there is also radiation. The satellite environment is power constrained and benefits from in situ information processing. An inference engine would be ideal. Radiation will probably negatively impact flash because it causes more traps in oxides. Satellite systems are less price sensitive, too. Both of these would give an advantage to memristor-based systems.
Q7. Audience (online): What computer science architecture dynamically performs neurogenesis and also apoptosis?
Schuman has not seen neurogenesis addressed at the architecture level but believes that it will be an important area for learning. It provides an opportunity for innovation, whether that is building adaptable structures into hardware, or building hardware that can be grown, perhaps even literally growable hardware.
Strachan asked about switching from a computer science architecture to a physical substrate. Talin thinks that software-defined architecture may allow this. In those systems, the algorithm can allow remaking connections or changing functionality, from memristor to transistor, for example. Another example would be morphing from one system optimized for a memory-intensive application to one that runs very fast in logic.
Skowronski asked whether there already is work of this type. Talin stated that his group has conducted some work to reconfigure devices during operation. Other researchers at Sandia are investigating reconfigurable electronics. Likely there is published literature from other researchers too.
Schuller noted that there is a lot of work on functional materials—materials that respond in unintended ways. This includes device- and material-level research probably covering properties for functionalities that change with voltage, current, temperature, or light. Complex materials can enable these features, but they also introduce other complications. Some of those may not appear until attempting industrial-scale fabrication, but they will eventually appear.
Q8. Audience in person, Jabez McClelland, National Institute of Standards and Technology: Is there some way that quantum computing and neuromorphic computing may come together in the future, perhaps a neuromorphic system using quantum computers at its core, or a quantum system that uses a neuromorphic algorithm?
Schuller’s research includes quantum materials for energy-efficient neuromorphic computing. Quantum computing and neuromorphic computing are highly distinct. Quantum computing seeks to perform calculations that presumably cannot be done with any other technology. The calculations are very precise and intensive. Neuromorphic computing, on the other hand, is a gestalt thing. It should be able to distinguish the chairs from the people from the computers within a room. That does not require calculating everything to its ultimate precision.
Quantum materials are entirely distinct from either computing paradigm. Quantum materials are complex and feature manipulable internal degrees of freedom. Manipulating these internal degrees of freedom allows making a system
more closely resemble a biological system or biological functionality, where specific components act in different ways depending on how they are manipulated.
Thus, it is important to distinguish quantum materials from quantum computing and from neuromorphic computing. Schuller believes that there is no real overlap in just mixing these things together. Perhaps doing so would ensure that neither of them will ever work.
Q9. Audience, in person: Can the memristor community write some guidelines, or a roadmap, for the different criteria that memristive devices should meet?
Talin recommends using the work of Geoff Burr from IBM Almaden. Burr has spoken at length about the need for guidelines and investigated many different materials. There are some really good publications outlining requirements. Those are very application specific.
Williams recognized that the lack of guidelines or a roadmap creates a problem because their absence leaves a gap in specificity to guide researchers. It would be hard to make a comparison to the international technology roadmap for semiconductors because semiconductors development had become well defined by the time ITRS started. At present, memristors are at about the equivalent of 1956 or 1957 for transistors, when it was unclear whether transistors would be made from germanium or silicon.
Additionally, the range of devices and applications present difficulties for authors and reviewers. Authors need to be as clear and reasonable as possible, and referees need to be reasonable.
Skowronski agreed that the technology is still in early development, yet there is a difference because there are functional examples at present implemented in silicon. While this raises some difficulties, it means that there are benchmarks for comparison. For instance, Strukov presented many examples showing memristors to be significantly more energy efficient than silicon. The community probably should be more aware of what is possible in silicon and should be paying more attention to it.
Schuller agreed that the community should be more aware of developments in silicon devices given the amount of ingenuity and intelligence going into silicon research and development. New materials for memristive devices are still at a very early stage.
Regarding benchmarks, bear in mind that each time a publication includes a benchmark, other researchers will object that it does not include certain other considerations. This is because of the nascence of the field.
Strachan questioned whether this is why Schuman’s survey found that 73 percent of publications do not mention applications.
Strukov countered that if a researcher can demonstrate a very compelling case such as making a system-level comparison, even without yet knowing how to implement it fully, they could still show how much improvement it could obtain on a specific metric. This will help authors avoid unnecessary criticism, as digital implementations are the main competitor.
Skowronski agreed that system-level evaluation is crucial, because at the device level, improvements do not guarantee system-level performance improvements. Device researchers cannot make that kind of assessment and comparison to digital systems in silicon, yet those are the most valuable comparisons.
Q10. Audience, Raymundo Arroyave, Texas A&M University: How can co-design take place for application, algorithm, device, material, and physics—will that require five different categories of things to optimize or consider? Where should researchers start?
Schuman believes that it is important to pick one place and just start, then branch out from there. For example, working in algorithms, she tries to influence the direction of materials research while also working with the materials available. Keeping abreast of advances at the circuit, device, and materials levels allows co-design, initiating a virtuous co-design cycle that will illuminate bottlenecks and hurdles along with possibilities to overcome them.
Selecting a system to build makes that possible because that dictates what should be optimized. Starting at a final application and trying to work down the compute stack to materials is really difficult. Starting in the middle and learning which way to go from there is more feasible.
Skowronski agreed and added that by the same token it is very difficult to start with a material and then determine what application it suits well.
Talin believes that there are guiding principles of value. For example, analog computing and nonvolatile devices involve moving ions instead of electrons. Materials where ions move have the potential to attain nonvolatility. Another example is to consider redox state density because those enable more analog states per volume, thus making a system smaller and requiring less energy to move an ion.
Guiding principles like these of desired properties for a device or materials are similar to those used in battery research. Utilizing them can be very valuable in elucidating properties to investigate and thus narrowing the range of candidates for investigation.
Q11. Audience, Brian Hoskins, National Institute of Standards and Technology: Besides money or access to user facilities or computing resources, what can the government provide to researchers or businesses in industry that can accelerate research in this area?
Williams believes that validation and excitement would be valuable. The U.S. Nanotechnology Initiative is one successful example. That brought together researchers from a far more heterogeneous set of disciplines into a community that then made a concerted effort to find commonalities and tell a compelling story. The initiative validated the effort, excited the community, and offered legitimacy and branding. Such an effort could work here, too.
Talin believes that facilities for testing of electrical programming and arrays would be very valuable. Array testing requires significant effort. Very few groups have the capability to conduct array testing or do it well. A user facility where researchers can test beyond single devices for computation, without strict limitations on device size, number of contacts, or specific mask required would be quite valuable and contribute to advancing understanding of these devices.
Schuman believes that more opportunities to bring together people from different disciplines and from the entire compute stack so that they engage and arrive at convergence will spur genuine innovation.
Q12. Audience, Nikolay Frick, North Carolina State University: One of the slides mentioned that biological systems learn because they have motivation to survive. Is there any good established method or some guidelines on how we can program this motivation into crossbar arrays of neuromorphic systems to make them self-contained?
Schuman mentioned some efforts around neuromodulation that encourages reward signals and reinforcement learning that includes a motivation component. These are inspired from two extremes, either biologically (or biochemically) or from very high level psychology. As yet, there does not seem to be anything in between.
Q13. Audience, Manoj K. Kolel-Veetil, Naval Research Laboratory: When designing, are the differences in material properties and performance between 2D and bulk production taken into account?
Schuller commented that large facilities play a crucial role in this. Partly, this challenge arises because the ability to characterize smaller and thinner structures is relatively new. Now instruments such as electron microscopes allow for in situ
characterization. However, many of the instruments are too expensive for most individual laboratories to own, so access to them through user facilities is crucial.
Skowronski noted that national facilities do exist, which points to the need mentioned earlier about a user facility for testing arrays. Likewise, there is a great need for integrating memristors and CMOS. That is difficult to do at the single group level. Consequently, government support would be of enormous value, for instance in the form of a facility or platform for testing.
Q14. Strachan: Will incorporating memristors address some of the high-level challenges to reproducing human-level intelligence?
Strachan introduced this question by observing that neural networks need millions of examples to learn, and yet humans—children—learn with just a few examples. He then asked the panelists to consider how humans can transfer learning from one domain to another unrelated domain and avoid catastrophic forgetting.
Williams reiterated that memristors are singular components of a system, while the behavior and capabilities mentioned are higher level. Perhaps a useful system to consider is the dynamic Bayesian network. Theoretically, it possesses many of the characteristics mentioned. These characteristics have not been implemented mainly because of the astronomically large amount of computation required to implement even a single instance. Memristors offer hope of machines that can perform the number of computations required to make a dynamic Bayesian network feasible. If that happens, then it would be possible to investigate whether that network exhibits some of these characteristics.
Talin suggested that the behavior mentioned is emergent, so if a computing system includes enough functionality and connects enough units, then at some point it can be tested to see whether features such as consciousness and intelligence have emerged.
Q15. Audience, Sandra Lindo-Talin: Does a good model exist capable of explaining brain functionality at the basic level?
Schuman believes that there is understanding of very low and very high level functionality, but not functionality between those.
Schuller believes that understanding brain function is not essential to building systems. Rather, systems need only inspiration, analogous to how airplanes are designed to fly using physics in a different way than birds rather than trying to mimic birds.
Comment, from Audience, Manoj Kolel-Veetil, Naval Research Laboratory
Kolel-Veetil offered a comment regarding the nonlinearity of computing and Bayesian statistics. Statistics is an important component for delivering data, especially with nonlinear systems. Embedding statistical methods in nonlinear systems provides more value because these systems have more extrapolative power than the more interpolative nature of current, digital systems. This allows finding new solutions, or making new predictions a priori, which otherwise would not be possible. For instance, when working with materials, such a system could take the five most common phenomena observed and use an iterative, extrapolative algorithm to devise new solutions.
Williams agreed on the value of utilizing Bayesian inferences and noted that there are other useful statistical methods and mathematical frameworks such as Markov chains. From that standpoint, neurophysiology thus remains an important area for investigation.