Frontiers in Memristive Materials for Neuromorphic Processing Applications Proceedings of a Workshop (2021) / Chapter Skim
Currently Skimming:
2 Workshop Plenaries
2 Workshop Plenaries
Pages 3-62
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 3... ...
Dmitri Strukov, University of California, Santa Barbara, covered device requirements and challenges of mixed-signal neuromorphic hard ware. Catherine Schuman, Oak Ridge National Laboratory, provided an overview of device architectures to meet the challenges of neuromorphic computing.
|
From page 4... ...
Origins and Bases for Neuromorphic Computing For nearly 100 years, developers have pursued a significant range of ideas for computing with neural networks, known as neuromorphic computing. According to Williams, all ideas for neuromorphic computing essentially derive from the work of Ivan Petrovich Pavlov and W
|
From page 5... ...
Memristor Devices Williams explained two types of memristive devices of interest in computing. One type is nonvolatile and could be called "synaptic" because it shares proper ties of synapses with the ability to store the state or state variable as resistance in the device.
|
From page 6... ...
Devices suitable for building locally active memristors can be based on Mott insula tors, dielectric matrices with mobile ions, coupled redox reactions, charge density waves, and almost the entire palette of correlated electron materials. Memristive Device Capabilities Williams explained that one memristor implementation for computation a cceleration is the crossbar, sometimes called a dot-product engine.
|
From page 7... ...
Another example computing system mentioned by Williams is an electronic Hopfield network built by John Paul Strachan and collaborators at Hewlett Packard Enterprise, which has been applied to NP-hard (non-deterministic polynomial time hardness) optimization problems such as "max-cut." Optimization routines can become stuck in local minima when searching for the global minimum because they search stepwise from the current location by testing adjacent locations without the ability to see any more of the surface.
|
From page 8... ...
This results in a thermal runaway process that is moderated by Newton's law of cooling, which states that hotter systems lose heat more rapidly. Thus, tem perature becomes the state variable for the system, and the coupled equations for 3D Frenkel-Poole tunneling and Newton's law of cooling result in the highly nonlinear and dynamical NDR, which creates the system's volatile memory.
|
From page 9... ...
We tracked the four st strated potentiation. When utilized in a neural network, these devices demonstrated lossless spike propagation at a constant velocity with uniform spike q1 , u2 , q2 )
|
From page 10... ...
Traditional neural networks are unable to find this global minimum because their multidimensional minimization techniques search stepwise from the current location and strongly prefer to go down a steep slope. Rather than climbing to the top of the volcano cone's slopes, such a routine will descend and settle in the basin.
|
From page 11... ...
, thermochemical memory, and electrochemical memory. Each device is a type of nonvolatile memristor that stores information in the positions of atoms that, in turn, control the device resistance.
|
From page 12... ...
Jurczak, and W Vandervorst, Three-dimensional observation of the conduc R tive filament in nanoscaled resistive memory devices, Nano Letters 14:2401, 2014.
|
From page 13... ...
It is quite clear that the operation of these devices must have been based on totally different phenomena and many reported characteristics are owing to excessive dissipated power and temperature. Revising the Model of Filament Formation in Memristor Devices The initial model of resistive switching was proposed by Waser et al.
|
From page 14... ...
14 M e m r i s t i v e M at e r i a l s f o r N e u r o m o r p h i c P r o c e s s i n g A p p l i c at i o n s 1. Oxygen ions cross the interface with anode creating vacancies 2.
|
From page 15... ...
Wo r k s h o p P l e n a r i e s 15 FIGURE 2.4 (a) I-V characteristics of TiN/TaO2.0/TiN device; (b)
|
From page 16... ...
Because these are highly nonlinear dynamical systems, very small changes in initial conditions can lead to very large changes in outcome. This means that the end product depends more on how the process is controlled -- how it is slowed -- than on the nature of the process itself.
|
From page 18... ...
18 M e m r i s t i v e M at e r i a l s f o r N e u r o m o r p h i c P r o c e s s i n g A p p l i c at i o n s pared to an average composition around the filament, which has stayed essentially unaffected. The image shows an accumulation of Ta ions forming a cylinder with a diameter of about 10 nm.
|
From page 19... ...
Wo r k s h o p P l e n a r i e s 19 hotter areas and the other toward the cooler areas of the sample. The theory re mains so nascent that it cannot predict which element will move in which direction and the experimental data are scarce.
|
From page 20... ...
Segregation can be significant in as few as 10 switching cycles. Unlike with metal oxides, this is an unwanted effect in PCMs because infor mation is stored in the material structure, amorphous or crystalline, rather than chemical composition.
|
From page 21... ...
They are highly nonlinear, meaning that very small changes in structure or testing procedures can lead to highly dif ferent outcomes. The driving forces and processes responsible for the segregation, formation, and stability of the filaments remain poorly understood.
|
From page 22... ...
Constructing deep neural net works capable of substantially advancing these kinds of applications will require materials that can meet certain performance specifications. For instance, for such a network to have the level of trainability, accuracy, and energy efficiency required to be useful, the hardware should exhibit a high degree of linearity, a high density of analog states, and high endurance, and operate at low current requirements, ideally in the tens of nanoamps.
|
From page 23... ...
These kinds of applications rely on edge computations, meaning that the processing of incoming data into actionable infor mation is performed at the same location as where the data are being collected, not in the "cloud." Doing so using deep neural networks (DNNs) executed on a digital platform is highly inefficient because DNNs require matrix operations that result in intensive shuttling of data between memory and logic.
|
From page 24... ...
Unlike digital CMOS computing, memristive devices do not depend on mobility of the carriers and should have low current. Keeping current low is essential, since useful device arrays will contain millions of memristive elements.
|
From page 25... ...
These have demon strated close to ideal performance for read, write, and erase cycles, meaning that they have potential applications such as multistate memory. However, the funda mental mechanism and how to apply these effectively in analog neural networks remains unclear.
|
From page 26... ...
. Because there is no need to build a filament, this is a relatively low energy process, requiring about half a millivolt.
|
From page 27... ...
Theoretical results show that the past challenge of implementing memristors to train in an energy-efficient manner can be overcome. Simulations suggest that at 8-bit precision, such a system would show significant benefits over digital devices in terms of energy efficiency, latency, and chip area requirements.
|
From page 28... ...
When spun on, the ionogel appears solid, but at the nanoscale it is still in a liquid state with high conductance. In Situ Computing In addition to the already-commercialized applications mentioned earlier, such as OLEDs, some of these memristor-type systems that could serve for neuro morphic computing are starting to be fabricated as integrated into silicon systems such as CMOS.
|
From page 29... ...
This is because the system is undergoing an electronic transition rather than trying to modulate a thermal distribution of electrons. This kind of transition could be very useful for neurons to compute integration and nonlinear functions as mentioned earlier.
|
From page 30... ...
Prussian blue and its analogs are another examples of coordination polymers that could be attractive for neuro morphic computing. Analogs that replace iron with manganese or copper have shown ultrafast proton transport with useful conductivity that is very promising for neuromorphic applications.
|
From page 31... ...
Strukov stated that the very largest and latest neural networks at the time of this workshop, such as attention networks,7 are used in natural language process ing. These types of networks require an enormous number of weights to operate.
|
From page 32... ...
Training would be typically performed at cloud, so that a primary performance metric for the hardware is throughput per chip area. Inference would be more relevant for the edge devices with a primary metric of energy efficiency.
|
From page 33... ...
For example, for metal-oxide memristors, highly nonlinear switching dynamics allow for tun ableImplementing conductance. Applying Basic Neuromorphic large voltages Operation allows changing the conductance of the devices.
|
From page 34... ...
The output circuitry calculates partial products to the specific kind of bit significance. Those partial products can be accumulated in analog fashion using the successive integration scaling technique or in digital fashion like in Hewlett Packard's ISAAC concept.
|
From page 35... ...
The main reason for this is to utilize weight reuse, a common feature of neural networks where they collect and temporarily store intermediate outputs. Digital domains are better suited for temporary storage, so the system should include circuitry for digital-to-analog and analog-to-digital conversions.
|
From page 36... ...
Trade-Off between Energy Efficiency and Throughput There is also an important trade-off between energy efficiency and through put. Strukov noted that this trade-off exists at different levels of abstractions (device, circuit, and architecture)
|
From page 37... ...
FIGURE 2.13 Trade-off between energy efficiency and throughput. SOURCE: Dmitri Strukov, Univer Mixed-signal for low-to-medium precision & EE optimal, digital for higher precision & throughput-optimal sity of California, Santa Barbara, presentation to the workshop.
|
From page 38... ...
Key Operations on Memristor Crossbar Circuits specific to ex-situ training Forming Write Tuning Read Inference |Vi | ≤ VR VF +VW/2 VR V1 0 0 V2 0 0 V3 float 0 0 V4 0 0 V5 0 0 V6 IR 0 float -VW/2 0 0 0 0 0 virtual 0 0 0 0 0 I1 I2 I3 I4 I5 I6 GND virtual GNDs FIGURE 2.14 Key operations on memristor crossbar circuits. SOURCE: Dmitri Strukov, University of California, Santa Barbara, presentation to the workshop.
|
From page 39... ...
Yang and V Sze, "Design Considerations for Efficient Deep Neural Networks on Processing in-Memory Accelerators," pp.
|
From page 40... ...
Poor device yield; b. Poor device I-V uniformity for 0T1R arrays (need <1% bad devices, sigma Vsw << Vsw / 2)
|
From page 41... ...
That chip was also a unique system featuring a tantalum bilayer with eight chips, each consisting of a 128 × 16 1T1R array of TiN/TaOx/HfOx/TiN devices board-integrated onto CMOS circuitry. The larger complexity allowed this group to demonstrate a convolutional neural network.
|
From page 42... ...
Last, the UCSB group also conducted a purely theoretical simulation of a 3D NAND inference accelerator, which is suitable for large-scale neural network models. This test was pure simulation, unlike the other discussed examples of experimental or hybrid experimental and simulation tests, because at present these systems are available only from companies that want to keep the technology as a
|
From page 43... ...
Of course, the ultimate goal, human-intelligence algorithms, does not yet exist. For these reasons, it seems likely that nonspiking machine learning and bio inspired artificial intelligence efforts will both remain useful.
|
From page 44... ...
Summary In summary, Strukov said, neuromorphic inference with ex situ training is the natural entry-level application of mixed-signal neural networks. It is essential for all other applications, and it is the simplest in terms of device requirements yet is also practical.
|
From page 45... ...
DEVICE ARCHITECTURES TO MEET NEUROMORPHIC COMPUTING CHALLENGES Catherine Schuman, Oak Ridge National Laboratory In this section, Schuman presented three important aspects of device archi tecture for neuromorphic computing. First, decisions made at lower levels of the computing stack, such as materials and devices (see Figure 2.15)
|
From page 46... ...
– Number and type of resistance states – Switching speeds TABLE 2.2 Characteristics of Metal Oxide Memristor Materials – Endurance Material TE/BE VSET/VRESET Switching Retention Time Endura Speed nce – Stability ZnO Ag/Cu 1.2V/-1.25V - - >500 – Reliability cycles – Cost TiO2 TiN/Pt +1V/-1.5V 1uS 104 s – Tunability LaO ITO/SrTiO3 5V/-1.6V - >4x104 s 2000 cycles Mohammad, Baker, et ate of the art of metal TaOx W/Pt - - >10 years 104 memristor cycles s." Nanotechnology ws 5.3 (2016) : 311- NiO Pt/Pt >10V/<-10V - >104 s SOURCE: Catherine Schuman, Oak Ridge National Laboratory, presentation to the workshop; data from B
|
From page 47... ...
Limited weight resolution is another point of concern. The key disadvantage is that the limited resistance levels may limit the programmable weight values visible in the implemented neural networks.
|
From page 48... ...
FIGURE 2.16 An example neural network device constructed using biomimetic synapses. SOURCE: Catherine Schuman, Oak Ridge National Laboratory, presentation to the workshop.
|
From page 49... ...
Deciding whether the structure should be optimized for a particular application or algorithm, or should be more dynamic to make it more usable across the neuro morphic computing field, is an important initial decision. Optimizing allows for a very fast, efficient system.
|
From page 50... ...
. WeNeuromorphic explicitly Computing and Neural Networks in Hardware," arXiv preprint arXiv:1705.06963, 2017, because all of the nature-based https://arxiv.org/abs/1705.06963.
|
From page 51... ...
Reservoir computing, sometimes called liquid state machine approaches, is a quite different algorithm. Reservoir computing is common for training spiking neural networks for neuromorphic systems because it relies largely on an untrained spiking neural network component as the reservoir connected to either a more typical kind of neural network as the training readout layer or some other readout layer component.
|
From page 52... ...
First, consider important hardware characteristics from an applications perspective, such as: • Low power; • Small/embeddable system size; • Processing speed required; • Robustness and resilience required; • Integration with sensors and other compute; and • On-chip training or learning. Bearing this in mind, next consider that the most common application so far for neuromorphic computing and neural networks in hardware is image classifica tion or processing followed by basic benchmark tests, very small-scale tasks like
|
From page 53... ...
, [1278] , [167 Neuromorphic Computing and Neural Networks in Hardware," arXiv preprint arXiv:1705.06963, 2017, and feature extraction [388]
|
From page 54... ...
An important pitfall to avoid is becoming locked into particular use cases that could limit innovation. The temptation with applications like processing, which have driven algorithms like deep learning, and, in turn, have influenced architec tures like crossbars, is to optimize the algorithms and architecture for image pro cessing.
|
From page 55... ...
As illustrated in Figure 2.19, there is a valuable opportunity to utilize co-design between all levels of the compute stack -- for instance, examining materials and tak ing inspiration from the capabilities already there in certain devices and materials and leveraging those at the algorithmic level, but also then providing requirements from the algorithms and driving innovation in devices and materials as well. As Opportunity for Co-Design Devices and Materials Drives new Drives new learning requirements/capabilities mechanisms Co-Design Algorithms Applications Drives new use cases 35 FIGURE 2.19 Cycle of co-design for neuromorphic computing offers a valuable opportunity.
|
From page 56... ...
More generally, the goal is to combine machine learning approaches on high-performance computing systems and neuromorphic systems, simulating neuromorphic systems in different types of devices and materials on high-performance computers, along with real-world scientific data applications from user facilities like the Spallation neutron source at Oak Ridge, in order to discover novel ways of conducting unsupervised learning and new plasticity mechanisms inspired by neuroscience and also by the device materials that can be implemented practically in real-world neuromorphic systems for real-world applications. Summary and Conclusions In summary, Schuman noted that there is an important opportunity for aterials research to inform the development of new algorithms.
|
From page 57... ...
Those mechanisms have historically been inspired by activity seen in biological brains. Consequently, it is a function of the activity in the network, in spiking neural networks where the spikes are occurring, and is not a function of the device itself necessarily.
|
From page 58... ...
FUTURE OF ENERGY-EFFICIENT COMPUTING BASED ON MEMRISTIVE ELEMENTS John Paul Strachan, Hewlett Packard Laboratories Examining the 70-year span from 1935 to 2005 of attempts to engineer intel ligence, Strachan reveals a competition between two approaches: the connectionist and the symbolic. In simplistic terms, the connectionist approach models thinking as consisting of massively parallel simple functions distributed across bio-inspired networks.
|
From page 59... ...
Opportunities for Memristors and Neuromorphic Computing Strachan said that there are also two opportunities for memristors and neuro morphic computing, as highlighted throughout the workshop's presentations. One opportunity is to increase the energy efficiency of current artificial neural networks and deep networks utilizing in-memory computing and analog computing.
|
From page 60... ...
SOURCE: John Paul Strachan, Hewlett Packard Laboratories, presentation to the workshop, from D Amodei, D
|
From page 61... ...
This work is described not as small signal analysis but as a rich, complex system of differential equations describing a nanoscale device that displays analogous types of reactance and behavior, when studied in small-scale signal limits. As mentioned in the first section of this workshop, Williams and his colleagues have built a device and demonstrated that it can reproduce many of the behaviors described in the Hodgkin-Huxley model, while the HRL Laboratories team has developed memristor circuits that can reproduce 23 of the 26 known biological neuron spiking behaviors.
|
From page 62... ...
This can be utilized, for example, in constructing analog tances in an array and leve circuits that solve linear systems of equations in constant rather than polyno- currents. This can perform mial time.62 Finally, the history-dependence intrinsic to memristors is described gle compute cycle, which a by Chua's coupled differential equations and drives highly nonlinear dynam- in modern neural networks ics and the capability for "local activity." This can be used in spiking neural networks, cellular nonlinear networks, random number generators, and chaotic ing linear systems of equatio oscillators.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.