Sandia explores neural computing to extend Moore’s Law
Computation is stuck in a rut. The integrated circuits that powered the past 50 years of technological revolution are reaching their physical limits.
This predicament has computer scientists scrambling for new ideas: new devices built using novel physics, new ways of organizing units within computers and even algorithms that use new or existing systems more efficiently. To help coordinate new ideas, Sandia National Laboratories has assisted organizing the Institute of Electrical and Electronics Engineers (IEEE) International Conference on Rebooting Computing held Oct. 17-19.
Researchers from Sandia’s Data-driven and Neural Computing Dept. will present three papers at the conference, highlighting the breadth of potential non-traditional neural computing applications.
“We’re taking a stab at the scope of what neural algorithms can do. We’re not trying to be exhaustive, but rather we’re trying to highlight the kind of application over which algorithms may be impactful,” said Brad Aimone, a computational neuroscientist and co-author of one paper. Historically, neural computing has been seen as approximate and fuzzy, he added; however, Sandia researchers in their papers aim to extend neural algorithms so they incorporate rigor and predictability, which shows they may have a role in high performance scientific computing.
The three papers are entitled “Overcoming the Static Learning Bottleneck — the Need for Adaptive Neural Learning” by Craig Vineyard and Steve Verzi; “Computing with Dynamical Systems” by Fred Rothganger; and “Spiking Network Algorithms for Scientific Computing” by William Severa, Ojas Parekh, Kris Carlson, Conrad James and Aimone.
Troubles and benefits of continuously learning
The brain is continually learning. “While we do learn in school, our learning doesn’t stop when school ends. Instead, our brains are continually adapting through processes, such as synaptic modifications. However, most machine-learning algorithms learn once and are done,” said Vineyard, a computer scientist.
Most so-called machine-learning algorithms have a learning phase and a separate testing and operation phase. This is really time consuming. Ambitious — and challenging — attempts to develop algorithms that learn continuously also run the risk of the algorithm “learning” something that’s wrong, Vineyard said.
His paper argues for continual learning and suggests the use of game theory — the mathematics of logical decisions, such as when to take out the trash and when to hope your roommate will do it for you — to bring precision to the decision of when an algorithm should learn.
What are dynamical systems anyway?
A dynamical system is an equation that describes how things change with time. A simple dynamical system is a function that describes the movement of a grandfather clock’s pendulum. “The idea behind using dynamical systems for computation is to build a machine such that its dynamics — which has to do with the structure of the machine or the structure of the math — will lead it to the answer based on feeding it the question,” said Rothganger, a computer scientist.
Both our brains and, in a way, conventional computers are dynamical systems: They find answers just based on the question and how the computers are constructed, said Rothganger. His paper proposes that if researchers think of a traditional scientific computing problem, matrix decomposition, as a dynamical system, they could solve them rigorously on neuro-inspired systems.
“There’s a lot of potential and also a lot of risk in the idea I’m working on,” said Rothganger. If his idea works, “it would provide a point of unification between neural algorithms and traditional numerical algorithms.”
Artisan mathematicians craft spiking network algorithms
The third paper identifies three hand-crafted algorithms that use the careful arrangement of spiking neuron-like nodes to perform precise computations. In the brain, each neuron is connected to many other neurons and uses spikes of electricity to communicate. Severa, a mathematician, and his co-authors took inspiration from these aspects of the brain.
An example of these innovative algorithms is a kind of flow estimation called particle image velocimetry. By taking two pictures of dust motes moving through the air and figuring out how far they moved in the time between photos, researchers can determine the speed of the air and any local eddies. This can be done on a conventional computer using fancy math, but Severa’s method uses the massively parallel nature of neurons to calculate all the possible shifts efficiently, he said.
“By carefully designing your networks and the properties of your neurons, you can do exact things,” said Severa. “You can push the envelope of what you can expect a neural network to do.”
Whether the future holds neuro-inspired computers in your cellphone that understand phrases like “Show me a cute picture of Fluffy” and “Order my favorite Chinese food,” or if neural computers can also work alongside future quantum computers in solving tough math problems quickly, computing needs to be reinvented, and soon, said Aimone. By bringing together experts in many different disciplines, he said the International Conference on Rebooting Computing aims to nurture new ideas and spur this revolution.
Funding for all the projects was provided by Sandia’s Laboratory Directed Research and Developmentoffice. Two projects also were part of the Hardware Acceleration of Adaptive Neural Algorithms (HAANA) Grand Challenge.
The broader rebooting computing effort
Sandia employees are among the organizers of the IEEE’s Rebooting Computing initiative and the conference. Sandia’s Chief Technology Officer Rob Leland will give the conference kickoff talk on the history of innovation in computing. Sandia researchers Erik DeBenedictis and Matt Marinella are members of the conference program committee.
DeBenedictis, Sapan Agarwal, Jeanine Cook and Michael Frank also are presenting four papers on low-energy logic and memory. Christopher DeRose and Tony Lentine are presenting a paper on optical communications.
Learn more: Turning to the brain to reboot computing
Researchers at North Carolina State University have developed new, nonlinear, chaos-based integrated circuits that enable computer chips to perform multiple functions with fewer transistors. These integrated circuits can be manufactured with “off the shelf” fabrication processes and could lead to novel computer architectures that do more with less circuitry and fewer transistors.
Moore’s law states that the number of transistors on an integrated circuit will double every two years in order to keep up with processing demands. Previously this goal has been addressed by shrinking the size of individual transistors so that more could be added to the chip. However, that solution is quickly becoming untenable, and the semiconductor industry is looking for new ways to create better computer chips.
“We’re reaching the limits of physics in terms of transistor size, so we need a new way to enhance the performance of microprocessors,” says Behnam Kia, senior research scholar in physics at NC State and lead author of a paper describing the work. “We propose utilizing chaos theory – the system’s own nonlinearity – to enable transistor circuits to be programmed to perform different tasks. A very simple nonlinear transistor circuit contains very rich patterns. Different patterns that represent different functions coexist within the nonlinear dynamics of the system, and they are selectable. We utilize these dynamics-level behaviors to perform different processing tasks using the same circuit. As a result we can get more out of less.”
Kia and NC State colleague William Ditto, professor of physics and dean of the College of Sciences, worked on the conception, design, development and fabrication of an integrated circuit chip that contains working nonlinear circuits to perform multiple different digital computations.
Traditionally, transistor-based circuits perform one task each. Computer processors operate by routing each instruction and its operands to the appropriate transistor circuit on the integrated circuit that implements that specific instruction. In Kia’s design, the transistor circuit can be programmed to implement different instructions by morphing between different operations and functions.
“In current processors you don’t utilize all the circuitry on the processor all the time, which is wasteful,” Kia says. “Our design allows the circuit to be rapidly morphed and reconfigured to perform a desired digital function in each clock cycle. The heart of the design is an analog nonlinear circuit, but the interface is fully digital, enabling the circuit to operate as a fully morphable digital circuit that can be easily connected to the other digital systems.”
The researchers have produced an alternative approach for computing that is compatible with existing technology and utilizes the same fabrication process and CAD tools as existing computer chips, which could aid commercial adoption.
“We believe that this chip will help solve the challenges of demands for more processing power from fewer transistors,” Kia says. “The potential of 100 morphable nonlinear chaos-based circuits doing work equivalent to 100 thousand circuits, or of 100 million transistors doing work equivalent to three billion transistors holds promise for extending Moore’s law – not through doubling the number of transistors every two years but through increasing what transistors are capable of when combined in nonlinear and chaotic circuits.”
“We are nearing commercial size and power and ease of programming in our evolving designs that could well be of significant commercial relevance within a few months with our three month design/fabrication cycle of improvements and implementations,” Ditto says.