Researchers at EPFL have developed a groundbreaking algorithm capable of training analog neural networks with accuracy comparable to their digital counterparts. This advancement opens the door to creating more energy-efficient alternatives to power-intensive deep learning hardware, addressing concerns about the environmental impact of large-scale deep neural networks.
As the capabilities of deep neural networks, exemplified by models like Chat-GPT, continue to expand, so do the challenges associated with their size, complexity, and energy consumption. Romain Fleury, from EPFL's Laboratory of Wave Engineering, and his colleagues are exploring physical alternatives to digital deep neural networks to mitigate these challenges.
In a Science paper, the researchers present an algorithm designed for training physical systems, demonstrating improved speed, enhanced robustness, and reduced power consumption compared to traditional methods. The algorithm was successfully tested on wave-based physical systems utilizing sound waves, light waves, and microwaves for information transfer.
The traditional approach to neural network training involves a forward pass and a backward pass (backpropagation). However, backpropagation is energy-intensive and less suitable for physical systems, often requiring a digital twin for simulation, which is inefficient.
The EPFL team proposes replacing backpropagation with a second forward pass through the physical system, allowing each network layer to update locally. This approach reduces power consumption, eliminates the need for a digital twin, and aligns more closely with biological learning processes.
Ali Momeni, the first author and LWE researcher, emphasizes the biologically plausible nature of their approach. By training each physical layer locally, the researchers aim to use the actual physical system without the need for a digital model, making the process more akin to biological learning.
The researchers, in collaboration with Philipp del Hougne of CNRS IETR and Babak Rahmani of Microsoft Research, applied their physical local learning algorithm (PhyLL) to train experimental acoustic and microwave systems and a modeled optical system. The results showcased comparable accuracy to traditional backpropagation-based training, along with increased robustness, adaptability, and resilience to external perturbations.
While the current approach involves some digital parameter updates, the researchers intend to minimize digital computation further. The next steps involve implementing the algorithm on a small-scale optical system with the goal of enhancing network scalability.
This breakthrough marks a significant step towards an analog future for deep neural networks, offering the promise of more energy-efficient and environmentally conscious solutions in the realm of deep learning hardware.
More: https://techxplore.com/news/2023-12-algorithm-barriers-deep-physical-neural.html
