Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 May 25;15(1):4464.
doi: 10.1038/s41467-024-47811-6.

Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip

Man Yao #  1 , Ole Richter #  2 , Guangshe Zhao #  3 , Ning Qiao #  2   4 , Yannan Xing  4 , Dingheng Wang  5 , Tianxiang Hu  1 , Wei Fang  6   7 , Tugba Demirci  2 , Michele De Marchi  2 , Lei Deng  8 , Tianyi Yan  9 , Carsten Nielsen  2   10 , Sadique Sheik  2 , Chenxi Wu  2   10 , Yonghong Tian  6   7 , Bo Xu  1 , Guoqi Li  11   12
Affiliations

Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip

Man Yao et al. Nat Commun. .

Abstract

By mimicking the neurons and synapses of the human brain and employing spiking neural networks on neuromorphic chips, neuromorphic computing offers a promising energy-efficient machine intelligence. How to borrow high-level brain dynamic mechanisms to help neuromorphic computing achieve energy advantages is a fundamental issue. This work presents an application-oriented algorithm-software-hardware co-designed neuromorphic system for this issue. First, we design and fabricate an asynchronous chip called "Speck", a sensing-computing neuromorphic system on chip. With the low processor resting power of 0.42mW, Speck can satisfy the hardware requirements of dynamic computing: no-input consumes no energy. Second, we uncover the "dynamic imbalance" in spiking neural networks and develop an attention-based framework for achieving the algorithmic requirements of dynamic computing: varied inputs consume energy with large variance. Together, we demonstrate a neuromorphic system with real-time power as low as 0.70mW. This work exhibits the promising potentials of neuromorphic computing with its asynchronous event-driven, sparse, and dynamic nature.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Fig. 1
Fig. 1. Neuromorphic computing vs. traditional computing from the view of dynamic computing.
a Spiking neuron vs. Artificial neuron. Left: spiking neurons communicate through spike trains coded in binary spikes, and the major operation is synaptic Accumulation (AC) between weights. Right: neurons in ANNs communicate using activations coded in analog values, and Multiply-and-Accumulate (MAC) of inputs and weights is the major operation. b From a dynamic computing perspective, we compare neuromorphic and traditional computing in three aspects: vanilla algorithm (top), dynamic algorithm (middle), and hardware (bottom). In traditional computing (right part), vanilla algorithms generally hold a static, fixed computational graph manner. Although some neurons have activation values of zero, all zero-based MAC operations must be performed. By adapting the structures of static models to different inputs, dynamic ANNs can lead to notable advantages in accuracy and computational efficiency. However, traditional computing hardware is mostly optimized for static models and not friendly to dynamic networks, and there is a gap between the theoretical and practical efficiency of dynamic ANNs. In neuromorphic computing (left part), SNNs are born with dynamic computational graphs, and neuromorphic hardware is naturally suitable for SNNs. However, we observed the dynamic imbalance in SNNs, which respond similarly to diverse inputs. c Dynamic imbalance. SNNs satisfy dynamic activation forms but are not good at dynamic functions, i.e. responding discriminatively. Spatio-temporal invariance (Figs. S2, S3) is the fundamental assumption of SNNs because they share parameters at different timesteps. Consequently, LSFRs (definition is given in Part “Details of algorithm evaluation'') at each timestep is similar, which indicates that the scales of the activated sub-networks of SNNs are similar for diverse input.
Fig. 2
Fig. 2. The design details of Speck.
a The power composition of AI systems. b The case of high resting power. When the resting power is too high, the gain brought by the advanced algorithm design is hard to lower the total power effectively. c The case of low resting power. Low resting power helps unleash the power of advanced algorithm design. d Speck physical display. e Speck is a sensing-computing end-to-end SoC that integrates DVS and asynchronous neuromorphic chip. f Typical application scenarios of always-on Speck. g The fully asynchronous architecture of Speck. The DVS events come from the on-chip sensor. After an asynchronous event pre-processing core, events can be routed to SNN cores for processing. In Fig. S1, we give the layout of Speck. h The SNN core microarchitecture (more details in Fig. S5). Each SNN core can be simply considered as a spiking convolution layer with an integrated pooling layer. When a spike (event) is received at the input of the core, a fully asynchronous convolution operation is performed to calculate all required neuron updates caused by the received input spike. i Asynchronous event-driven convolution. Based on the address of the input event or spike, the address mapping function outputs the address of the neuron and synapse that need to perform synaptic operations (more details in Fig. S4).
Fig. 3
Fig. 3. Brain-inspired dynamic framework for neuromorphic computing.
a Attention-based dynamic response in neuroscience. The brain’s dynamic responses are associated with visual attention. Since attention is a limited resource, the brain only selectively processes a part of sensory input. The neural correlates of attention can be roughly divided into four structural levels,. Attention neural circuit. The top-down versus bottom-up dichotomy is one of the classic classifications of attention neural circuits, which encompass multiple visual areas. Top-down deploys the attention to internal, behavioral goals of the brain, which can be present through the priority map. Bottom-up allocates attention according to the physical salience of a stimulus, which the salience map can illustrate. Visual area. The regulation of attention involves multiple brain areas, which generally results in changes in neuronal firing rate within the areas. Neuron. Attention-related neuronal modulations. Recordings from individual cells have shown that attention is associated with the change in neuron firing, which can enhance the quality of sensory representations. Synaptic. Attention fine-tunes neuronal communication by selectively modifying synaptic weights, enabling enhanced detection of important information in a noisy environment. b A typical spiking neuron model: Leaky Integrate and Fire (LIF). c Attention-based dynamic SNNs. The proposed dynamic framework exists as plug-and-play attention modules that optimize the membrane potential in a data-dependent manner in both temporal and channel dimensions. The dynamic framework provides two types of combinable strategies, refinement, and masking, to expand the strategy space and establish a better trade-off between accuracy and energy consumption.
Fig. 4
Fig. 4. Analysis of dynamic SNNs regarding performance and spiking activity.
a Examples of event-based sample. b Effects of Attention-based Refine (AR) policies on accuracy. Optimizing the membrane potential in both temporal and channel dimensions yields the best accuracy gain. c Effects of AR policies on spiking firing. Exploiting attention to optimize the membrane potential can drop spikes. d Effects of Attention-based Mask (AM) policies on accuracy. Increasing the masking ratios will generally result in a loss of performance. e Effects of AM policies on spiking firing. Adding the mask ratios does not always reduce spiking firing. f Spiking responses of vanilla and dynamic SNN. The proposed dynamic framework alleviates dynamic imbalance. g Visualization of overall spiking response on Gait. From top to bottom: spiking features in the first layer (64 channels) of vanilla, AR, and AM-SNN. The redder the pixel, the higher NSFR (i.e., neuron spike firing rate, specific definition is given in Part “Details of algorithm evaluation''); the bluer the pixel, the closer the NSFR is to 0. Attention drives the network to focus on the target and suppress the redundant background channels, where the latter leads to a significant reduction in spikes. h Structural and functional correspondence between dynamic processing in neuroscience and dynamic SNNs on the network (circuit)-level and layer (area)-level. When generating saliency maps, the proposed dynamic framework recombines intermediate features according to their importance. i Neuronal dynamics in vanilla and dynamic SNNs. j Optimizing the membrane potential is mathematically equivalent to moderating the weights.
Fig. 5
Fig. 5. Placement of the dynamic SNNs on Speck.
a Speck-based neuromorphic system. b Dynamic SNN architecture deployed on Speck. We made some algorithmic adjustments based on the proposed dynamic framework, to adapt to the hardware. We only employ temporal-wise attention on the event streams to wean out which inputs can be masked. On the other hand, the spiking neuron model on Speck is Integrate and Fire (IF), i.e., LIF neuron without leaky operation. Xt,n, Ht,n, and St,n (specific definitions are given in Part “Spiking neuron models'' section) represent the spatial input, temporal input, and spike output of the spiking neuron, respectively. c Overall of Speck-based neuromorphic system with dynamic SNNs. The DVS camera only perceives and encodes the brightness change information in the visual scene (the red/green dots in the graph represent brightness increase/decrease respectively.), significantly reducing spatial redundancy compared with the traditional camera. However, the high temporal resolution of the DVS causes information redundancy in the temporal dimension. We adaptively mask some inputs using temporal attention. Since Speck is event-driven, less input means lower energy consumption. Moreover, the width of Speck kit shows is equivalent to the diameter of a coin, about 25 mm, which is convenient for edge computing scenarios. d, e On-chip real-time power based on vanilla and dynamic SNNs, respectively.
See this image and copyright information in PMC

Similar articles

See all similar articles

References

    1. Roy K, Jaiswal A, Panda P. Towards spike-based machine intelligence with neuromorphic computing. Nature. 2019;575:607–617. doi: 10.1038/s41586-019-1677-2. - DOI - PubMed
    1. Schuman CD, et al. Opportunities for neuromorphic computing algorithms and applications. Nat. Comput. Sci. 2022;2:10–19. doi: 10.1038/s43588-021-00184-y. - DOI - PubMed
    1. Bartolozzi C, Indiveri G, Donati E. Embodied neuromorphic intelligence. Nat. Commun. 2022;13:1024. doi: 10.1038/s41467-022-28487-2. - DOI - PMC - PubMed
    1. Mehonic A, Kenyon AJ. Brain-inspired computing needs a master plan. Nature. 2022;604:255–260. doi: 10.1038/s41586-021-04362-w. - DOI - PubMed
    1. Schemmel, J. et al. A wafer-scale neuromorphic hardware system for large-scale neural modeling. In 2010 IEEE International Symposium on Circuits and Systems (ISCAS) 1947–1950 (IEEE, 2010).

LinkOut - more resources

-