Quadratic integrate-and-fire neurons exhibit less fragmented loss landscapes and outperform leaky integrate-and-fire neurons in spike-based gradient descent

Problem
The paper addresses the challenge of training spiking neural networks (SNNs), particularly the instability associated with leaky integrate-and-fire (LIF) neurons, which can lead to erratic spike behavior and ineffective learning during spike-based gradient descent. This issue is critical for both biological modeling and neuromorphic computing. The authors highlight that while quadratic integrate-and-fire (QIF) neurons have been proposed as a solution, empirical validation of their advantages over LIF neurons remains underexplored. This work is a preprint and has not undergone peer review.

Method
The authors conducted a controlled comparison between LIF and QIF neurons using the Spiking Heidelberg Digits dataset. They performed a comprehensive hyperparameter optimization for both neuron types to ensure fair evaluation. The QIF model is characterized by its continuous spiking dynamics, which mitigates the discontinuities seen in LIF neurons. The authors visualized the loss and gradient landscapes for both models, revealing the fragmented nature of the LIF landscapes compared to the smoother QIF landscapes. The training compute details were not explicitly disclosed, but the focus was on the comparative performance and landscape analysis.

Results
The results indicate a significant performance advantage for QIF neurons over LIF neurons. Specifically, QIF neurons achieved a lower loss and more stable gradients during training, as evidenced by the smoother loss landscapes. The authors did not provide specific numerical results or effect sizes in the abstract, but they emphasize the qualitative differences in the loss and gradient landscapes, which are critical for understanding the training dynamics. The findings suggest that QIF neurons can maintain more stable representations and avoid the pitfalls of spike disappearance that plague LIF neurons.

Limitations
The authors acknowledge that their study is limited to a specific dataset (Spiking Heidelberg Digits) and may not generalize across all types of spiking neural network applications. They also note that while QIF neurons show promise, further exploration into their performance across diverse tasks and datasets is necessary. Additionally, the paper does not address the computational efficiency of QIF neurons compared to LIF neurons, which could be a critical factor in practical applications.

Why it matters
This work has significant implications for the design and training of spiking neural networks, particularly in neuromorphic computing applications where stability and efficiency are paramount. The findings advocate for the adoption of neuron models with continuous dynamics, such as QIF, to enhance the training process and improve model performance. This research contributes to the ongoing discourse on optimizing SNN architectures and training methodologies, as highlighted in related literature on spiking neural networks and their applications in arXiv.