SymbolicLight V1: Spike-Gated Dual-Path Language Modeling with High Activation Sparsity and Sub-Billion-Scale Pre-Training Evidence

Problem
This paper addresses the limitations of natively trained spiking language models in achieving Transformer-like language quality, stable multi-domain pre-training, and high activation sparsity. The authors present SymbolicLight V1, a novel architecture that aims to bridge these gaps. This work is a preprint and has not yet undergone peer review.

Method
SymbolicLight V1 employs a spike-gated dual-path architecture that integrates binary Leaky Integrate-and-Fire (LIF) dynamics with a continuous residual stream. The model features a Dual-Path SparseTCAM module, which substitutes traditional dense self-attention with two distinct paths: an exponential-decay aggregation path for long-range memory and a spike-gated local attention path for short-range precision. Additionally, it incorporates a dynamic context-conditioned decoding head and a bilingual tokenizer. The model is trained from scratch on a 3 billion token Chinese-English corpus, with a total of 194 million parameters. The training process emphasizes achieving over 89% per-element activation sparsity.

Results
SymbolicLight V1 achieves a held-out validation perplexity (PPL) of 8.88-8.93 across four independent runs, which is a 7.7% degradation compared to GPT-2 (201M) but surpasses GPT-2 (124M). Component ablation studies indicate that the spike-gated local attention path is the most significant contributor to performance, while replacing LIF dynamics with a deterministic top-k mask at matched sparsity levels results in a more substantial performance drop. A scale-up run with 0.8 billion parameters trained on 48.8 billion tokens demonstrates optimization and preservation of sparsity, although it is not primarily intended for quality comparison. The authors note that current dense-hardware inference is slower than GPT-2, suggesting that neuromorphic deployment could be a future avenue for leveraging the model’s sparsity.

Limitations
The authors acknowledge that the current implementation of SymbolicLight V1 does not achieve hardware speedup compared to GPT-2, indicating a gap in practical deployment efficiency. Additionally, the model’s performance is still inferior to the larger GPT-2 variant, which may limit its applicability in scenarios requiring state-of-the-art language modeling. The reliance on specific hardware for neuromorphic deployment is also a potential limitation, as it may not be widely accessible.

Why it matters
The development of SymbolicLight V1 has significant implications for the field of neuromorphic computing and spiking neural networks, particularly in natural language processing. By demonstrating that high activation sparsity can coexist with competitive language modeling performance, this work opens avenues for more efficient models that leverage temporal dynamics. The findings could influence future research on hybrid architectures that combine the strengths of spiking and traditional neural networks, potentially leading to advancements in both model efficiency and deployment in resource-constrained environments.

Authors: Ting Liu
Source: arXiv:2605.21333
URL: https://arxiv.org/abs/2605.21333v1