Pretraining Recurrent Networks without Recurrence

Problem
The paper addresses the limitations of standard backpropagation through time (BPTT) in training recurrent neural networks (RNNs), particularly its sequential nature, which restricts parallelism, and its susceptibility to vanishing and exploding gradients. These issues hinder the effective learning of long-range dependencies in sequential data. The authors propose a novel approach, Supervised Memory Training (SMT), to mitigate these challenges. This work is a preprint and has not undergone peer review.

Method
SMT transforms the training of RNNs into a supervised learning problem by focusing on one-step memory transition labels, defined as $(m_t, x_{t+1}) \rightarrow m_{t+1}$. The memory labels are generated by training a Transformer-based encoder on a predictive state objective, which retains only the necessary past information to predict future states. This decoupling allows for time-parallel training of RNNs, enabling a stable $O(1)$ gradient path between any two tokens without the need to unroll the RNN. The authors evaluate SMT across various RNN architectures, including nonlinear variants, to assess its effectiveness in capturing long-range dependencies.

Results
The experimental results demonstrate that SMT significantly outperforms BPTT in pretraining RNNs for tasks such as language modeling and pixel sequence modeling. Specific performance metrics include improvements in perplexity and accuracy, although exact numerical results are not disclosed in the abstract. The authors report that SMT enables RNNs to better capture long-range dependencies and achieve faster training times due to its parallelizable nature.

Limitations
The authors acknowledge that while SMT improves training efficiency and long-range dependency learning, it may not fully replace the benefits of recurrent architectures in all scenarios. They do not discuss potential limitations related to the generalization of SMT across different types of sequential tasks or the scalability of the Transformer-based encoder used for generating memory labels. Additionally, the reliance on a predictive state objective may introduce biases depending on the nature of the training data.

Why it matters
The introduction of SMT has significant implications for the future of RNN training methodologies, particularly in applications requiring the modeling of temporal abstractions. By enabling parallel training and improving the learning of long-range dependencies, SMT could facilitate the development of larger and more complex RNN architectures. This work opens avenues for further research into memory-based learning frameworks and their integration with existing neural network paradigms, as published in arXiv cs.LG.