FFR: Forward-Forward Learning for Regression

Problem
The paper addresses the challenge of adapting the Forward-Forward (FF) learning algorithm, which is primarily designed for classification tasks, to regression problems. The authors highlight that FF’s reliance on contrastive positive-negative sample pairs is not directly applicable to continuous target spaces, where such pairs do not exist. Additionally, traditional goodness functions used in regression do not convey information about target magnitude or ordering. This work is significant as it presents the first framework, FFR (Forward-Forward for Regression), to tackle these issues, and it is a preprint, thus unreviewed.

Method
FFR introduces three core innovations to facilitate regression using FF principles:

1. Ordinal Competitive Goodness Function: This replaces the contrastive learning paradigm with a competitive learning approach among partitioned neuron groups, utilizing distance-aware ordinal supervision to guide the learning process.
2. Stratified Ladder Architecture: The architecture is designed with shallow layers focusing on coarse ordinal discrimination, while deeper layers refine predictions into fine-grained outputs. This multi-scale feature aggregation allows for effective inter-layer collaboration, enhancing the model’s ability to learn complex relationships in the data.
3. Hierarchical Prediction with Uncertainty Estimation: FFR employs multiple predictors at different scales that not only provide robust predictions but also estimate prediction confidence, effectively offering uncertainty quantification as an additional benefit. The authors do not disclose specific training compute details but emphasize the efficiency of their approach.

Results
FFR demonstrates impressive performance across five real-world regression benchmarks, recovering an average of 98.6% of the accuracy achieved by backpropagation (BP). In terms of resource efficiency, FFR reduces peak training memory usage to 27% of BP’s at a depth of 8 and to 8% at a depth of 32. Additionally, the per-iteration training time is approximately 72% of that required by BP. FFR significantly outperforms all BP-free competitors, showcasing its effectiveness in practical applications.

Limitations
The authors acknowledge that while FFR shows competitive performance, it may still be limited by the inherent challenges of regression tasks, such as the potential for overfitting in complex datasets. They do not discuss the scalability of the method to extremely large datasets or its performance in highly noisy environments, which could be critical for real-world applications.

Why it matters
The introduction of FFR has significant implications for the field of machine learning, particularly in scenarios where backpropagation is computationally prohibitive or infeasible. By providing a biologically plausible alternative that maintains high accuracy while reducing resource consumption, FFR opens avenues for more efficient neural network training in regression tasks. This work contributes to the ongoing exploration of non-backpropagation methods in deep learning, as discussed in related literature, and is available on arXiv.