The Stable Recovery Manifold: Geometric Principles Governing Recoverability in Continual Learning

Problem — This work addresses the gap in understanding the geometric underpinnings of recoverability in continual learning, particularly in the context of catastrophic forgetting. The authors build on the Accessibility Collapse framework to explore how knowledge can be preserved during sequential learning tasks. This is a preprint and has not undergone peer review.

Method — The authors utilize a sequentially trained ResNet-18 architecture on the Split CIFAR-100 dataset to analyze recoverability across ten tasks. They introduce a novel metric called Recovery Subspace Dimensionality (k_t), which quantifies the minimum number of singular directions necessary to maintain 90% of full probe performance. The study employs principal-angle drift as a predictor of recoverability, establishing a correlation coefficient of r = -0.862. A geometric model is proposed to explain the variance in recoverability, achieving an R² of 0.822.

Results — The findings reveal that the mean Recovery Subspace Dimensionality (k_t) remains stable at 8.0 throughout the training process, despite significant representational drift. This stability contradicts the authors’ initial Recoverability Diffusion hypothesis, suggesting that the dimensionality of recovery does not fluctuate as expected. The strong predictive relationship between principal-angle drift and recoverability indicates that the geometric alignment of knowledge plays a crucial role in mitigating catastrophic forgetting.

Limitations — The authors acknowledge that their findings are based on a specific architecture (ResNet-18) and dataset (Split CIFAR-100), which may limit the generalizability of the results. They do not explore the implications of different architectures or datasets on the Recovery Subspace Dimensionality. Additionally, the study does not address the potential impact of varying task complexities on recoverability metrics.

Why it matters — The implications of this research are significant for the field of continual learning, as it reframes catastrophic forgetting as an issue of accessibility and manifold alignment rather than mere information loss. This perspective could inform the development of new algorithms and strategies aimed at enhancing recoverability in neural networks. The findings support the notion that forgotten knowledge can be effectively retrieved, which may lead to more robust continual learning systems. This work contributes to the ongoing discourse in the field, as published in arXiv cs.LG.