MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference

Problem

The paper addresses a gap in the efficient application of microscaling quantization specifically in convolutional layers. The authors highlight the need for improved methods that can effectively reduce computational costs while maintaining performance, particularly in the context of continual learning. This work is presented as a preprint and has not undergone peer review.

Method

The core technical contribution is the MicroQonv architecture, which implements a unique quantization method that quantizes each tensor only once. This approach utilizes a channel-batch-first im2col transformation, which modifies the activation tensor processing by applying im2col before quantization. The method achieves significant reductions in quantization costs: weights and gradients are reduced by a factor of 2, while activations can be reduced by up to a factor of 9. Additionally, the architecture facilitates a memory movement and storage reduction of up to 7.53 times compared to full-precision implementations. MicroQonv is particularly designed to enable 4-bit microscaling in a quantized latent replay strategy, which is beneficial for continual learning scenarios.

Results

The results demonstrate substantial improvements in memory movement reduction for specific models. For the YOLOV8nano architecture, memory movement is reduced by a factor of 3.5 compared to full-precision, while for YOLOV26nano, the reduction is 2.2 times. Furthermore, the accuracy in continual learning tasks shows an improvement ranging from 5.7% to 11% over baseline methods, indicating that the proposed architecture not only enhances efficiency but also boosts performance in learning scenarios.

Limitations

The authors do not report any limitations in their work, and no obvious limitations are noted in the available text.

Why it matters

The implications of this work are significant for downstream applications in machine learning, particularly in scenarios requiring efficient resource utilization, such as edge computing and mobile applications. The ability to perform effective microscaling quantization while maintaining or improving accuracy opens new avenues for deploying deep learning models in resource-constrained environments, thereby enhancing the feasibility of continual learning in practical applications.