HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps

Problem
This paper addresses the limitations of existing chain-of-thought (CoT) reasoning methods in large language models (LLMs), particularly the high computational and memory costs associated with extended CoT traces. While prior CoT compression techniques utilize memory tokens to condense thought steps, they often sacrifice fine-grained information, leading to increased error rates in subsequent reasoning tasks. The authors propose HybridThinker to retain both compressed representations and temporary access to thought steps, thereby improving reasoning accuracy. This work is a preprint and has not undergone peer review.

Method
HybridThinker employs a dual approach to CoT reasoning. It retains both memory tokens for compressed thought steps and allows temporary access to these steps during inference. The core innovation lies in a hybrid training scheme where only a subset of thought steps is directly accessible to subsequent reasoning steps, while others are masked. This design compels the model to rely on memory tokens for information retrieval, ensuring that the model learns effective compression and retrieval strategies. The architecture specifics, including the model size and training compute, are not disclosed in the paper.

Results
HybridThinker demonstrates significant improvements across four reasoning benchmarks, achieving an average accuracy increase of 5.8 points compared to uncompressed baselines, while maintaining similar inference times. The results indicate that the hybrid training scheme and the retention of temporary thought steps are critical for these gains. The paper includes ablation studies that validate the contributions of both components to the overall performance enhancement.

Limitations
The authors acknowledge that the hybrid training scheme may introduce complexity in model training and could require careful tuning to balance the accessibility of thought steps. Additionally, the reliance on temporary thought-step retention may not generalize well to all reasoning tasks or model architectures. The paper does not address potential scalability issues when applied to larger models or more complex reasoning scenarios.

Why it matters
The advancements presented in HybridThinker have significant implications for the development of more efficient LLMs capable of complex reasoning tasks. By improving the balance between memory efficiency and reasoning accuracy, this work paves the way for future research in CoT compression techniques. The findings contribute to the ongoing discourse on optimizing LLM architectures for practical applications, as discussed in related literature on CoT reasoning and memory efficiency, available on arXiv.