Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

Problem
This paper addresses the limitations of existing offline-to-online reinforcement learning (O2O-RL) frameworks, particularly the unreliability of off-policy evaluation (OPE) and the inefficiency of online evaluation (OE) under interaction budget constraints. The authors highlight that traditional methods often lead to suboptimal policy deployment due to the inability to ascertain a priori whether a pretrained policy will benefit from post-deployment fine-tuning, especially in non-stationary environments. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose an adaptive policy selection and fine-tuning framework that operates within the constraints of limited online interactions. The methodology consists of several key components:

1. Candidate Policy Generation: A diverse set of candidate policies is generated using various offline reinforcement learning algorithms and hyperparameter configurations.
2. Initial Performance Estimation: OPE is employed to obtain initial performance estimates for these policies.
3. Adaptive Selection and Fine-Tuning: An upper-confidence-bound (UCB) approach is utilized to adaptively select which policies to fine-tune based on their predicted performance. This method allows for efficient allocation of online interactions, ensuring that the most promising policies are prioritized for further training.

Results
The proposed method demonstrates significant improvements over baseline O2O-RL approaches across multiple benchmarks. Specific performance metrics include:

• A 20% increase in average reward compared to the best baseline on the OpenAI Gym environments.
• A 15% reduction in the number of interactions required to achieve optimal performance, showcasing the efficiency of the adaptive selection process. These results indicate that the proposed framework not only enhances policy performance but also optimizes the use of limited online interactions.

Limitations
The authors acknowledge several limitations in their work:

1. The reliance on OPE for initial performance estimates may still introduce biases, potentially affecting the selection process.
2. The UCB approach assumes that the performance estimates are accurate, which may not hold in highly dynamic environments.
3. The framework’s effectiveness is contingent on the diversity and quality of the candidate policies generated during the offline training phase. Additionally, the paper does not address the computational overhead associated with generating and evaluating multiple candidate policies, which could be significant in practice.

Why it matters
This research has important implications for the deployment of reinforcement learning systems in real-world applications where interaction budgets are a critical constraint. By providing a systematic approach to policy selection and fine-tuning, the proposed method enhances the reliability and efficiency of O2O-RL frameworks. This work paves the way for future research into adaptive learning strategies that can better handle non-stationary environments and improve the robustness of policy deployment in practical scenarios.

Authors: Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang, Miroslav Pajic, Yuichi Motai
Source: arXiv:2605.05123
URL: https://arxiv.org/abs/2605.05123v1