Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

Problem

Standard supervised fine-tuning (SFT) in reinforcement learning typically applies loss only to action tokens generated by the agent, neglecting the environment observations. This paper addresses this gap by proposing a method that incorporates observation supervision, which is crucial for improving agent exploration and performance in complex environments. The work is presented as a preprint and has not undergone peer review.

Method

The proposed method, ActObs, supervises observation tokens within agent trajectories. This approach encourages the policy to model the consequences of actions without the need for additional data, parameters, sequence tokens, or forward passes. By integrating observation supervision, ActObs aims to enhance the agent's understanding of the environment, thereby improving its exploration capabilities.

Results

The results demonstrate that ActObs outperforms traditional action-only training methods on several benchmarks:

• On the Terminal-Bench 2.0, the Qwen3-4B model with ActObs achieved a higher pass@k compared to its action-only counterpart.
• The Qwen3-8B model showed a trade-off between reliability and performance, achieving a +3.4 percentage point increase at pass@16 while maintaining a competitive pass@1 score.
• The Aider-polyglot model with ActObs achieved a +4.2 percentage point improvement at 4B on unseen tasks, indicating enhanced generalization capabilities.

The available text does not report quantitative results for other metrics or comparisons.

Limitations

The authors note that after SFT, the gradients for actions and observations become orthogonal, which may hinder the learning process. Additionally, they observe that training solely on actions can lead to a degradation in the model's ability to predict environmental outcomes compared to the base model. These limitations suggest that while ActObs improves exploration, it may also introduce challenges in balancing action and observation learning.

Why it matters

The implications of this work are significant for downstream applications in reinforcement learning, particularly in environments where understanding the consequences of actions is critical. By demonstrating that observation supervision can enhance exploration, this research opens avenues for developing more robust and capable agents that can better navigate complex tasks. Future work may build on these findings to further refine the balance between action and observation learning, potentially leading to more effective reinforcement learning strategies.