SafeCtrl-RL: Inference-Time Adaptive Behaviour Control for LLM Dialogue via RL-Driven Prompt Optimisation

Problem
This paper addresses the critical challenge of ensuring safe and contextually appropriate behavior in Large Language Models (LLMs) during real-world deployment. The authors identify a gap in existing methodologies that require model retraining or parameter modification to adaptively regulate behavior. This work is presented as a preprint and has not undergone peer review.

Method
The core technical contribution is the SafeCtrl-RL framework, which formulates dialogue generation as a sequential decision-making process. The framework employs a reinforcement learning (RL) agent that dynamically selects prompt adjustment strategies based on contextual feedback from the dialogue. This approach allows for inference-time behavioral unlearning, where the model can suppress unsafe behaviors through iterative refinement of prompts without the need for retraining. The authors do not disclose specific architectural details of the LLMs used, but they evaluate SafeCtrl-RL across multiple models and unsafe dialogue scenarios. The training compute requirements are not explicitly mentioned, but the focus is on inference-time adaptations.

Results
SafeCtrl-RL demonstrates significant improvements in safety and response quality compared to existing prompt-based optimization methods. The authors report that their framework consistently outperforms baseline methods across various benchmarks, although specific numerical results and effect sizes are not detailed in the summary provided. The performance-efficiency trade-offs are described as favorable, indicating that SafeCtrl-RL achieves its objectives without excessive computational overhead.

Limitations
The authors acknowledge that while SafeCtrl-RL improves safety and response quality, it may still be susceptible to certain edge cases where unsafe behaviors could emerge. They do not discuss the potential limitations related to the generalizability of the RL agent across diverse dialogue contexts or the scalability of the approach to larger or more complex LLMs. Additionally, the reliance on contextual feedback may introduce latency in real-time applications, which is not addressed.

Why it matters
The implications of this work are significant for the deployment of LLMs in sensitive applications, such as customer service, mental health support, and educational tools, where safety and appropriateness are paramount. By enabling adaptive safety regulation at inference time, SafeCtrl-RL provides a mechanism for real-time behavioral control, potentially reducing the risks associated with harmful outputs. This framework could pave the way for more robust and reliable LLM applications, fostering trust and safety in AI-driven interactions. Future research may build upon this framework to explore more complex safety mechanisms and broader applications in dialogue systems.

Authors: Michael Orme, Yanchao Yu, Zhiyuan Tan
Source: arXiv:2605.25984
URL: https://arxiv.org/abs/2605.25984v1