Reinforcement Learning with Conformal Action Sets: An Application to Sequential Recommendation

Problem

Sequential recommenders typically operate with a fixed slate size, which does not account for the varying number of useful alternatives available to users. This limitation can hinder the effectiveness of recommendations, as it does not adapt to the dynamic nature of user preferences. The work addresses this gap by proposing a method that allows for adaptive action sets in reinforcement learning, enhancing the recommendation process. Notably, this paper is a preprint and has not undergone peer review.

Method

The authors propose a Reinforcement Learning framework with Calibrated Pruning (RLCP) that adapts action sets based on user feedback. Key components of the method include:

• Action Set Adaptation: The framework utilizes critic scores and an online threshold that is updated based on binary feedback from users, allowing for dynamic adjustment of the action set.
• Theoretical Contribution: The authors provide a deterministic bound on the observed proxy miss rate along adaptive trajectories, which is crucial for understanding the performance of the proposed method.
• Value Loss Decomposition: They derive an exact decomposition of the value loss into filtering and selection losses, contingent on explicit proxy and critic approximation conditions. This decomposition aids in analyzing the performance of the RLCP framework.
• Reward Bound: The framework establishes a finite session reward bound that accounts for the imperfections in selection and the truncation of the action set, ensuring that the recommendations remain effective even with these limitations.

Results

The proposed RLCP framework demonstrates significant improvements in catalog diversity, achieving a performance increase of $1.11 imes$ to $5.21 imes$ compared to the strongest baseline across four reinforcement learning baselines. Additionally, the session depth performance is reported to be competitive without requiring larger retained sets, indicating efficiency in the recommendation process.

Limitations

The authors do not report any limitations in their work, which may suggest a lack of comprehensive evaluation or acknowledgment of potential weaknesses in the proposed method.

Why it matters

This research has important implications for the field of sequential recommendation systems, as it introduces a method that can dynamically adjust to user preferences, potentially leading to more personalized and effective recommendations. The theoretical contributions regarding reward bounds and value loss decomposition also provide a foundation for future work in adaptive reinforcement learning, encouraging further exploration of dynamic action sets in various applications.