Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning

Problem
The paper addresses the gap in multi-objective reinforcement learning (MORL) concerning fairness in policy selection, particularly in scenarios with dynamic or unknown user preferences. Existing single-policy MORL methods, which utilize welfare functions like the generalized Gini welfare function (GGF), are limited in their ability to provide a diverse set of policies that adapt to varying user needs. This work formalizes the fair optimization problem in a multi-policy context, aiming to learn a set of Pareto-optimal policies that ensure fairness across all potential user preferences. The research is presented as a preprint and has not undergone peer review.

Method
The authors propose a three-pronged approach to tackle the problem. First, they establish that for concave, piecewise-linear welfare functions (e.g., GGF), fair policies reside within the convex coverage set (CCS), which serves as an approximated Pareto front for linear scalarization. Second, they introduce non-stationary policies that leverage accrued reward histories to enhance fairness by adapting to historical inequities. Third, they develop three novel algorithms: (1) a multi-policy multi-objective Q-Learning (MOQL) algorithm that integrates GGF, (2) a state-augmented MOQL for learning non-stationary policies, and (3) an extension for learning stochastic policies. The training compute and specific datasets used for evaluation are not disclosed.

Results
The proposed algorithms were evaluated across various domains, demonstrating significant improvements in fairness compared to state-of-the-art MORL baselines. The results indicate that the new methods successfully learn a diverse set of fair policies that can accommodate different user preferences. While specific numerical results are not detailed in the abstract, the authors claim that their methods outperform existing approaches in terms of both fairness and policy diversity.

Limitations
The authors acknowledge that their approach may be limited by the assumptions regarding the concavity and piecewise-linear nature of the welfare functions. Additionally, the performance of the algorithms may vary across different environments, and the scalability of the proposed methods in high-dimensional action spaces is not thoroughly examined. The lack of peer review also raises questions about the robustness of the findings.

Why it matters
This work has significant implications for the development of fair decision-making systems in reinforcement learning, particularly in applications where user preferences are not static. By providing a framework for learning fair Pareto-optimal policies, the research paves the way for more equitable AI systems that can adapt to diverse user needs. This is particularly relevant in fields such as healthcare, finance, and social policy, where fairness is critical. The findings contribute to the ongoing discourse on fairness in AI, as published in arXiv cs.AI.