PIXLRelight: Controllable Relighting via Intrinsic Conditioning

Problem
This paper addresses the limitations of existing single-image relighting methods, which either offer restricted control over lighting (e.g., through textual descriptions or environment maps), suffer from cumulative errors in the rendering pipeline, or necessitate expensive optimization for each image. The authors propose PIXLRelight, a novel approach that integrates physically based rendering (PBR) with learned image synthesis, utilizing a shared intrinsic conditioning mechanism. This work is presented as a preprint and has not yet undergone peer review.

Method
PIXLRelight employs a feed-forward architecture that leverages intrinsic conditioning derived from both real photographs and PBR renders. During training, the model processes paired multi-illumination photographs, decomposing them into albedo, diffuse shading, and non-diffuse residuals, which serve as conditioning inputs. At inference, the model computes the same conditioning from a path-traced render of a coarse 3D reconstruction of the input image under user-defined PBR lighting conditions. The core of the architecture is a transformer-based neural renderer that applies the specified illumination to the source photograph, utilizing per-pixel affine modulation to maintain fine image details. The model is designed for efficiency, achieving inference times of under 0.1 seconds per image.

Results
PIXLRelight demonstrates state-of-the-art performance in relighting quality compared to existing methods. The authors report significant improvements in visual fidelity and control over lighting effects, although specific quantitative metrics and comparisons to named baselines are not detailed in the abstract. The results indicate that the proposed method can effectively handle arbitrary PBR-style lighting scenarios, outperforming traditional techniques that rely on less flexible or more computationally intensive approaches.

Limitations
The authors acknowledge that while PIXLRelight achieves high-quality relighting, it may still be limited by the quality of the initial 3D reconstruction and the intrinsic conditioning derived from the input image. Additionally, the reliance on paired multi-illumination photographs for training could restrict the model’s applicability in scenarios where such data is scarce. The paper does not address potential issues related to generalization across diverse lighting conditions or the model’s performance on images with complex materials or textures.

Why it matters
The implications of PIXLRelight are significant for applications in computer graphics, virtual reality, and augmented reality, where realistic lighting manipulation is crucial. By bridging the gap between PBR and learned synthesis, this work opens avenues for more intuitive and efficient relighting tools that can be integrated into existing workflows. The ability to control lighting in a physically plausible manner with minimal computational overhead could enhance user experiences in various domains, including gaming, film production, and interactive design.

Authors: Miguel Farinha, Ronald Clark
Source: arXiv:2605.18735v1
URL: https://arxiv.org/abs/2605.18735v1