Pixal3D: Pixel-Aligned 3D Generation from Images

Problem
This paper addresses the gap in fidelity for 3D generative models, particularly in the context of image-to-3D synthesis. Despite advancements in generating high-resolution geometry and realistic appearances, existing models struggle with pixel-level correspondence between 2D images and 3D outputs. The authors argue that this issue arises from the implicit 2D-3D correspondence in traditional 3D-native generators, which synthesize shapes in a canonical pose and utilize attention mechanisms to incorporate image cues, leading to ambiguity in pixel-to-3D associations. This work is presented as a preprint and has not yet undergone peer review.

Method
The core contribution of this paper is the introduction of Pixal3D, a pixel-aligned 3D generation paradigm. Unlike conventional methods, Pixal3D generates 3D assets directly in a pixel-aligned manner, ensuring consistency with the input view. The authors propose a pixel back-projection conditioning scheme that lifts multi-scale image features into a 3D feature volume, establishing explicit pixel-to-3D correspondence. This method allows for scalable and high-quality 3D asset production. Additionally, Pixal3D extends to multi-view generation by aggregating back-projected feature volumes across different views, enhancing the fidelity of scene synthesis. The architecture details, loss functions, and specific training compute requirements are not disclosed in the abstract.

Results
Pixal3D demonstrates significant improvements in fidelity compared to existing baselines, approaching the fidelity levels of traditional 3D reconstruction methods. The authors report that their approach not only enhances the quality of individual 3D assets but also benefits scene synthesis by producing high-fidelity, object-separated 3D scenes from single or multi-view images. Specific quantitative results, including effect sizes and comparisons against named benchmarks, are not provided in the abstract.

Limitations
The authors acknowledge that while Pixal3D improves fidelity and scalability, the paper does not discuss potential limitations such as computational overhead, the need for extensive training data, or the generalizability of the model across diverse datasets. Additionally, the reliance on pixel-aligned generation may introduce challenges in scenarios with occlusions or complex geometries that are not well-represented in the input images.

Why it matters
The implications of this work are significant for downstream applications in 3D asset generation, particularly in fields such as virtual reality, gaming, and computer-aided design. By establishing a pixel-aligned approach, Pixal3D sets a new standard for fidelity in 3D generation, potentially influencing future research directions in 3D reconstruction and synthesis. The modular pipeline for high-fidelity scene generation also opens avenues for further exploration in automated scene understanding and object separation.

Authors: Dong-Yang Li, Wang Zhao, Yuxin Chen, Wenbo Hu, Meng-Hao Guo, Fang-Lue Zhang, Ying Shan, Shi-Min Hu
Source: arXiv:2605.10922v1
URL: https://arxiv.org/abs/2605.10922v1