Supercharging Thermal Gaussian Splatting with Depth Estimation

Problem
This preprint addresses the gap in efficient 3D scene representation using thermal imagery, particularly in applications like autonomous driving and robotics. Traditional methods often rely on RGB images and multimodal data fusion, which can introduce latency and complexity. The authors propose a novel approach that minimizes reliance on visible light by leveraging thermal images and depth estimation, aiming for faster processing and improved performance in 3D reconstruction tasks.

Method
The core technical contribution is the Thermal-to-Depth Gaussian Splatting (TDg) architecture, which integrates thermal images with depth estimation to construct radiance fields. The method operates primarily on a single modality, thermal infrared, to streamline the data processing pipeline. The authors utilize a Gaussian splatting technique, which allows for efficient representation of 3D scenes. The training process is optimized to reduce computational overhead, achieving a significant reduction in training time by 55%. Specific loss functions and training compute details are not disclosed, but the focus on thermal data suggests a tailored approach to loss calculation that emphasizes the unique characteristics of thermal imagery.

Results
TDg demonstrates superior performance compared to the Multiple Single-Modal Gaussians (MSMG) baseline across multiple metrics on the RGBT-Scenes and ThermalMix datasets. The average improvements in rendering quality metrics are as follows: learned perceptual image patch similarity (LPIPS) improved by 1.12%, structural similarity index measure (SSIM) by 0.034%, and peak signal-to-noise ratio (PSNR) by 0.01%. Additionally, the method significantly reduces training time by 12 minutes and 47 seconds, marking a 55% improvement over the MSMG baseline. These results indicate that TDg not only enhances rendering quality but also optimizes computational efficiency.

Limitations
The authors acknowledge that their method is limited to thermal imagery and depth estimation, which may not capture all relevant scene information compared to multimodal approaches. They do not discuss potential challenges related to the generalizability of the model across diverse environments or the impact of varying thermal conditions on performance. Furthermore, the reliance on depth estimation introduces a dependency on the accuracy of depth sensors, which could affect the overall robustness of the method.

Why it matters
The implications of this work are significant for applications requiring efficient 3D scene reconstruction, particularly in environments where visible light is insufficient or unreliable. By demonstrating that thermal imagery can be effectively utilized for high-quality 3D representations, this research opens avenues for advancements in surveillance, search and rescue operations, and industrial inspections. The reduction in training time also suggests potential for real-time applications, making TDg a promising candidate for deployment in critical scenarios where rapid decision-making is essential.

Authors: Manoj Biswanath, Chenxin Cai, Hannah Schieber, Daniel Roth, Benjamin Busam
Source: arXiv:2605.30328
URL: https://arxiv.org/abs/2605.30328v1