HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation

Problem
This paper addresses the gap in existing driving world models that primarily focus on future scene generation while neglecting comprehensive 3D scene understanding. Current methodologies often fail to integrate semantic interpretation with physical simulation, particularly in the context of autonomous driving. The authors propose HERMES++, a unified model that aims to bridge this disparity by combining 3D scene understanding with future geometric prediction. This work is presented as a preprint and has not yet undergone peer review.

Method
HERMES++ employs a multi-faceted architecture designed to synergistically address the requirements of 3D scene understanding and future geometry prediction. Key components include:

1. BEV Representation: A Bird’s Eye View (BEV) representation consolidates multi-view spatial information, making it compatible with Large Language Models (LLMs).
2. LLM-Enhanced World Queries: This mechanism facilitates knowledge transfer from the understanding branch to the prediction branch, enhancing the model’s reasoning capabilities.
3. Current-to-Future Link: This design bridges the temporal gap by conditioning future geometric evolution on the semantic context derived from the current scene.
4. Joint Geometric Optimization: This strategy integrates explicit geometric constraints with implicit latent regularization, ensuring that internal representations align with geometry-aware priors.

The authors do not disclose specific training compute or dataset details, but they emphasize the model’s ability to leverage existing knowledge from LLMs to improve performance in both tasks.

Results
HERMES++ demonstrates superior performance across multiple benchmarks, outperforming specialist models in both future point cloud prediction and 3D scene understanding tasks. The paper reports significant effect sizes, although specific numerical results and comparisons to named baselines are not detailed in the abstract. The authors claim that their model achieves state-of-the-art results, indicating a substantial improvement over existing methods.

Limitations
The authors acknowledge that while HERMES++ integrates scene understanding and future prediction, the model’s performance may still be limited by the quality of the input data and the complexity of real-world scenarios. They do not address potential issues related to the scalability of the model or its generalization to diverse driving environments. Additionally, the reliance on LLMs may introduce biases inherent to the training data of those models.

Why it matters
The development of HERMES++ has significant implications for the field of autonomous driving and 3D scene understanding. By unifying scene comprehension and future prediction, this model could enhance the robustness and reliability of autonomous systems in dynamic environments. The integration of LLMs into the driving world model framework may also pave the way for more advanced reasoning capabilities in AI systems, potentially influencing future research in both computer vision and natural language processing.

Authors: Xin Zhou, Dingkang Liang, Xiwu Chen, Feiyang Tan, Dingyuan Zhang, Hengshuang Zhao, Xiang Bai
Source: arXiv:2604.28196
URL: https://arxiv.org/abs/2604.28196v1