TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

Problem

The paper addresses a significant tradeoff in point tracking models, where existing approaches either focus on sparse long horizon tracking or dense short clips. This gap limits the ability to effectively track all visible points over extended periods, particularly in complex 3D environments. The work is presented as a preprint and has not undergone peer review.

Method

The authors introduce TrackEverything, a 3D point tracker designed to optimize tracking performance across long sequences. Key components of the method include:

• De-duplication Mechanism: Utilizes voxelization at sliding-window boundaries to merge co-located tracks, effectively reducing redundancy in tracking data.
• Tracking Decomposition: Incorporates an endpoint refiner for destination prediction and a classification system to distinguish between static and dynamic points. Additionally, a lightweight trajectory refiner is employed specifically for dynamic points.
• 3D WAFT (Weighted Adaptive Feature Tracking): This innovative approach replaces traditional 4D correlation volumes with efficient feature sampling directly from the scene cloud, enhancing computational efficiency.
• Memory Usage: The model is designed to track all visible points while operating within a memory constraint of 40 GB of GPU memory, making it feasible for real-time applications.

Results

The performance of TrackEverything is evaluated on the TAPVid-3D benchmark, where it achieves an Average Precision Drop (APD) improvement of over 20% compared to open-source all-frame dense 3D trackers. Furthermore, its performance on long sequences is reported to be competitive with state-of-the-art sparse trackers, indicating its effectiveness in maintaining tracking fidelity over extended durations.

Limitations

The authors do not report any limitations in their work, and no obvious shortcomings are identified in the available text.

Why it matters

The development of TrackEverything has significant implications for downstream applications in computer vision and robotics, particularly in scenarios requiring robust long-term tracking in dynamic environments. By bridging the gap between sparse and dense tracking, this work paves the way for advancements in real-time 3D scene understanding and interaction, potentially enhancing applications in autonomous navigation, augmented reality, and surveillance.