Accelerating Video Diffusion via Training-Free Trajectory Routing

Problem

The paper addresses the high computational cost associated with video diffusion, which arises from the necessity of performing extensive model evaluations across numerous denoising steps. This inefficiency limits the practical application of video diffusion techniques in real-time scenarios. The work is presented as a preprint and has not undergone peer review.

Method

The authors introduce a novel component called TRACK (TRajectory-Aware Capacity routing via top-K selection). This method employs a strategy of heterogeneous denoising through model switching, allowing for dynamic selection of models based on their performance. The calibration process involves rolling out a reference trajectory using a large model, and then comparing the predictions of a smaller model against those of the large model to compute disagreement scores. Both models receive the same latent, timestep, conditioning, and guidance inputs, ensuring consistency in evaluation. The output is a disagreement score map that informs the switching policy during inference, enabling the system to evaluate only the selected model at each step. Notably, this approach does not require retraining or modifications to the model architecture, which simplifies implementation and enhances efficiency.

Results

The proposed method demonstrates substantial speedups across various benchmarks: 1.95× on Wan 2.1, 2.04× to 2.73× on Cosmos 3, 2.69× on TurboDiffusion, and 2.17× on FastVideo. However, the paper does not specify the baselines against which these speedups are measured, limiting the contextual understanding of these improvements.

Limitations

The authors do not report any limitations in their work. However, the lack of specified baselines for the reported speedups may hinder the ability to fully assess the effectiveness of the proposed method in comparison to existing techniques.

Why it matters

This research has significant implications for the field of video processing, particularly in enhancing the efficiency of video diffusion methods. By reducing the computational burden without requiring retraining, the proposed approach could facilitate the deployment of video diffusion in real-time applications, thereby broadening its accessibility and usability in various domains such as video editing, gaming, and virtual reality.