Variance Reduction for Expectations with Diffusion Teachers

Problem
This paper addresses the high computational cost associated with Monte Carlo (MC) estimators used in downstream applications of pretrained diffusion models, such as text-to-3D generation, single-step distillation, and data attribution. The authors highlight that the variance of these MC estimators, which arise from sampling noise levels and Gaussian noise, significantly impacts the efficiency of these pipelines. The work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose CARV, a compute-aware variance-accounting framework designed to optimize the MC estimation process. CARV introduces a hierarchical MC estimator that amortizes the expensive upstream computations (e.g., rendering, simulation, encoding) over cheaper resampling of diffusion noise. Key components of the method include:

• Timestep Importance Sampling: This technique is employed to prioritize certain timesteps in the diffusion process, enhancing the efficiency of the sampling.
• Stratified-Inverse-CDF Construction: This approach is used to improve the quality of the samples drawn from the diffusion model, further reducing variance.

The framework allows for effective reuse of computations, leading to significant reductions in the overall compute cost while maintaining the original objective of the downstream tasks.

Results
In experiments focused on text-to-3D distillation and data attribution, CARV achieved effective compute multipliers of 2-3x. The majority of this improvement stems from the amortized reuse of computations, with an additional ~25% gain attributed to the incorporation of importance sampling and stratification techniques. In the context of single-step distillation, CARV reduced gradient variance by an order of magnitude; however, it did not yield improvements in downstream Fréchet Inception Distance (FID) scores, indicating that MC variance is not the primary bottleneck in this scenario.

Limitations
The authors acknowledge that while CARV significantly reduces variance and computational costs, it does not enhance performance in all contexts, particularly in single-step distillation where FID scores remain unchanged. They do not discuss potential limitations related to the generalizability of the framework across different types of diffusion models or its performance in scenarios with varying noise distributions. Additionally, the reliance on importance sampling and stratification may introduce complexity that could affect implementation in practice.

Why it matters
The implications of this work are substantial for the efficiency of downstream applications utilizing diffusion models. By providing a method to reduce computational overhead without altering the objective, CARV enables more scalable implementations of text-to-3D generation and other related tasks. This advancement could facilitate broader adoption of diffusion models in resource-constrained environments and inspire further research into variance reduction techniques in probabilistic modeling.

Authors: Jesse Bettencourt, Xindi Wu, Matan Atzmon, James Lucas, Jonathan Lorraine
Source: arXiv:2605.21489
URL: https://arxiv.org/abs/2605.21489v1