Swift Sampling: Selecting Temporal Surprises via Taylor Series

Problem
This paper addresses the challenge of efficiently selecting frames from long-form videos for downstream tasks, particularly in scenarios where most frames are redundant. The authors identify a gap in existing methods that either rely on auxiliary networks or require extensive hyperparameter tuning specific to each video. The proposed method, Swift Sampling, is a training-free approach that aims to identify temporally surprising frames—moments where visual features deviate significantly from their predicted trajectory. This work is presented as a preprint and has not yet undergone peer review.

Method
Swift Sampling models a video as a differentiable trajectory in the visual latent space, leveraging the principles of predictive coding. The core technical contribution involves computing the velocity and acceleration of visual features to project the expected path of subsequent frames using Taylor series expansion. Frames that exhibit significant divergence from this predicted trajectory are classified as temporally surprising and selected for sampling. The algorithm is designed to be lightweight, incurring only a 0.02x increase in computational cost over baseline methods, which is reported to be 30x cheaper than leading alternatives. The authors do not disclose specific details regarding the training compute or datasets used for evaluation.

Results
The performance of Swift Sampling is evaluated across three long-video question answering benchmarks and ten downstream tasks. The method demonstrates a substantial improvement over uniform sampling and prior query-agnostic baselines, achieving accuracy gains of up to +12.5 points. The results indicate that Swift Sampling is particularly effective for long videos with limited frame budgets, showcasing its ability to enhance task performance by focusing on high-information frames.

Limitations
The authors acknowledge that while Swift Sampling is lightweight and effective, it may not generalize well to all types of video content, particularly those with less predictable visual dynamics. Additionally, the reliance on Taylor series expansion may introduce limitations in scenarios where the visual features do not conform to the assumptions of smoothness or continuity. The paper does not address potential computational overhead in real-time applications or the impact of varying frame rates on performance.

Why it matters
The implications of this work are significant for the fields of video analysis and machine learning, particularly in applications requiring efficient processing of long-form video content. By providing a method that identifies and prioritizes temporally surprising frames, Swift Sampling can enhance the performance of various downstream tasks, such as video classification, summarization, and question answering. This approach could lead to more efficient video processing pipelines, reducing computational costs while maintaining or improving accuracy.

Authors: Dahye Kim, Bhuvan Sachdeva, Karan Uppal, Naman Gupta, Vineeth N. Balasubramanian, Deepti Ghadiyaram
Source: arXiv:2605.22678
URL: https://arxiv.org/abs/2605.22678v1