Strait: Perceiving Priority and Interference in ML Inference Serving

Problem
This paper addresses the limitations of existing machine learning (ML) inference serving systems, particularly in their ability to prioritize tasks and accurately estimate latency under concurrent execution conditions. The authors highlight that current systems struggle with deadline satisfaction for dual-priority inference traffic, especially in high GPU utilization scenarios. This work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution of this paper is the design and implementation of the Strait serving system. Strait enhances deadline satisfaction by employing an adaptive prediction model that estimates potential contention during data transfer and accounts for kernel execution interference. The architecture incorporates priority-aware scheduling, which differentiates the handling of high-priority and low-priority inference requests. The authors do not disclose specific details regarding the training compute or the datasets used for evaluation, focusing instead on the system’s operational characteristics and performance metrics.

Results
Strait was evaluated under intense workloads, demonstrating a reduction in deadline violations for high-priority tasks ranging from 1.02 to 11.18 percentage points compared to baseline systems. The performance of Strait was benchmarked against software-defined preemption approaches, where it exhibited more equitable performance across tasks. The results indicate that Strait effectively balances the needs of high-priority tasks while maintaining acceptable performance for low-priority tasks, although specific baseline systems and metrics for comparison are not detailed in the summary.

Limitations
The authors acknowledge that while Strait improves deadline satisfaction, it may incur some costs on low-priority tasks, which could affect overall system throughput. Additionally, the adaptive prediction model’s accuracy and robustness under varying workloads and task distributions are not thoroughly evaluated. The paper does not address potential scalability issues or the impact of different hardware configurations on performance, which could be critical in real-world applications.

Why it matters
The implications of this work are significant for the deployment of ML inference systems in environments where task prioritization is crucial, such as real-time applications and on-premises solutions. By providing a framework for priority-aware scheduling and improved latency estimation, Strait could enhance the reliability and efficiency of inference serving systems, paving the way for more sophisticated applications that require stringent deadline adherence. This research could inform future developments in ML serving architectures, particularly in optimizing resource allocation and managing competing inference requests.

Authors: Haidong Zhao, Nikolaos Georgantas
Source: arXiv:2604.28175
URL: https://arxiv.org/abs/2604.28175v1