SPARK: Share-Per-tick Adaptive Redistribution with Kernel-gating

초록

Modern AI systems increasingly run training, inference, and model serving concurrently on heterogeneous accelerators. Efficiently sharing a single accelerator among such diverse workloads while preserving predictable performance remains challenging; naïve policies often lead to low resource utilization or high latency variability. Consequently, prior work has typically relied on either time slicing or spatial partitioning, but each in isolation is insufficient. Time slicing tends to incur driver-level context-switch overheads and head-of-line blocking, amplifying latency penalties, whereas spatial partitioning—e.g., static schemes like MIG—provides isolation but cannot promptly reclaim and redistribute idle capacity when demand shifts, causing inefficiency. To bridge this gap, we propose SPARK (Share-Per-tick Adaptive Redistribution with Kernel-gating), which tightly integrates temporal and spatial multiplexing. SPARK divides accelerator execution into short ticks and, at each tick, enforces per-job target shares while adaptively redistributing any residual capacity based on observed demand and usage. A kernel-gating mechanism at kernel-dispatch boundaries regulates submission flow, preventing monopolization and starvation; when a latency-sensitive request arrives, SPARK grants priority in the next tick to protect SLOs.

키워드