Constraint-aware GPU scheduler beats FIFO by up to 33 points on utilization, benchmark shows

Dharma-AI has published a benchmark comparing its constraint-aware GPU allocator against a FIFO — first in, first out — scheduler across seven workload scenarios. On identical hardware running the same jobs, the allocator raised GPU utilization by as much as 33 percentage points, and priority-weighted output rose in every scenario, by as much as 105 percent, the company reports.

A GPU cluster running enterprise AI has to serve four kinds of jobs at once — training, batch inference, model quantization, and real-time inference. The first three are batch-like and need a block of GPUs held without interruption until they finish, while real-time inference is elastic and rises and falls with user traffic. Two shapes competing for the same hardware in the same window is the core of the problem.

According to the writeup, FIFO pays for the mismatch in two ways. To guarantee real-time capacity, it reserves each application's peak demand for the entire day, leaving GPUs idle through the off-peak hours. And because it places jobs in arrival order without weighing what a job is worth, high-priority work waits behind whichever request landed first.

The Dharma-AI allocator treats real-time demand as a curve rather than a ceiling, allocating against that demand timestep by timestep and letting batch-like jobs occupy the troughs. Batch jobs are ordered by priority across the full scheduling horizon rather than by arrival. In a training-heavy 8-GPU scenario, utilization rose from 53.6 percent to 87 percent and priority-weighted value more than doubled.

Across five contended scenarios, utilization moved from a 52–88 percent band under FIFO to a 72–88 percent band under the allocator, with priority-weighted gains averaging 52 percent, the company reports.