unslothai/unsloth · warning · LlamaAdmissionQueueFull

llama-server generation queue is full

Error message

llama-server generation queue is full

What it means

LlamaAdmissionQueueFull from LlamaAdmissionQueue.reserve (llama_admission.py:592): the admission controller hands out a fixed pool of generation slots (capacity, typically llama-server --parallel); when no slot is free AND the number of live waiters already queued has reached config.queue_limit(capacity), new callers are rejected immediately instead of queueing forever. The limit is max_queue if set, else queue_per_slot * capacity (floored by min_queue); None or <= 0 means unbounded. The exception carries a LlamaAdmissionSnapshot (key, capacity, active, queued, free) for reporting.

Source

Thrown at studio/backend/core/inference/llama_admission.py:592

            )

        loop = asyncio.get_running_loop()
        with self._lock:
            self._resize_pool_locked(capacity)
            self._grant_waiters_locked()
            if not self._waiters:
                slot = self._take_slot_locked(len(self._unpark_tickets))
                if slot is not None:
                    # No snapshot here: callers read it through snapshot_now(),
                    # which re-reads the queue, so building one per admitted
                    # request would be pure allocation on the hot path.
                    return LlamaAdmissionReservation(
                        queue = self,
                        lease = LlamaAdmissionLease(self, slot),
                    )
            limit = config.queue_limit(self._capacity)
            if limit is not None and self._live_waiters_locked() >= limit:
                raise LlamaAdmissionQueueFull(
                    "llama-server generation queue is full",
                    snapshot = self._snapshot_locked(),
                )
            waiter = _Waiter(
                loop = loop,
                future = loop.create_future(),
            )
            self._waiters.append(waiter)
            return LlamaAdmissionReservation(
                queue = self,
                waiter = waiter,
            )

    def _release_slot_locked(self, slot: Optional[int]) -> None:
        # A slot id at or past a shrunk capacity retires instead of returning.
        if slot is None or not self._in_use >> slot & 1:
            return
        self._in_use &= ~(1 << slot)

View on GitHub (pinned to 203007d190)

Solutions

1. Catch LlamaAdmissionQueueFull at the API layer and return HTTP 429/503 with the snapshot's active/queued counts so clients back off and retry.
2. Raise capacity: start llama-server with a higher --parallel so more requests get slots and the scaled queue limit grows with it.
3. Tune the queue: set max_queue (or QUEUE_PER_SLOT / MIN_QUEUE env) higher, or to 0/unset for an unbounded line, if you prefer waiting over rejecting.
4. Fix slot leaks: ensure every reservation/lease is released in a finally block — leaked leases shrink the effective pool until everything 409s.
5. Client-side: use exponential backoff with jitter on 429 rather than immediate retries, which only re-fill the queue.

Example fix

# before
reservation = queue.reserve(capacity = n_parallel, config = cfg)  # raises when full

# after
from core.inference.llama_admission import LlamaAdmissionQueueFull
try:
    reservation = queue.reserve(capacity = n_parallel, config = cfg)
except LlamaAdmissionQueueFull as e:
    snap = e.snapshot
    raise HTTPException(429, detail={
        "error": "generation queue full",
        "active": snap.active, "queued": snap.queued,
        "capacity": snap.capacity,
        "retry_after": 5,
    })

snap = queue.snapshot_now()
if snap.queued >= config.queue_limit(snap.capacity or 1):
    return HTTPException(429, "queue full, retry later")

def queue_full(exc: Exception) -> bool:
    return type(exc).__name__ == "LlamaAdmissionQueueFull"

from core.inference.llama_admission import LlamaAdmissionQueueFull
try:
    res = queue.reserve(capacity = n, config = cfg)
except LlamaAdmissionQueueFull as e:
    s = e.snapshot
    raise HTTPException(429, detail={"active": s.active, "queued": s.queued,
        "retry_after": backoff_seconds})

When it happens

Trigger: Issuing more concurrent llama-server generation requests than slots + queue allowance: e.g. --parallel 1 with the default queue limits and several simultaneous chat requests — the first takes the slot, waiters fill the queue, the next reserve() raises LlamaAdmissionQueueFull.

Common situations: Load tests or many simultaneous studio users against a small --parallel; slow/stuck generations holding slots so the queue drains slowly; capacity shrunk while slots are still held (held slots count against the ceiling); queue limits tightened via env (QUEUE_PER_SLOT / MAX_QUEUE) below real traffic.

Related errors

• deadline reached while pacing before {method} {_redact_url(u
• job already running
• A transformers installation is replacing the latest sidecar;
• A diffusion load is already in progress.
• Diffusion generation was cancelled.

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/ef8d07fcad78aedd. Report an issue: GitHub.