{"record":{"id":"ef8d07fcad78aedd","repo":"unslothai/unsloth","slug":"llama-server-generation-queue-is-full","errorCode":null,"errorMessage":"llama-server generation queue is full","messagePattern":"llama-server generation queue is full","errorType":"exception","errorClass":"LlamaAdmissionQueueFull","httpStatus":null,"severity":"warning","filePath":"studio/backend/core/inference/llama_admission.py","lineNumber":592,"sourceCode":"            )\n\n        loop = asyncio.get_running_loop()\n        with self._lock:\n            self._resize_pool_locked(capacity)\n            self._grant_waiters_locked()\n            if not self._waiters:\n                slot = self._take_slot_locked(len(self._unpark_tickets))\n                if slot is not None:\n                    # No snapshot here: callers read it through snapshot_now(),\n                    # which re-reads the queue, so building one per admitted\n                    # request would be pure allocation on the hot path.\n                    return LlamaAdmissionReservation(\n                        queue = self,\n                        lease = LlamaAdmissionLease(self, slot),\n                    )\n            limit = config.queue_limit(self._capacity)\n            if limit is not None and self._live_waiters_locked() >= limit:\n                raise LlamaAdmissionQueueFull(\n                    \"llama-server generation queue is full\",\n                    snapshot = self._snapshot_locked(),\n                )\n            waiter = _Waiter(\n                loop = loop,\n                future = loop.create_future(),\n            )\n            self._waiters.append(waiter)\n            return LlamaAdmissionReservation(\n                queue = self,\n                waiter = waiter,\n            )\n\n    def _release_slot_locked(self, slot: Optional[int]) -> None:\n        # A slot id at or past a shrunk capacity retires instead of returning.\n        if slot is None or not self._in_use >> slot & 1:\n            return\n        self._in_use &= ~(1 << slot)","sourceCodeStart":574,"sourceCodeEnd":610,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/llama_admission.py#L574-L610","documentation":"LlamaAdmissionQueueFull from LlamaAdmissionQueue.reserve (llama_admission.py:592): the admission controller hands out a fixed pool of generation slots (capacity, typically llama-server --parallel); when no slot is free AND the number of live waiters already queued has reached config.queue_limit(capacity), new callers are rejected immediately instead of queueing forever. The limit is max_queue if set, else queue_per_slot * capacity (floored by min_queue); None or <= 0 means unbounded. The exception carries a LlamaAdmissionSnapshot (key, capacity, active, queued, free) for reporting.","triggerScenarios":"Issuing more concurrent llama-server generation requests than slots + queue allowance: e.g. --parallel 1 with the default queue limits and several simultaneous chat requests — the first takes the slot, waiters fill the queue, the next reserve() raises LlamaAdmissionQueueFull.","commonSituations":"Load tests or many simultaneous studio users against a small --parallel; slow/stuck generations holding slots so the queue drains slowly; capacity shrunk while slots are still held (held slots count against the ceiling); queue limits tightened via env (QUEUE_PER_SLOT / MAX_QUEUE) below real traffic.","solutions":["Catch LlamaAdmissionQueueFull at the API layer and return HTTP 429/503 with the snapshot's active/queued counts so clients back off and retry.","Raise capacity: start llama-server with a higher --parallel so more requests get slots and the scaled queue limit grows with it.","Tune the queue: set max_queue (or QUEUE_PER_SLOT / MIN_QUEUE env) higher, or to 0/unset for an unbounded line, if you prefer waiting over rejecting.","Fix slot leaks: ensure every reservation/lease is released in a finally block — leaked leases shrink the effective pool until everything 409s.","Client-side: use exponential backoff with jitter on 429 rather than immediate retries, which only re-fill the queue."],"exampleFix":"# before\nreservation = queue.reserve(capacity = n_parallel, config = cfg)  # raises when full\n\n# after\nfrom core.inference.llama_admission import LlamaAdmissionQueueFull\ntry:\n    reservation = queue.reserve(capacity = n_parallel, config = cfg)\nexcept LlamaAdmissionQueueFull as e:\n    snap = e.snapshot\n    raise HTTPException(429, detail={\n        \"error\": \"generation queue full\",\n        \"active\": snap.active, \"queued\": snap.queued,\n        \"capacity\": snap.capacity,\n        \"retry_after\": 5,\n    })","handlingStrategy":"retry","validationCode":"snap = queue.snapshot_now()\nif snap.queued >= config.queue_limit(snap.capacity or 1):\n    return HTTPException(429, \"queue full, retry later\")","typeGuard":"def queue_full(exc: Exception) -> bool:\n    return type(exc).__name__ == \"LlamaAdmissionQueueFull\"","tryCatchPattern":"from core.inference.llama_admission import LlamaAdmissionQueueFull\ntry:\n    res = queue.reserve(capacity = n, config = cfg)\nexcept LlamaAdmissionQueueFull as e:\n    s = e.snapshot\n    raise HTTPException(429, detail={\"active\": s.active, \"queued\": s.queued,\n        \"retry_after\": backoff_seconds})","preventionTips":["Size --parallel to expected concurrent requests so slots, not the queue, absorb load.","Always release reservations/leases in finally blocks to avoid slot leaks.","Expose 429 + Retry-After and back off with jitter client-side.","Tune max_queue / QUEUE_PER_SLOT env to match whether you prefer queueing or fast rejection."],"tags":["llama-server","admission-control","backpressure","concurrency","capacity"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}