{"record":{"id":"8d2942dff2fc49b1","repo":"huggingface/transformers","slug":"cb-worker-died-during-request-request-id-resul","errorCode":null,"errorMessage":"CB worker died during request {request_id}: {result.error}","messagePattern":"CB worker died during request (.+?): (.+?)","errorType":"http","errorClass":"CBWorkerDeadError","httpStatus":503,"severity":"critical","filePath":"src/transformers/cli/serving/utils.py","lineNumber":959,"sourceCode":"        cb.register_result_handler(request_id, _on_result)\n\n        cb.add_request(\n            input_ids,\n            request_id=request_id,\n            max_new_tokens=gen_config.max_new_tokens,\n            streaming=False,\n            eos_token_id=gen_config.eos_token_id,\n        )\n        result = await future\n        # CB signals a failed request by setting ``error`` (and ``status = FAILED``) on the\n        # delivered GenerationOutput, often with empty ``generated_tokens``. Surface it instead\n        # of returning an empty success that downstream parsing/decoding would silently mask.\n        # If the worker itself died, route to CBWorkerDeadError so the client gets the same 503\n        # as requests submitted post-crash; otherwise it's a per-request failure (e.g. unsupported\n        # logit-processor kwarg) and a plain RuntimeError -> 500 is appropriate.\n        if result.error is not None:\n            if cb.fatal_error is not None:\n                raise CBWorkerDeadError(f\"CB worker died during request {request_id}: {result.error}\")\n            raise RuntimeError(f\"CB generation failed for {request_id}: {result.error}\")\n        generated_ids = result.generated_tokens\n        text = processor.decode(generated_ids, skip_special_tokens=True)\n        return text, input_len, generated_ids\n\n    @property\n    def scheduler(self) -> \"Scheduler\":\n        \"\"\"The CB scheduler (for testing/monitoring).\"\"\"\n        if self._cb is None or self._cb.batch_processor is None:\n            raise RuntimeError(\"Continuous batching processor not initialized.\")\n        return self._cb.batch_processor.scheduler\n\n    def stop(self) -> None:\n        if self._cb is not None:\n            self._cb.stop(block=True, timeout=2)\n\n\nclass GenerationState:","sourceCodeStart":941,"sourceCodeEnd":977,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/cli/serving/utils.py#L941-L977","documentation":"CBWorkerDeadError raised after awaiting a CB generation future when the delivered result carries an error and the worker's fatal_error is set — i.e. the worker died while this specific request was in flight. It is distinguished from a plain per-request RuntimeError (result.error with a healthy worker) so both map to a 503 with the underlying failure attached.","triggerScenarios":"A chat/generation request completes with status FAILED and result.error set, and cb.fatal_error is not None at delivery time — typically the worker OOMed or hit a CUDA fault mid-batch, taking in-flight requests down with it.","commonSituations":"Long generations that exhaust GPU memory mid-batch; batch sizes tuned too aggressively; intermittent driver faults that kill the worker during peak load.","solutions":["Read the appended result.error to identify the worker-fatal cause (OOM is most common)","Lower CB capacity (max batch size, max sequence length) or move to a model that fits memory, then restart","Idempotent clients may retry once on a fresh server instance; treat the 503 as 'worker gone', not 'request invalid'"],"exampleFix":"# before\nmax_batch_size = 64  # OOMs mid-batch\n# after\nmax_batch_size = 16  # leaves headroom for long generations","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for attempt in range(2):\n    try:\n        return client.post('/v1/chat/completions', json=body, timeout=60)\n    except ServiceUnavailable as e:\n        if 'CB worker died during request' not in str(e):\n            raise\n        wait_for_server_ready(base)  # worker must restart before retrying once","preventionTips":["Keep GPU memory headroom for peak batch load to prevent mid-batch OOM","Make generation requests idempotent so a single restart-and-retry is safe","Log the embedded result.error — it names the true worker-fatal cause"],"tags":["serving","continuous-batching","crash","oom","http-503"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}