huggingface/transformers · critical · CBWorkerDeadError

CB worker is dead and cannot accept request {request_id}: {s

Error message

CB worker is dead and cannot accept request {request_id}: {self._cb.fatal_error}

What it means

CBWorkerDeadError raised at request entry when the continuous-batching worker process has a non-None fatal_error. Once the CB worker dies, every queued or new request would silently hang, so the guard fails fast with (typically) a 503 instead. The message embeds the worker's original fatal error for diagnosis.

Source

Thrown at src/transformers/cli/serving/utils.py:856

            return

        self._cb = model.init_continuous_batching(
            generation_config=gen_config, continuous_batching_config=self._cb_config
        )
        self._cb.start()

    def is_alive(self) -> bool:
        """Whether the CB worker is healthy. ``True`` before ``init_cb()`` is called."""
        return self._cb is None or self._cb.fatal_error is None

    def _check_alive(self, request_id: str) -> None:
        """Raise :class:`CBWorkerDeadError` if the CB worker has died.

        Called at request entry to fail fast — submitting to a dead worker would otherwise
        enqueue the request into a void where it never gets processed.
        """
        if self._cb is not None and self._cb.fatal_error is not None:
            raise CBWorkerDeadError(
                f"CB worker is dead and cannot accept request {request_id}: {self._cb.fatal_error}"
            )

    def generate_streaming(
        self,
        model: "PreTrainedModel",
        processor: "ProcessorMixin | PreTrainedTokenizerFast",
        inputs: dict,
        gen_config: "GenerationConfig",
        request_id: str,
        response_parser: "ResponseParser | None" = None,
    ) -> tuple[asyncio.Queue, CBStreamer]:
        """Start streaming CB generation. Registers a per-request output handler."""
        cb = self._cb
        if cb is None:
            raise RuntimeError("CB manager not initialized. Call `init_cb()` first.")
        self._check_alive(request_id)

View on GitHub (pinned to a597f97485)

Solutions

Inspect the embedded fatal_error in the message to find the root cause (usually OOM or a CUDA fault)
Reduce memory pressure (smaller max batch / max length, smaller model, more GPU memory) and restart the server
Catch CBWorkerDeadError client-side and treat it as 503: stop retrying immediately and alert instead of hammering the server

Example fix

# before
resp = client.post('/v1/chat/completions', json=body)  # keeps retrying a dead worker
# after
try:
    resp = client.post('/v1/chat/completions', json=body)
except HTTPError as e:
    if e.response.status_code == 503:
        alert('CB worker dead, restart serving process')
    raise

Defensive patterns

Strategy: try-catch

Try / catch

try:
    resp = client.post('/v1/chat/completions', json=body, timeout=30)
except (httpx.HTTPStatusError, requests.HTTPError) as e:
    if getattr(e.response, 'status_code', None) == 503 and 'CB worker is dead' in e.response.text:
        alert_and_stop_retries(e.response.text)  # worker crashed; needs restart
    raise

Prevention

Monitor the CB worker process and auto-restart the server on fatal_error
Cap batch size and sequence length to avoid worker OOM
Treat 503 'worker dead' as a terminal condition, not a transient retry candidate

When it happens

Trigger: Any generation request (e.g. /v1/chat/completions) submitted after the CB worker crashed earlier — OOM kill, CUDA error, or an unhandled exception in the batch processor set fatal_error on the worker handle.

Common situations: GPU out-of-memory inside the CB worker killing the process; a driver/runtime crash; requests continuing to arrive after the worker died because the HTTP server itself is still healthy.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/d39e91eff047b5de. Report an issue: GitHub.