huggingface/transformers · critical · CBWorkerDeadError

CB worker died during request {request_id}: {result.error}

Error message

CB worker died during request {request_id}: {result.error}

What it means

CBWorkerDeadError raised after awaiting a CB generation future when the delivered result carries an error and the worker's fatal_error is set — i.e. the worker died while this specific request was in flight. It is distinguished from a plain per-request RuntimeError (result.error with a healthy worker) so both map to a 503 with the underlying failure attached.

Source

Thrown at src/transformers/cli/serving/utils.py:959

        cb.register_result_handler(request_id, _on_result)

        cb.add_request(
            input_ids,
            request_id=request_id,
            max_new_tokens=gen_config.max_new_tokens,
            streaming=False,
            eos_token_id=gen_config.eos_token_id,
        )
        result = await future
        # CB signals a failed request by setting ``error`` (and ``status = FAILED``) on the
        # delivered GenerationOutput, often with empty ``generated_tokens``. Surface it instead
        # of returning an empty success that downstream parsing/decoding would silently mask.
        # If the worker itself died, route to CBWorkerDeadError so the client gets the same 503
        # as requests submitted post-crash; otherwise it's a per-request failure (e.g. unsupported
        # logit-processor kwarg) and a plain RuntimeError -> 500 is appropriate.
        if result.error is not None:
            if cb.fatal_error is not None:
                raise CBWorkerDeadError(f"CB worker died during request {request_id}: {result.error}")
            raise RuntimeError(f"CB generation failed for {request_id}: {result.error}")
        generated_ids = result.generated_tokens
        text = processor.decode(generated_ids, skip_special_tokens=True)
        return text, input_len, generated_ids

    @property
    def scheduler(self) -> "Scheduler":
        """The CB scheduler (for testing/monitoring)."""
        if self._cb is None or self._cb.batch_processor is None:
            raise RuntimeError("Continuous batching processor not initialized.")
        return self._cb.batch_processor.scheduler

    def stop(self) -> None:
        if self._cb is not None:
            self._cb.stop(block=True, timeout=2)


class GenerationState:

View on GitHub (pinned to a597f97485)

Solutions

1. Read the appended result.error to identify the worker-fatal cause (OOM is most common)
2. Lower CB capacity (max batch size, max sequence length) or move to a model that fits memory, then restart
3. Idempotent clients may retry once on a fresh server instance; treat the 503 as 'worker gone', not 'request invalid'

Example fix

# before
max_batch_size = 64  # OOMs mid-batch
# after
max_batch_size = 16  # leaves headroom for long generations

for attempt in range(2):
    try:
        return client.post('/v1/chat/completions', json=body, timeout=60)
    except ServiceUnavailable as e:
        if 'CB worker died during request' not in str(e):
            raise
        wait_for_server_ready(base)  # worker must restart before retrying once

When it happens

Trigger: A chat/generation request completes with status FAILED and result.error set, and cb.fatal_error is not None at delivery time — typically the worker OOMed or hit a CUDA fault mid-batch, taking in-flight requests down with it.

Common situations: Long generations that exhaust GPU memory mid-batch; batch sizes tuned too aggressively; intermittent driver faults that kill the worker during peak load.

Related errors

• CB worker is dead and cannot accept request {request_id}: {s
• Audio transcription requires sequential generation (not CB)
• Unknown error
• No server currently running on {url}. To run a local server,
• Missing dependencies for serving. Install with `pip install

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/8d2942dff2fc49b1. Report an issue: GitHub.