huggingface/transformers · error · RuntimeError

Unknown error

Error message

Unknown error

What it means

The chat CLI consumes a server-sent-events stream from `transformers serve` and expects status events: ready, loading, or error. When an event with status=='error' arrives, it raises RuntimeError with the server's message, falling back to 'Unknown error' when the event carries no 'message' field. This means model loading or server-side inference failed before the session became ready.

Source

Thrown at src/transformers/cli/chat.py:281

            TimeElapsedColumn(),
            console=self._console,
        )
        task_id = progress.add_task(_label("processor"), total=None)
        cached = False

        with Live(progress, console=self._console, transient=True):
            for line in response.iter_lines():
                if not line or not line.startswith(b"data: "):
                    continue
                event = json.loads(line[6:])
                status = event.get("status")

                if status == "ready":
                    cached = event.get("cached", False)
                    break

                if status == "error":
                    raise RuntimeError(event.get("message", "Unknown error"))

                if status == "loading":
                    stage = event.get("stage")
                    prog = event.get("progress")
                    label = _label(stage)

                    if prog:
                        unit = "bytes" if stage == "download" else "items"
                        progress.update(
                            task_id, description=label, completed=prog["current"], total=prog.get("total"), unit=unit
                        )
                    else:
                        progress.update(task_id, description=label, completed=0, total=None)

        if cached:
            self._console.print(Markdown(f"_*{model} was already loaded.*_"))
        else:
            self._console.print(Markdown(f"_*{model} is warm.*_"))

View on GitHub (pinned to a597f97485)

Solutions

1. Check the server logs in the shell running `transformers serve` for the real traceback
2. Verify the model id loads standalone: AutoModelForCausalLM.from_pretrained(<id>)
3. Reduce memory pressure: smaller model, --dtype float16, quantization, or CPU
4. Ensure required extras are installed (bitsandbytes for bnb, kernels) and hub auth for gated repos
5. Restart `transformers serve` and re-run `transformers chat` once fixed

Example fix

# before
transformers serve --model_id bigmodel --dtype bfloat16   # OOM on load
transformers chat

# after
transformers serve --model_id bigmodel --dtype float16 --quantization bnb-4bit
transformers chat

# Pre-flight: confirm the model loads before chatting
import subprocess, sys
model = "<model_id>"
r = subprocess.run([sys.executable, "-c", f"from transformers import AutoConfig; AutoConfig.from_pretrained('{model}')"])
if r.returncode != 0:
    sys.exit("Model unreachable; fix id/auth/network before chat")

try:
    run_chat_session()
except RuntimeError as e:
    if "Unknown error" in str(e) or "error" in str(e).lower():
        log_server_side_failure(); sys.exit(1)
    raise

When it happens

Trigger: Starting `transformers chat` against a server whose model fails to download or load (OOM, corrupted weights, unsupported architecture, bad dtype/device combo); the server sends {"status": "error"} with or without a message; model files missing on disk or hub auth failures during load.

Common situations: GPU out-of-memory while loading a large model; mistyped or nonexistent model id passed to serve; CUDA/dtype mismatches (e.g. bnb-4bit without bitsandbytes); network/proxy issues fetching weights; hub token missing for a gated repo.

Related errors

• No server currently running on {url}. To run a local server,
• You need to install rich to use the chat interface. (`pip in
• The server running on {url} returned status code {output.sta
• Failed to convert `generate_flags` into a valid JSON object.
• Missing dependencies for serving. Install with `pip install

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/128f497c03fec7fe. Report an issue: GitHub.