huggingface/transformers · error · ValueError

TP and DP cannot be used together

Error message

TP and DP cannot be used together

What it means

Raised by `validate_bnb_compute_capabilities`/device support validation in transformers' bitsandbytes integration: the set of devices actually available on the machine (from `get_available_devices()`, e.g. {cuda, mps, xpu, npu}) has no intersection with `bnb.supported_torch_devices` reported by the installed bitsandbytes package. In other words, your installed bitsandbytes build does not support any device present on the system. The error is only raised when `raise_exception=True`; otherwise it logs a warning and returns False.

Source

Thrown at benchmark_v2/benchmark_scripts/continuous_batching_overall.py:175

    time_seconds: float | None = None
    num_tokens: int | None = None
    throughput_tok_per_sec: float | None = None
    peak_memory_gb: float | None = None
    accuracy: float | None = None
    error: str | None = None


class BenchmarkResults:
    """Holds all CB benchmark runs and the shared model they execute against."""

    def __init__(self, model_id: str, attn_impl: str, tp_size: int = 1, dp_size: int = 1):
        self.model_id = model_id
        self.attn_impl = attn_impl
        self.tp_size = tp_size
        self.dp_size = dp_size
        # For now, TP and DP are mutually exclusive
        if self.tp_size > 1 and self.dp_size > 1:
            raise ValueError("TP and DP cannot be used together")
        # torchrun sets these per worker
        self.global_rank = int(os.environ.get("RANK", 0))
        self.local_rank = int(os.environ.get("LOCAL_RANK", 0))
        # Pin this worker to its own GPU and open a process group to gather results later
        if self.dp_size > 1:
            disable_progress_bar()
            torch.cuda.set_device(self.local_rank)
            if not torch.distributed.is_initialized():  # type: ignore
                torch.distributed.init_process_group(backend="gloo")  # type: ignore
        # Entries accumulator
        self.entries: list[BenchmarkEntry] = []

    def cleanup(self) -> None:
        torch.cuda.empty_cache()
        gc.collect()
        torch.cuda.reset_peak_memory_stats()

    def _get_model(self) -> Any:

View on GitHub (pinned to a597f97485)

Solutions

1. Upgrade bitsandbytes to a version whose `supported_torch_devices` includes your device (e.g. `pip install -U bitsandbytes`; >=0.43 adds cpu/mpu backends, newer releases add more) per https://huggingface.co/docs/bitsandbytes/main/en/installation
2. Verify the device is actually visible: `python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"` and fix the CUDA/driver install if not
3. If you are on a non-CUDA accelerator, install the backend-specific bnb build documented for it (xpu/npu/rocm)
4. If you did not intend to quantize, remove `BitsAndBytesConfig` / `load_in_8bit`/`load_in_4bit` from the `from_pretrained` call so bnb is never loaded

Example fix

// before
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8B",
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),  # old bnb, CPU-only box
)

// after
# pip install -U bitsandbytes  (or drop quantization on unsupported devices)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")

import bitsandbytes as bnb
from transformers.integrations.bitsandbytes import get_available_devices

avail = set(get_available_devices())
supported = set(getattr(bnb, "supported_torch_devices", set()))
if not avail & supported:
    raise SystemExit("bitsandbytes does not support this machine's devices; skipping quantized load")

try:
    model = AutoModelForCausalLM.from_pretrained(name, quantization_config=bnb_cfg)
except RuntimeError as e:
    if "supported by the bitsandbytes version" in str(e):
        model = AutoModelForCausalLM.from_pretrained(name)  # unquantized fallback
    else:
        raise

When it happens

Trigger: Calling a quantization path that loads bitsandbytes (e.g. loading a model with `BitsAndBytesConfig`, `load_in_8bit=True`/`load_in_4bit=True`) on a machine whose devices are all unsupported by the installed bnb build — e.g. an old bitsandbytes (<0.43) that only lists `cuda` while running on CPU-only/MPS, or a bnb build without the backend for mps/xpu/npu. Also triggered when CUDA is not actually visible (driver/cudart mismatch) so `get_available_devices()` returns something bnb does not list.

Common situations: CPU-only machine or Mac (MPS) trying 4/8-bit loading with an old `bitsandbytes`; newer bnb version needed for AMD/Intel/NPU backends but not installed; broken CUDA install so torch reports no cuda device; CI runners without GPUs running quantization tests.

Related errors

• No baseline with name '{name}' in {RESULTS_DIR}
• CUDA is not available in this environment; cannot export to
• Tensor parallelism was requested, but WORLD_SIZE is not set
• You need to install optimum-quanto in order to use KV cache
• `nbits` for `quanto` backend has to be one of [`2`, `4`] but

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/c0a9c5a8b396a5cc. Report an issue: GitHub.