{"record":{"id":"c0a9c5a8b396a5cc","repo":"huggingface/transformers","slug":"tp-and-dp-cannot-be-used-together","errorCode":null,"errorMessage":"TP and DP cannot be used together","messagePattern":"TP and DP cannot be used together","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"benchmark_v2/benchmark_scripts/continuous_batching_overall.py","lineNumber":175,"sourceCode":"    time_seconds: float | None = None\n    num_tokens: int | None = None\n    throughput_tok_per_sec: float | None = None\n    peak_memory_gb: float | None = None\n    accuracy: float | None = None\n    error: str | None = None\n\n\nclass BenchmarkResults:\n    \"\"\"Holds all CB benchmark runs and the shared model they execute against.\"\"\"\n\n    def __init__(self, model_id: str, attn_impl: str, tp_size: int = 1, dp_size: int = 1):\n        self.model_id = model_id\n        self.attn_impl = attn_impl\n        self.tp_size = tp_size\n        self.dp_size = dp_size\n        # For now, TP and DP are mutually exclusive\n        if self.tp_size > 1 and self.dp_size > 1:\n            raise ValueError(\"TP and DP cannot be used together\")\n        # torchrun sets these per worker\n        self.global_rank = int(os.environ.get(\"RANK\", 0))\n        self.local_rank = int(os.environ.get(\"LOCAL_RANK\", 0))\n        # Pin this worker to its own GPU and open a process group to gather results later\n        if self.dp_size > 1:\n            disable_progress_bar()\n            torch.cuda.set_device(self.local_rank)\n            if not torch.distributed.is_initialized():  # type: ignore\n                torch.distributed.init_process_group(backend=\"gloo\")  # type: ignore\n        # Entries accumulator\n        self.entries: list[BenchmarkEntry] = []\n\n    def cleanup(self) -> None:\n        torch.cuda.empty_cache()\n        gc.collect()\n        torch.cuda.reset_peak_memory_stats()\n\n    def _get_model(self) -> Any:","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/benchmark_v2/benchmark_scripts/continuous_batching_overall.py#L157-L193","documentation":"Raised by `validate_bnb_compute_capabilities`/device support validation in transformers' bitsandbytes integration: the set of devices actually available on the machine (from `get_available_devices()`, e.g. {cuda, mps, xpu, npu}) has no intersection with `bnb.supported_torch_devices` reported by the installed bitsandbytes package. In other words, your installed bitsandbytes build does not support any device present on the system. The error is only raised when `raise_exception=True`; otherwise it logs a warning and returns False.","triggerScenarios":"Calling a quantization path that loads bitsandbytes (e.g. loading a model with `BitsAndBytesConfig`, `load_in_8bit=True`/`load_in_4bit=True`) on a machine whose devices are all unsupported by the installed bnb build — e.g. an old bitsandbytes (<0.43) that only lists `cuda` while running on CPU-only/MPS, or a bnb build without the backend for mps/xpu/npu. Also triggered when CUDA is not actually visible (driver/cudart mismatch) so `get_available_devices()` returns something bnb does not list.","commonSituations":"CPU-only machine or Mac (MPS) trying 4/8-bit loading with an old `bitsandbytes`; newer bnb version needed for AMD/Intel/NPU backends but not installed; broken CUDA install so torch reports no cuda device; CI runners without GPUs running quantization tests.","solutions":["Upgrade bitsandbytes to a version whose `supported_torch_devices` includes your device (e.g. `pip install -U bitsandbytes`; >=0.43 adds cpu/mpu backends, newer releases add more) per https://huggingface.co/docs/bitsandbytes/main/en/installation","Verify the device is actually visible: `python -c \"import torch; print(torch.cuda.is_available(), torch.cuda.device_count())\"` and fix the CUDA/driver install if not","If you are on a non-CUDA accelerator, install the backend-specific bnb build documented for it (xpu/npu/rocm)","If you did not intend to quantize, remove `BitsAndBytesConfig` / `load_in_8bit`/`load_in_4bit` from the `from_pretrained` call so bnb is never loaded"],"exampleFix":"// before\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama/Llama-3-8B\",\n    quantization_config=BitsAndBytesConfig(load_in_8bit=True),  # old bnb, CPU-only box\n)\n\n// after\n# pip install -U bitsandbytes  (or drop quantization on unsupported devices)\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-3-8B\")","handlingStrategy":"validation","validationCode":"import bitsandbytes as bnb\nfrom transformers.integrations.bitsandbytes import get_available_devices\n\navail = set(get_available_devices())\nsupported = set(getattr(bnb, \"supported_torch_devices\", set()))\nif not avail & supported:\n    raise SystemExit(\"bitsandbytes does not support this machine's devices; skipping quantized load\")","typeGuard":null,"tryCatchPattern":"try:\n    model = AutoModelForCausalLM.from_pretrained(name, quantization_config=bnb_cfg)\nexcept RuntimeError as e:\n    if \"supported by the bitsandbytes version\" in str(e):\n        model = AutoModelForCausalLM.from_pretrained(name)  # unquantized fallback\n    else:\n        raise","preventionTips":["Verify torch.cuda.is_available() and device availability before requesting 4/8-bit loading","Pin a recent bitsandbytes version in requirements when using non-CUDA backends","Gate quantized loading behind a device-support check in shared codepaths"],"tags":["bitsandbytes","quantization","hardware","cuda","installation"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}