{"record":{"id":"ea65319ffdae4e65","repo":"Lightning-AI/pytorch-lightning","slug":"mode-r-only-works-with-dtype-torch-float16-bu","errorCode":null,"errorMessage":"{mode!r} only works with `dtype=torch.float16`, but you chose `{dtype}`","messagePattern":"(.+?) only works with `dtype=torch\\.float16`, but you chose `(.+?)`","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/plugins/precision/bitsandbytes.py","lineNumber":88,"sourceCode":"    def __init__(\n        self,\n        mode: Literal[\"nf4\", \"nf4-dq\", \"fp4\", \"fp4-dq\", \"int8\", \"int8-training\"],\n        dtype: Optional[torch.dtype] = None,\n        ignore_modules: Optional[set[str]] = None,\n    ) -> None:\n        _import_bitsandbytes()\n\n        if dtype is None:\n            # try to be smart about the default selection\n            if mode.startswith(\"int8\"):\n                dtype = torch.float16\n            else:\n                dtype = (\n                    torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16\n                )\n        if mode.startswith(\"int8\") and dtype is not torch.float16:\n            # this limitation is mentioned in https://huggingface.co/blog/hf-bitsandbytes-integration#usage\n            raise ValueError(f\"{mode!r} only works with `dtype=torch.float16`, but you chose `{dtype}`\")\n\n        globals_ = globals()\n        mode_to_cls = {\n            \"nf4\": globals_[\"_NF4Linear\"],\n            \"nf4-dq\": globals_[\"_NF4DQLinear\"],\n            \"fp4\": globals_[\"_FP4Linear\"],\n            \"fp4-dq\": globals_[\"_FP4DQLinear\"],\n            \"int8-training\": globals_[\"_Linear8bitLt\"],\n            \"int8\": globals_[\"_Int8LinearInference\"],\n        }\n        self._linear_cls = mode_to_cls[mode]\n        self.dtype = dtype\n        self.ignore_modules = ignore_modules or set()\n\n    @override\n    def convert_module(self, module: torch.nn.Module) -> torch.nn.Module:\n        # avoid naive users thinking they quantized their model\n        if not any(isinstance(m, torch.nn.Linear) for m in module.modules()):","sourceCodeStart":70,"sourceCodeEnd":106,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/plugins/precision/bitsandbytes.py#L70-L106","documentation":"BitsandbytesPrecision validates that int8 quantization modes ('int8', 'int8-no-fp16-outlayers') only work with torch.float16 compute dtype. Choosing another dtype (e.g. bf16) raises ValueError, per the bitsandbytes int8 limitation.","triggerScenarios":"BitsandbytesPrecision(mode='int8', dtype=torch.bfloat16) or relying on auto-dtype selection that picks bf16 on Ampere+ GPUs while mode starts with 'int8'.","commonSituations":"On A100/H100 GPUs where bfloat16 is auto-selected because torch.cuda.is_bf16_supported() is True, so the user gets this error without ever passing dtype explicitly.","solutions":["Pass dtype=torch.float16 explicitly for int8 modes","Or switch mode to 'nf4'/'fp4' (4-bit) which support bf16","If bf16 is required for stability, use nf4 with dtype=torch.bfloat16"],"exampleFix":"# before\nplugin = BitsandbytesPrecision(mode=\"int8\")  # auto-picks bf16 on A100\n\n# after\nplugin = BitsandbytesPrecision(mode=\"int8\", dtype=torch.float16)\n# or\nplugin = BitsandbytesPrecision(mode=\"nf4\", dtype=torch.bfloat16)","handlingStrategy":"validation","validationCode":"import torch\nmode, dtype = \"int8\", torch.bfloat16\nif mode.startswith(\"int8\") and dtype is not torch.float16:\n    dtype = torch.float16  # or pick a 4-bit mode\nplugin = BitsandbytesPrecision(mode=mode, dtype=dtype)","typeGuard":"def bnb_mode_dtype_ok(mode: str, dtype: torch.dtype) -> bool:\n    return not (mode.startswith(\"int8\") and dtype is not torch.float16)","tryCatchPattern":null,"preventionTips":["Always pass dtype explicitly when using int8 modes","On bf16-capable GPUs remember auto-selection picks bf16; override it"],"tags":["bitsandbytes","quantization","int8","dtype","pytorch-lightning"],"backgroundTag":"incompatible-quantization-config","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}