Lightning-AI/pytorch-lightning · error · ValueError

{mode!r} only works with `dtype=torch.float16`, but you chos

Error message

{mode!r} only works with `dtype=torch.float16`, but you chose `{dtype}`

What it means

BitsandbytesPrecision validates that int8 quantization modes ('int8', 'int8-no-fp16-outlayers') only work with torch.float16 compute dtype. Choosing another dtype (e.g. bf16) raises ValueError, per the bitsandbytes int8 limitation.

Source

Thrown at src/lightning/fabric/plugins/precision/bitsandbytes.py:88

    def __init__(
        self,
        mode: Literal["nf4", "nf4-dq", "fp4", "fp4-dq", "int8", "int8-training"],
        dtype: Optional[torch.dtype] = None,
        ignore_modules: Optional[set[str]] = None,
    ) -> None:
        _import_bitsandbytes()

        if dtype is None:
            # try to be smart about the default selection
            if mode.startswith("int8"):
                dtype = torch.float16
            else:
                dtype = (
                    torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16
                )
        if mode.startswith("int8") and dtype is not torch.float16:
            # this limitation is mentioned in https://huggingface.co/blog/hf-bitsandbytes-integration#usage
            raise ValueError(f"{mode!r} only works with `dtype=torch.float16`, but you chose `{dtype}`")

        globals_ = globals()
        mode_to_cls = {
            "nf4": globals_["_NF4Linear"],
            "nf4-dq": globals_["_NF4DQLinear"],
            "fp4": globals_["_FP4Linear"],
            "fp4-dq": globals_["_FP4DQLinear"],
            "int8-training": globals_["_Linear8bitLt"],
            "int8": globals_["_Int8LinearInference"],
        }
        self._linear_cls = mode_to_cls[mode]
        self.dtype = dtype
        self.ignore_modules = ignore_modules or set()

    @override
    def convert_module(self, module: torch.nn.Module) -> torch.nn.Module:
        # avoid naive users thinking they quantized their model
        if not any(isinstance(m, torch.nn.Linear) for m in module.modules()):

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Pass dtype=torch.float16 explicitly for int8 modes
2. Or switch mode to 'nf4'/'fp4' (4-bit) which support bf16
3. If bf16 is required for stability, use nf4 with dtype=torch.bfloat16

Example fix

# before
plugin = BitsandbytesPrecision(mode="int8")  # auto-picks bf16 on A100

# after
plugin = BitsandbytesPrecision(mode="int8", dtype=torch.float16)
# or
plugin = BitsandbytesPrecision(mode="nf4", dtype=torch.bfloat16)

import torch
mode, dtype = "int8", torch.bfloat16
if mode.startswith("int8") and dtype is not torch.float16:
    dtype = torch.float16  # or pick a 4-bit mode
plugin = BitsandbytesPrecision(mode=mode, dtype=dtype)

def bnb_mode_dtype_ok(mode: str, dtype: torch.dtype) -> bool:
    return not (mode.startswith("int8") and dtype is not torch.float16)

When it happens

Trigger: BitsandbytesPrecision(mode='int8', dtype=torch.bfloat16) or relying on auto-dtype selection that picks bf16 on Ampere+ GPUs while mode starts with 'int8'.

Common situations: On A100/H100 GPUs where bfloat16 is auto-selected because torch.cuda.is_bf16_supported() is True, so the user gets this error without ever passing dtype explicitly.

Related errors

• You are using the bitsandbytes precision plugin, but your mo
• Instantiating your model under the `init_module` context man
• str(_BITSANDBYTES_AVAILABLE)
• Bitsandbytes is only supported on CUDA GPUs.
• Cannot set the dtype explicitly. Please use module.to(new_dt

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/ea65319ffdae4e65. Report an issue: GitHub.