hiyouga/LlamaFactory · error · ValueError

This device does not support `pure_bf16`.

Error message

This device does not support `pure_bf16`.

What it means

Raised in parser.py:500 when finetuning_args.pure_bf16 is set but the hardware does not support bfloat16: neither is_torch_bf16_gpu_available() nor (NPU with torch.npu.is_bf16_supported()). pure_bf16 (DeepSpeed-style full-bf16 training) is a hardware capability, not just a dtype switch.

Source

Thrown at src/llamafactory/hparams/parser.py:500

    ):
        raise ValueError("Please make sure eval_dataset be provided or val_size >1e-6")

    if training_args.predict_with_generate:
        if is_deepspeed_zero3_enabled():
            raise ValueError("`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.")

        if finetuning_args.compute_accuracy:
            raise ValueError("Cannot use `predict_with_generate` and `compute_accuracy` together.")

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:
            raise ValueError("Distributed training does not support layer-wise APOLLO.")

        if finetuning_args.use_badam:
            if finetuning_args.badam_mode == "ratio":
                raise ValueError("Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.")
            elif not is_deepspeed_zero3_enabled():
                raise ValueError("Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.")

    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove `pure_bf16: true` and keep default mixed precision (bf16 auto-falls back or use fp16: true on Ampere- hardware)
2. If bf16 is required, switch to an Ampere+ GPU (A100, A10, RTX 30xx+, H100) or a bf16-capable NPU
3. Set `fp16: true` instead for pre-Ampere cards

Example fix

# before (YAML)
pure_bf16: true  # on T4/V100

# after
# pure_bf16 removed
fp16: true

import torch
bf16_ok = torch.cuda.is_available() and torch.cuda.is_bf16_supported()
if config.get("pure_bf16") and not bf16_ok:
    raise SystemExit("pure_bf16 needs a bf16-capable GPU (Ampere+); use fp16 here")

When it happens

Trigger: `pure_bf16: true` on pre-Ampere GPUs (e.g. Tesla T4, V100, GTX series) or unsupported accelerators; also on NPUs whose CANN/torch build lacks bf16 support.

Common situations: Copying a config tuned on an A100/H100 and running it on a cloud T4 instance or an older on-prem GPU; Colab/kaggle T4 sessions.

Related errors

• FP8 training is not compatible with quantization. Please dis
• `virtual_pipeline_model_parallel_size` must be >= 1 when set
• `sequence_parallel` requires `tensor_model_parallel_size` >
• `recompute_granularity` must be 'full' or 'selective'.
• `recompute_method` must be 'uniform' or 'block'.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/14e138708f46428e. Report an issue: GitHub.