hiyouga/LlamaFactory · error · ValueError

Cannot use device map for quantized models in training.

Error message

Cannot use device map for quantized models in training.

What it means

Raised in parser.py:493 when do_train is true and model_args.quantization_device_map == "auto". `device_map: auto` scatters a quantized (GPTQ/AWQ/bitsandbytes) model across devices, which breaks gradient checkpointing/optimizer state placement during training.

Source

Thrown at src/llamafactory/hparams/parser.py:493

        raise ValueError("Please specify `max_steps` in streaming mode.")

    if training_args.do_train and data_args.dataset is None:
        raise ValueError("Please specify dataset for training.")

    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (
        data_args.eval_dataset is None and data_args.val_size < 1e-6
    ):
        raise ValueError("Please make sure eval_dataset be provided or val_size >1e-6")

    if training_args.predict_with_generate:
        if is_deepspeed_zero3_enabled():
            raise ValueError("`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.")

        if finetuning_args.compute_accuracy:
            raise ValueError("Cannot use `predict_with_generate` and `compute_accuracy` together.")

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:
            raise ValueError("Distributed training does not support layer-wise APOLLO.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove `device_map: auto` from the training config; let LlamaFactory place the quantized model on the training device
2. If memory is the reason for device_map, lower quantization_bit, enable LoRA (already typical), reduce per_device batch, or use gradient checkpointing instead

Example fix

# before (YAML)
quantization_bit: 4
device_map: auto
do_train: true

# after
quantization_bit: 4
do_train: true
# device_map removed

if config.get("do_train") and config.get("device_map") == "auto":
    raise SystemExit("device_map: auto is inference-only; remove it for training")

When it happens

Trigger: Config with `quantization_bit: 4/8` (or quantization_method) together with `device_map: auto` while do_train is set — usually copied from an inference config.

Common situations: Reusing a chat/inference YAML (where device_map: auto is common for offloading large quantized models on limited VRAM) as the base for LoRA training; adding quantization to save memory and keeping the inference-style device_map key.

Related errors

• Quantization is only compatible with the LoRA or OFT method.
• Please specify dataset for training.
• FP8 training is not compatible with quantization. Please dis
• vLLM/SGLang backend is only available for API, CLI and Web.
• Quantized models can only be used for the LoRA or OFT tuning

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/72ab507dca416e5f. Report an issue: GitHub.