{"record":{"id":"72ab507dca416e5f","repo":"hiyouga/LlamaFactory","slug":"cannot-use-device-map-for-quantized-models-in-trai","errorCode":null,"errorMessage":"Cannot use device map for quantized models in training.","messagePattern":"Cannot use device map for quantized models in training\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":493,"sourceCode":"        raise ValueError(\"Please specify `max_steps` in streaming mode.\")\n\n    if training_args.do_train and data_args.dataset is None:\n        raise ValueError(\"Please specify dataset for training.\")\n\n    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (\n        data_args.eval_dataset is None and data_args.val_size < 1e-6\n    ):\n        raise ValueError(\"Please make sure eval_dataset be provided or val_size >1e-6\")\n\n    if training_args.predict_with_generate:\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.\")\n\n        if finetuning_args.compute_accuracy:\n            raise ValueError(\"Cannot use `predict_with_generate` and `compute_accuracy` together.\")\n\n    if training_args.do_train and model_args.quantization_device_map == \"auto\":\n        raise ValueError(\"Cannot use device map for quantized models in training.\")\n\n    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.\")\n\n    if finetuning_args.pure_bf16:\n        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):\n            raise ValueError(\"This device does not support `pure_bf16`.\")\n\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`pure_bf16` is incompatible with DeepSpeed ZeRO-3.\")\n\n    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:\n        if finetuning_args.use_galore and finetuning_args.galore_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise GaLore.\")\n\n        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise APOLLO.\")\n","sourceCodeStart":475,"sourceCodeEnd":511,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L475-L511","documentation":"Raised in parser.py:493 when do_train is true and model_args.quantization_device_map == \"auto\". `device_map: auto` scatters a quantized (GPTQ/AWQ/bitsandbytes) model across devices, which breaks gradient checkpointing/optimizer state placement during training.","triggerScenarios":"Config with `quantization_bit: 4/8` (or quantization_method) together with `device_map: auto` while do_train is set — usually copied from an inference config.","commonSituations":"Reusing a chat/inference YAML (where device_map: auto is common for offloading large quantized models on limited VRAM) as the base for LoRA training; adding quantization to save memory and keeping the inference-style device_map key.","solutions":["Remove `device_map: auto` from the training config; let LlamaFactory place the quantized model on the training device","If memory is the reason for device_map, lower quantization_bit, enable LoRA (already typical), reduce per_device batch, or use gradient checkpointing instead"],"exampleFix":"# before (YAML)\nquantization_bit: 4\ndevice_map: auto\ndo_train: true\n\n# after\nquantization_bit: 4\ndo_train: true\n# device_map removed","handlingStrategy":"validation","validationCode":"if config.get(\"do_train\") and config.get(\"device_map\") == \"auto\":\n    raise SystemExit(\"device_map: auto is inference-only; remove it for training\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never inherit inference YAMLs for training jobs","Strip device_map when adding quantization_bit to a training config"],"tags":["quantization","device-map","qlora","training","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}