hiyouga/LlamaFactory · error · ValueError

KTransformers is incompatible with DeepSpeed ZeRO-3.

Error message

KTransformers is incompatible with DeepSpeed ZeRO-3.

What it means

Raised in get_train_args when model_args.use_kt is true and DeepSpeed ZeRO-3 is enabled in the training config. KTransformers offloads layers to custom CPU/GPU kernels, which is fundamentally incompatible with ZeRO-3's sharding of parameters across processes. The check runs before any training starts, so it fails fast at argument-parsing time.

Source

Thrown at src/llamafactory/hparams/parser.py:536

    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):
        raise ValueError("GaLore and APOLLO are incompatible with DeepSpeed yet.")

    if (
        not finetuning_args.use_mca
        and not finetuning_args.use_megatron_bridge
        and training_args.fp8
        and model_args.quantization_bit is not None
    ):
        raise ValueError("FP8 training is not compatible with quantization. Please disable one of them.")

    if model_args.infer_backend != EngineName.HF:
        raise ValueError("vLLM/SGLang backend is only available for API, CLI and Web.")

    if model_args.use_unsloth and is_deepspeed_zero3_enabled():
        raise ValueError("Unsloth is incompatible with DeepSpeed ZeRO-3.")

    if model_args.use_kt and is_deepspeed_zero3_enabled():
        raise ValueError("KTransformers is incompatible with DeepSpeed ZeRO-3.")

    _set_env_vars()
    _verify_model_args(model_args, data_args, finetuning_args)
    _check_extra_dependencies(model_args, finetuning_args, training_args)
    _verify_trackio_args(training_args)

    if (
        not finetuning_args.use_mca
        and not finetuning_args.use_megatron_bridge
        and training_args.fp8_enable_fsdp_float8_all_gather
        and not training_args.fp8
    ):
        logger.warning_rank0("fp8_enable_fsdp_float8_all_gather requires fp8=True. Setting fp8=True.")
        model_args.fp8 = True

    if (
        training_args.do_train
        and finetuning_args.finetuning_type == "lora"

View on GitHub (pinned to f28afaf635)

Solutions

1. Switch the DeepSpeed config to ZeRO stage 0/1/2 (or remove the deepspeed argument entirely) when using KTransformers.
2. Remove enable_kt: true from the model arguments and let the standard HF/DeepSpeed path load the model.
3. Use ZeRO-2 with offload if memory was the reason for ZeRO-3, since KT already handles its own layer placement.

Example fix

# before
model_args:
  enable_kt: true
deepspeed: examples/deepspeed/ds_z3_config.json

# after
model_args:
  enable_kt: true
deepspeed: examples/deepspeed/ds_z2_config.json

# before launching training
import json, os
if cfg.get("model_args", {}).get("enable_kt"):
    ds = cfg.get("deepspeed")
    if ds and os.path.isfile(ds):
        stage = json.load(open(ds))["zero_optimization"]["stage"]
        assert stage != 3, "KTransformers cannot be combined with ZeRO-3; use stage <= 2"

When it happens

Trigger: Set enable_kt: true (use_kt) in the model section of a YAML training config while the DeepSpeed config file (e.g. ds_z3_config.json with zero_optimization.stage=3) is passed via deepspeed. Any combination of use_kt + stage-3 ZeRO triggers it.

Common situations: Copying a ZeRO-3 multi-GPU example config and adding KTransformers flags to speed up local training; upgrading a previously working KT setup after switching to a ZeRO-3 deepspeed stage to fit a larger model.

Related errors

• DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized
• Only 4-bit quantized model can use fsdp+qlora or auto device
• HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FS
• EETQ quantization is incompatible with DeepSpeed ZeRO-3 or F
• DeepSpeed config_file is required in dist_config

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/5d516b217064aa32. Report an issue: GitHub.