hiyouga/LlamaFactory · error · ValueError

`quantization_bit` cannot be combined with KT weight caches.

Error message

`quantization_bit` cannot be combined with KT weight caches.

What it means

For FP8 PTQ checkpoints, LlamaFactory can skip dequantization and consume KT-managed weight caches (a proprietary fast-loading path). That path stores pre-quantized weights, so additionally requesting on-the-fly quantization_bit would be contradictory; the code raises ValueError when quantization_bit is set together with KT weight caches.

Source

Thrown at src/llamafactory/model/model_utils/quantization.py:117

        quant_method = quantization_config.get("quant_method", "")

        if quant_method not in (QuantizationMethod.MXFP4, QuantizationMethod.FP8) and (
            is_deepspeed_zero3_enabled() or is_fsdp_enabled()
        ):
            # mxfp4 will dequant the model weights
            raise ValueError("DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized models.")

        if quant_method == QuantizationMethod.MXFP4:
            from transformers import Mxfp4Config

            quant_config = Mxfp4Config(dequantize=True)
            init_kwargs["quantization_config"] = quant_config
            init_kwargs["ignore_mismatched_sizes"] = True

        if quant_method == QuantizationMethod.FP8:
            if _uses_kt_non_expert_cache(model_args):
                if model_args.quantization_bit is not None:
                    raise ValueError("`quantization_bit` cannot be combined with KT weight caches.")

                logger.info_rank0("Skipping source FP8 dequantization because KT weight caches are configured.")
                return

            from transformers import FineGrainedFP8Config

            quant_config = FineGrainedFP8Config(dequantize=True)
            init_kwargs["quantization_config"] = quant_config
            init_kwargs["ignore_mismatched_sizes"] = True

        if quant_method == QuantizationMethod.GPTQ:
            check_version("gptqmodel>=2.0.0", mandatory=True)
            quantization_config.pop("disable_exllama", None)  # remove deprecated args
            quantization_config["use_exllama"] = False  # disable exllama

        if quant_method == QuantizationMethod.AWQ:
            check_version("autoawq", mandatory=True)

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove quantization_bit (and quantization_method) from the model_args — the FP8 checkpoint plus KT cache already provides quantized weights.
2. If you want classic on-the-fly quantization, disable the KT weight cache options so _uses_kt_non_expert_cache is false.
3. Validate the YAML against a known-good KT example config for your LlamaFactory version.

Example fix

# before (yaml)
quantization_bit: 8   # combined with KT weight caches on FP8 ckpt

# after (yaml)
# section removed entirely; keep KT cache options only

if fp8_checkpoint_with_kt_cache(model_args):
    assert model_args.quantization_bit is None, (
        "remove quantization_bit when KT weight caches are configured"
    )

When it happens

Trigger: Loading an FP8-quantized checkpoint with KT non-expert weight caches configured (model_args flags like enable_thu_kt + cache options detected by _uses_kt_non_expert_cache) while model_args.quantization_bit is not None.

Common situations: KT-accelerated runs that copy quantization_bit: 4/8 from a template YAML; mixing bitsandbytes on-the-fly quantization with the KT FP8 cache path.

Related errors

• FP8 training is not compatible with quantization. Please dis
• Quantization dataset is necessary for exporting.
• These `kt_config` values are derived from LLaMA-Factory argu
• Put KTransformers settings in the LLaMA-Factory training YAM
• LLaMA-Factory YAML and Accelerate config cannot define diffe

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/0c3ae9c7723ef077. Report an issue: GitHub.