{"record":{"id":"0c3ae9c7723ef077","repo":"hiyouga/LlamaFactory","slug":"quantization-bit-cannot-be-combined-with-kt-weig","errorCode":null,"errorMessage":"`quantization_bit` cannot be combined with KT weight caches.","messagePattern":"`quantization_bit` cannot be combined with KT weight caches\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/quantization.py","lineNumber":117,"sourceCode":"        quant_method = quantization_config.get(\"quant_method\", \"\")\n\n        if quant_method not in (QuantizationMethod.MXFP4, QuantizationMethod.FP8) and (\n            is_deepspeed_zero3_enabled() or is_fsdp_enabled()\n        ):\n            # mxfp4 will dequant the model weights\n            raise ValueError(\"DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized models.\")\n\n        if quant_method == QuantizationMethod.MXFP4:\n            from transformers import Mxfp4Config\n\n            quant_config = Mxfp4Config(dequantize=True)\n            init_kwargs[\"quantization_config\"] = quant_config\n            init_kwargs[\"ignore_mismatched_sizes\"] = True\n\n        if quant_method == QuantizationMethod.FP8:\n            if _uses_kt_non_expert_cache(model_args):\n                if model_args.quantization_bit is not None:\n                    raise ValueError(\"`quantization_bit` cannot be combined with KT weight caches.\")\n\n                logger.info_rank0(\"Skipping source FP8 dequantization because KT weight caches are configured.\")\n                return\n\n            from transformers import FineGrainedFP8Config\n\n            quant_config = FineGrainedFP8Config(dequantize=True)\n            init_kwargs[\"quantization_config\"] = quant_config\n            init_kwargs[\"ignore_mismatched_sizes\"] = True\n\n        if quant_method == QuantizationMethod.GPTQ:\n            check_version(\"gptqmodel>=2.0.0\", mandatory=True)\n            quantization_config.pop(\"disable_exllama\", None)  # remove deprecated args\n            quantization_config[\"use_exllama\"] = False  # disable exllama\n\n        if quant_method == QuantizationMethod.AWQ:\n            check_version(\"autoawq\", mandatory=True)\n","sourceCodeStart":99,"sourceCodeEnd":135,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/quantization.py#L99-L135","documentation":"For FP8 PTQ checkpoints, LlamaFactory can skip dequantization and consume KT-managed weight caches (a proprietary fast-loading path). That path stores pre-quantized weights, so additionally requesting on-the-fly quantization_bit would be contradictory; the code raises ValueError when quantization_bit is set together with KT weight caches.","triggerScenarios":"Loading an FP8-quantized checkpoint with KT non-expert weight caches configured (model_args flags like enable_thu_kt + cache options detected by _uses_kt_non_expert_cache) while model_args.quantization_bit is not None.","commonSituations":"KT-accelerated runs that copy quantization_bit: 4/8 from a template YAML; mixing bitsandbytes on-the-fly quantization with the KT FP8 cache path.","solutions":["Remove quantization_bit (and quantization_method) from the model_args — the FP8 checkpoint plus KT cache already provides quantized weights.","If you want classic on-the-fly quantization, disable the KT weight cache options so _uses_kt_non_expert_cache is false.","Validate the YAML against a known-good KT example config for your LlamaFactory version."],"exampleFix":"# before (yaml)\nquantization_bit: 8   # combined with KT weight caches on FP8 ckpt\n\n# after (yaml)\n# section removed entirely; keep KT cache options only","handlingStrategy":"validation","validationCode":"if fp8_checkpoint_with_kt_cache(model_args):\n    assert model_args.quantization_bit is None, (\n        \"remove quantization_bit when KT weight caches are configured\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep KT run configs minimal; strip all quantization_* keys when KT caches are on.","Maintain separate template YAMLs for KT and bnb paths to avoid key bleed-over."],"tags":["fp8","kt-kernel","quantization","config-conflict"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}