{"record":{"id":"62193bf80b45b961","repo":"hiyouga/LlamaFactory","slug":"ktransformers-does-not-support-lora-reward-model","errorCode":null,"errorMessage":"KTransformers does not support lora reward model.","messagePattern":"KTransformers does not support lora reward model\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":458,"sourceCode":"        if training_args.deepspeed is not None:\n            raise ValueError(\"Megatron Bridge is incompatible with DeepSpeed.\")\n        if mb_args is None:\n            raise ValueError(\"Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.\")\n        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)\n        finetuning_args.megatron_bridge_args = mb_args\n\n    if finetuning_args.stage in [\"rm\", \"ppo\"] and training_args.load_best_model_at_end:\n        raise ValueError(\"RM and PPO stages do not support `load_best_model_at_end`.\")\n\n    if finetuning_args.stage == \"ppo\":\n        if not training_args.do_train:\n            raise ValueError(\"PPO training does not support evaluation, use the SFT stage to evaluate models.\")\n\n        if model_args.shift_attn:\n            raise ValueError(\"PPO training is incompatible with S^2-Attn.\")\n\n        if finetuning_args.reward_model_type == \"lora\" and model_args.use_kt:\n            raise ValueError(\"KTransformers does not support lora reward model.\")\n\n        if finetuning_args.reward_model_type == \"lora\" and model_args.use_unsloth:\n            raise ValueError(\"Unsloth does not support lora reward model.\")\n\n        if training_args.report_to and any(\n            logger not in (\"wandb\", \"tensorboard\", \"trackio\", \"none\") for logger in training_args.report_to\n        ):\n            raise ValueError(\"PPO only accepts wandb, tensorboard, or trackio logger.\")\n\n    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:\n        raise ValueError(\"Please launch distributed training with `llamafactory-cli` or `torchrun`.\")\n\n    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:\n        raise ValueError(\"Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.\")\n\n    if training_args.max_steps == -1 and data_args.streaming:\n        raise ValueError(\"Please specify `max_steps` in streaming mode.\")\n","sourceCodeStart":440,"sourceCodeEnd":476,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L440-L476","documentation":"For PPO with a LoRA-based reward model (finetuning_args.reward_model_type == 'lora'), the KTransformers backend (model_args.use_kt) cannot serve the LoRA reward head, so the combination is rejected. KTransformers offloads layers to CPU/GPU in a way that does not support attaching the lora RM adapter.","triggerScenarios":"stage: ppo with reward_model: <path>, reward_model_type: lora and use_kt: true in model_args; raised during argument checking before model load.","commonSituations":"Running PPO on large models (e.g. 70B) with KTransformers offloading while pointing at a LoRA reward model checkpoint saved from RM training; copying a kt chat config into a ppo config.","solutions":["Use a merged full reward model: export the LoRA RM (llamafactory-cli export) and set reward_model to the merged path with reward_model_type: full, keeping use_kt.","Or disable KTransformers (use_kt: false) to use the LoRA reward model directly — requires enough GPU memory.","Retrain the reward model with finetuning_type: full for a native full RM checkpoint."],"exampleFix":"# before\nstage: ppo\nreward_model: saves/rm_lora\nreward_model_type: lora\nuse_kt: true\n\n# after (merge RM LoRA first via export, then)\nstage: ppo\nreward_model: saves/rm_merged_full\nreward_model_type: full\nuse_kt: true","handlingStrategy":"validation","validationCode":"if cfg.get(\"stage\") == \"ppo\" and cfg.get(\"reward_model_type\") == \"lora\" and cfg.get(\"use_kt\"):\n    raise SystemExit(\"KTransformers cannot serve a LoRA reward model; merge the RM LoRA via export and use reward_model_type: full\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Standardize on merged/full reward models for PPO so backend choice never blocks the run.","Automate the RM export step right after RM training so a full checkpoint always exists."],"tags":["ppo","ktransformers","reward-model","lora","config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}