{"record":{"id":"e7efcf6c50ca0c4f","repo":"hiyouga/LlamaFactory","slug":"kt-cpu-activation-recompute-requires-gpu-gradie","errorCode":null,"errorMessage":"`kt_cpu_activation: recompute` requires GPU gradient checkpointing. Set `disable_gradient_checkpointing: false` or use `kt_cpu_activation: retain`.","messagePattern":"`kt_cpu_activation: recompute` requires GPU gradient checkpointing\\. Set `disable_gradient_checkpointing: false` or use `kt_cpu_activation: retain`\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/model_args.py","lineNumber":540,"sourceCode":"            \"kt_skip_expert_loading\",\n            \"kt_train_mode\",\n            \"kt_use_lora_experts\",\n            \"kt_weight_path\",\n        }\n    )\n\n    def __post_init__(self) -> None:\n        if self.kt_cpu_activation not in {None, \"retain\", \"recompute\"}:\n            raise ValueError(\"`kt_cpu_activation` must be `retain` or `recompute`.\")\n        if not self.use_kt and self.kt_cpu_activation is not None:\n            raise ValueError(\"`kt_cpu_activation` is only valid when `use_kt: true`.\")\n\n    def get_kt_activation_policy(self) -> dict[str, str]:\n        r\"\"\"Resolve LF's GPU checkpoint switch and KT's CPU activation setting.\"\"\"\n        gpu_activation = \"retain\" if self.disable_gradient_checkpointing else \"recompute\"\n        cpu_activation = self.kt_cpu_activation or gpu_activation\n        if cpu_activation == \"recompute\" and gpu_activation == \"retain\":\n            raise ValueError(\n                \"`kt_cpu_activation: recompute` requires GPU gradient checkpointing. \"\n                \"Set `disable_gradient_checkpointing: false` or use `kt_cpu_activation: retain`.\"\n            )\n\n        return {\"cpu\": cpu_activation, \"gpu\": gpu_activation}\n\n    @staticmethod\n    def _get_accelerator_kt_config(training_args: Any) -> Any:\n        accelerator_config = getattr(training_args, \"accelerator_config\", None)\n        if isinstance(accelerator_config, dict):\n            return accelerator_config.get(\"kt_config\")\n        return getattr(accelerator_config, \"kt_config\", None)\n\n    def _normalize_advanced_kt_config(self, raw_config: Any) -> dict[str, Any]:\n        if raw_config is None:\n            return {}\n        if not isinstance(raw_config, dict):\n            raise TypeError(\"LLaMA-Factory `kt_config` must be a flat mapping.\")","sourceCodeStart":522,"sourceCodeEnd":558,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/model_args.py#L522-L558","documentation":"Raised by get_kt_activation_policy (model_args.py:540) when the resolved CPU policy is 'recompute' but the GPU policy is 'retain' — i.e. kt_cpu_activation: recompute while gradient checkpointing is disabled (disable_gradient_checkpointing: true). CPU recompute on backward only works if the GPU side also checkpoints, otherwise activations the CPU needs are gone; the error text tells you the two consistent resolutions.","triggerScenarios":"kt_cpu_activation: recompute together with disable_gradient_checkpointing: true; leaving kt_cpu_activation unset while disabling gradient checkpointing is fine (policy follows GPU), but explicitly requesting recompute with checkpointing off is not.","commonSituations":"Users disabling gradient checkpointing to speed up step time, then asking CPU to recompute anyway; memory-tuning CPU RAM without considering the GPU-side coupling; merging configs where disable_gradient_checkpointing comes from a base file.","solutions":["Set disable_gradient_checkpointing: false so GPU gradient checkpointing is on","Or switch to kt_cpu_activation: retain (or remove it) if you want to keep checkpointing disabled","Keep the pair in one YAML block so they are edited together"],"exampleFix":"# before\ndisable_gradient_checkpointing: true\nkt_cpu_activation: recompute\n\n# after\ndisable_gradient_checkpointing: false\nkt_cpu_activation: recompute","handlingStrategy":"validation","validationCode":"policy = cfg.get('kt_cpu_activation')\ngc_disabled = cfg.get('disable_gradient_checkpointing', False)\nif policy == 'recompute' and gc_disabled:\n    cfg['disable_gradient_checkpointing'] = False  # or drop the policy","typeGuard":"def kt_policy_consistent(cfg: dict) -> bool:\n    return not (cfg.get('kt_cpu_activation') == 'recompute' and cfg.get('disable_gradient_checkpointing', False))","tryCatchPattern":null,"preventionTips":["Remember the coupling: CPU recompute requires GPU gradient checkpointing","Keep disable_gradient_checkpointing and kt_cpu_activation adjacent in the YAML"],"tags":["ktransformers","gradient-checkpointing","memory"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}