{"record":{"id":"b00e3b700959a02e","repo":"hiyouga/LlamaFactory","slug":"predict-with-generate-is-not-supported-in-ktrans","errorCode":null,"errorMessage":"`predict_with_generate` is not supported in KTransformers SFT yet.","messagePattern":"`predict_with_generate` is not supported in KTransformers SFT yet\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/train/sft/workflow.py","lineNumber":78,"sourceCode":"        setattr(model, \"_hf_peft_config_loaded\", True)  # hack here: make model compatible with prediction\n\n    data_collator = SFTDataCollatorWith4DAttentionMask(\n        template=template,\n        model=model if not training_args.predict_with_generate else None,\n        pad_to_multiple_of=8 if training_args.do_train else None,  # for shift short attention\n        label_pad_token_id=IGNORE_INDEX if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id,\n        block_diag_attn=model_args.block_diag_attn,\n        neat_packing=data_args.neat_packing,\n        attn_implementation=getattr(model.config, \"_attn_implementation\", None),\n        compute_dtype=model_args.compute_dtype,\n        **tokenizer_module,\n    )\n\n    # Metric utils\n    metric_module = {}\n    if model_args.use_kt:\n        if training_args.predict_with_generate:\n            raise NotImplementedError(\"`predict_with_generate` is not supported in KTransformers SFT yet.\")\n        elif finetuning_args.compute_accuracy:\n            raise NotImplementedError(\"`compute_accuracy` is not supported in KTransformers SFT yet.\")\n\n    if training_args.predict_with_generate:\n        metric_module[\"compute_metrics\"] = ComputeSimilarity(tokenizer=tokenizer)\n    elif finetuning_args.compute_accuracy:\n        metric_module[\"compute_metrics\"] = ComputeAccuracy()\n        metric_module[\"preprocess_logits_for_metrics\"] = eval_logit_processor\n\n    # Keyword arguments for `model.generate`\n    gen_kwargs = generating_args.to_dict(obey_generation_config=True)\n\n    # Compatible with Transformers v4 and Transformers v5\n    if is_transformers_version_greater_than(\"4.58.0\"):\n        extra_ids = getattr(tokenizer, \"additional_special_tokens_ids\", None)\n        if not isinstance(extra_ids, list):\n            extra_special_tokens = getattr(tokenizer, \"_extra_special_tokens\", [])\n            string_tokens = [str(t) for t in extra_special_tokens]","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/train/sft/workflow.py#L60-L96","documentation":"When `model_args.use_kt` is true, training runs through the KTransformers engine (src/llamafactory/train/sft/workflow.py:78). KTransformers does not expose HuggingFace-style logits or generation during evaluation, so `predict_with_generate` is explicitly unsupported and raises NotImplementedError.","triggerScenarios":"A training YAML with both `use_kt: true` and `predict_with_generate: true` (the latter often copied from a standard SFT eval config), invoking `llamafactory-cli train` on it.","commonSituations":"Users migrating an existing SFT eval recipe to KTransformers for large local models (e.g. DeepSeek-R1 style) and leaving evaluation-generation flags enabled; CI configs shared across model backends.","solutions":["Remove `predict_with_generate: true` from the YAML (or set it to false) when use_kt is enabled.","Evaluate generation quality separately after training using llamafactory-cli chat or the KTransformers inference path.","If you need predict_with_generate during eval, drop use_kt and run the standard HF/vLLM training path."],"exampleFix":"# before (yaml)\nuse_kt: true\npredict_with_generate: true\n\n# after\nuse_kt: true\npredict_with_generate: false","handlingStrategy":"validation","validationCode":"def validate_kt_config(model_args, training_args):\n    if model_args.use_kt and training_args.predict_with_generate:\n        raise SystemExit(\"disable predict_with_generate when use_kt is true\")\n    return True","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep a separate YAML template for KTransformers runs without eval-generation flags.","Add a CI lint that rejects use_kt + predict_with_generate configs."],"tags":["ktransformers","sft","evaluation","generation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}