{"record":{"id":"28714e09cfaf2dd9","repo":"hiyouga/LlamaFactory","slug":"compute-accuracy-is-not-supported-in-ktransforme","errorCode":null,"errorMessage":"`compute_accuracy` is not supported in KTransformers SFT yet.","messagePattern":"`compute_accuracy` is not supported in KTransformers SFT yet\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/train/sft/workflow.py","lineNumber":80,"sourceCode":"    data_collator = SFTDataCollatorWith4DAttentionMask(\n        template=template,\n        model=model if not training_args.predict_with_generate else None,\n        pad_to_multiple_of=8 if training_args.do_train else None,  # for shift short attention\n        label_pad_token_id=IGNORE_INDEX if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id,\n        block_diag_attn=model_args.block_diag_attn,\n        neat_packing=data_args.neat_packing,\n        attn_implementation=getattr(model.config, \"_attn_implementation\", None),\n        compute_dtype=model_args.compute_dtype,\n        **tokenizer_module,\n    )\n\n    # Metric utils\n    metric_module = {}\n    if model_args.use_kt:\n        if training_args.predict_with_generate:\n            raise NotImplementedError(\"`predict_with_generate` is not supported in KTransformers SFT yet.\")\n        elif finetuning_args.compute_accuracy:\n            raise NotImplementedError(\"`compute_accuracy` is not supported in KTransformers SFT yet.\")\n\n    if training_args.predict_with_generate:\n        metric_module[\"compute_metrics\"] = ComputeSimilarity(tokenizer=tokenizer)\n    elif finetuning_args.compute_accuracy:\n        metric_module[\"compute_metrics\"] = ComputeAccuracy()\n        metric_module[\"preprocess_logits_for_metrics\"] = eval_logit_processor\n\n    # Keyword arguments for `model.generate`\n    gen_kwargs = generating_args.to_dict(obey_generation_config=True)\n\n    # Compatible with Transformers v4 and Transformers v5\n    if is_transformers_version_greater_than(\"4.58.0\"):\n        extra_ids = getattr(tokenizer, \"additional_special_tokens_ids\", None)\n        if not isinstance(extra_ids, list):\n            extra_special_tokens = getattr(tokenizer, \"_extra_special_tokens\", [])\n            string_tokens = [str(t) for t in extra_special_tokens]\n            extra_ids = tokenizer.convert_tokens_to_ids(string_tokens)\n        all_eos_ids = [tokenizer.eos_token_id] + [i for i in extra_ids if i != -1]","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/train/sft/workflow.py#L62-L98","documentation":"Same KTransformers limitation as predict_with_generate: with `use_kt: true`, token-level accuracy cannot be computed because KTransformers does not surface standard logits to the metric pipeline (src/llamafactory/train/sft/workflow.py:80). The guard raises NotImplementedError before the trainer starts.","triggerScenarios":"A training YAML combining `use_kt: true` with `finetuning_args.compute_accuracy: true`; running `llamafactory-cli train` on such a config.","commonSituations":"Users enabling compute_accuracy (common for pretrain-style eval) in a KTransformers config template; copying flags from a non-KT config.","solutions":["Set `compute_accuracy: false` (or remove it) in the YAML when use_kt is true.","Run a separate post-training evaluation with the standard HF engine if token accuracy is required.","Fall back to the normal training path (use_kt: false) when metrics are mandatory."],"exampleFix":"# before (yaml)\nuse_kt: true\ncompute_accuracy: true\n\n# after\nuse_kt: true\ncompute_accuracy: false","handlingStrategy":"validation","validationCode":"def validate_kt_config(model_args, finetuning_args, training_args):\n    if model_args.use_kt:\n        assert not training_args.predict_with_generate\n        assert not finetuning_args.compute_accuracy\n    return True","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat use_kt as training-only; plan post-training evaluation separately.","Document which metrics are unavailable per backend in your config repo."],"tags":["ktransformers","sft","evaluation","accuracy"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}