hiyouga/LlamaFactory · error · NotImplementedError

`compute_accuracy` is not supported in KTransformers SFT yet

Error message

`compute_accuracy` is not supported in KTransformers SFT yet.

What it means

Same KTransformers limitation as predict_with_generate: with `use_kt: true`, token-level accuracy cannot be computed because KTransformers does not surface standard logits to the metric pipeline (src/llamafactory/train/sft/workflow.py:80). The guard raises NotImplementedError before the trainer starts.

Source

Thrown at src/llamafactory/train/sft/workflow.py:80

    data_collator = SFTDataCollatorWith4DAttentionMask(
        template=template,
        model=model if not training_args.predict_with_generate else None,
        pad_to_multiple_of=8 if training_args.do_train else None,  # for shift short attention
        label_pad_token_id=IGNORE_INDEX if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id,
        block_diag_attn=model_args.block_diag_attn,
        neat_packing=data_args.neat_packing,
        attn_implementation=getattr(model.config, "_attn_implementation", None),
        compute_dtype=model_args.compute_dtype,
        **tokenizer_module,
    )

    # Metric utils
    metric_module = {}
    if model_args.use_kt:
        if training_args.predict_with_generate:
            raise NotImplementedError("`predict_with_generate` is not supported in KTransformers SFT yet.")
        elif finetuning_args.compute_accuracy:
            raise NotImplementedError("`compute_accuracy` is not supported in KTransformers SFT yet.")

    if training_args.predict_with_generate:
        metric_module["compute_metrics"] = ComputeSimilarity(tokenizer=tokenizer)
    elif finetuning_args.compute_accuracy:
        metric_module["compute_metrics"] = ComputeAccuracy()
        metric_module["preprocess_logits_for_metrics"] = eval_logit_processor

    # Keyword arguments for `model.generate`
    gen_kwargs = generating_args.to_dict(obey_generation_config=True)

    # Compatible with Transformers v4 and Transformers v5
    if is_transformers_version_greater_than("4.58.0"):
        extra_ids = getattr(tokenizer, "additional_special_tokens_ids", None)
        if not isinstance(extra_ids, list):
            extra_special_tokens = getattr(tokenizer, "_extra_special_tokens", [])
            string_tokens = [str(t) for t in extra_special_tokens]
            extra_ids = tokenizer.convert_tokens_to_ids(string_tokens)
        all_eos_ids = [tokenizer.eos_token_id] + [i for i in extra_ids if i != -1]

View on GitHub (pinned to f28afaf635)

Solutions

1. Set `compute_accuracy: false` (or remove it) in the YAML when use_kt is true.
2. Run a separate post-training evaluation with the standard HF engine if token accuracy is required.
3. Fall back to the normal training path (use_kt: false) when metrics are mandatory.

Example fix

# before (yaml)
use_kt: true
compute_accuracy: true

# after
use_kt: true
compute_accuracy: false

def validate_kt_config(model_args, finetuning_args, training_args):
    if model_args.use_kt:
        assert not training_args.predict_with_generate
        assert not finetuning_args.compute_accuracy
    return True

When it happens

Trigger: A training YAML combining `use_kt: true` with `finetuning_args.compute_accuracy: true`; running `llamafactory-cli train` on such a config.

Common situations: Users enabling compute_accuracy (common for pretrain-style eval) in a KTransformers config template; copying flags from a non-KT config.

Related errors

• `predict_with_generate` is not supported in KTransformers SF
• `predict_with_generate` cannot be set as True except SFT.
• Cannot process the logits.
• The length of packed example should be identical to the cuto
• `save_dir` already exists, use another one.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/28714e09cfaf2dd9. Report an issue: GitHub.