hiyouga/LlamaFactory · error · NotImplementedError

`predict_with_generate` is not supported in KTransformers SF

Error message

`predict_with_generate` is not supported in KTransformers SFT yet.

What it means

When `model_args.use_kt` is true, training runs through the KTransformers engine (src/llamafactory/train/sft/workflow.py:78). KTransformers does not expose HuggingFace-style logits or generation during evaluation, so `predict_with_generate` is explicitly unsupported and raises NotImplementedError.

Source

Thrown at src/llamafactory/train/sft/workflow.py:78

        setattr(model, "_hf_peft_config_loaded", True)  # hack here: make model compatible with prediction

    data_collator = SFTDataCollatorWith4DAttentionMask(
        template=template,
        model=model if not training_args.predict_with_generate else None,
        pad_to_multiple_of=8 if training_args.do_train else None,  # for shift short attention
        label_pad_token_id=IGNORE_INDEX if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id,
        block_diag_attn=model_args.block_diag_attn,
        neat_packing=data_args.neat_packing,
        attn_implementation=getattr(model.config, "_attn_implementation", None),
        compute_dtype=model_args.compute_dtype,
        **tokenizer_module,
    )

    # Metric utils
    metric_module = {}
    if model_args.use_kt:
        if training_args.predict_with_generate:
            raise NotImplementedError("`predict_with_generate` is not supported in KTransformers SFT yet.")
        elif finetuning_args.compute_accuracy:
            raise NotImplementedError("`compute_accuracy` is not supported in KTransformers SFT yet.")

    if training_args.predict_with_generate:
        metric_module["compute_metrics"] = ComputeSimilarity(tokenizer=tokenizer)
    elif finetuning_args.compute_accuracy:
        metric_module["compute_metrics"] = ComputeAccuracy()
        metric_module["preprocess_logits_for_metrics"] = eval_logit_processor

    # Keyword arguments for `model.generate`
    gen_kwargs = generating_args.to_dict(obey_generation_config=True)

    # Compatible with Transformers v4 and Transformers v5
    if is_transformers_version_greater_than("4.58.0"):
        extra_ids = getattr(tokenizer, "additional_special_tokens_ids", None)
        if not isinstance(extra_ids, list):
            extra_special_tokens = getattr(tokenizer, "_extra_special_tokens", [])
            string_tokens = [str(t) for t in extra_special_tokens]

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove `predict_with_generate: true` from the YAML (or set it to false) when use_kt is enabled.
2. Evaluate generation quality separately after training using llamafactory-cli chat or the KTransformers inference path.
3. If you need predict_with_generate during eval, drop use_kt and run the standard HF/vLLM training path.

Example fix

# before (yaml)
use_kt: true
predict_with_generate: true

# after
use_kt: true
predict_with_generate: false

def validate_kt_config(model_args, training_args):
    if model_args.use_kt and training_args.predict_with_generate:
        raise SystemExit("disable predict_with_generate when use_kt is true")
    return True

When it happens

Trigger: A training YAML with both `use_kt: true` and `predict_with_generate: true` (the latter often copied from a standard SFT eval config), invoking `llamafactory-cli train` on it.

Common situations: Users migrating an existing SFT eval recipe to KTransformers for large local models (e.g. DeepSeek-R1 style) and leaving evaluation-generation flags enabled; CI configs shared across model backends.

Related errors

• `compute_accuracy` is not supported in KTransformers SFT yet
• `predict_with_generate` cannot be set as True except SFT.
• Please enable `predict_with_generate` to save model predicti
• Cannot process the logits.
• The length of packed example should be identical to the cuto

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/b00e3b700959a02e. Report an issue: GitHub.