hiyouga/LlamaFactory · warning · NotImplementedError

Evaluation will be deprecated in the future.

Error message

Evaluation will be deprecated in the future.

What it means

Raised by the launcher for the 'eval' subcommand as a deliberate NotImplementedError: the standalone evaluation flow is being deprecated in LlamaFactory and the command intentionally does nothing. It is a planned-removal signal, not a runtime failure.

Source

Thrown at src/llamafactory/launcher.py:147

                .split(),
                env=env,
                check=True,
            )

        sys.exit(process.returncode)

    elif command == "api":
        from .api.app import run_api

        run_api()

    elif command == "chat":
        from .chat.chat_model import run_chat

        run_chat()

    elif command == "eval":
        raise NotImplementedError("Evaluation will be deprecated in the future.")

    elif command == "export":
        from .train.tuner import export_model

        export_model()

    elif command == "train":
        from .train.tuner import run_exp

        run_exp()

    elif command == "webchat":
        from .webui.interface import run_web_demo

        run_web_demo()

    elif command == "webui":
        from .webui.interface import run_web_ui

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove the eval step from your workflow; use the API/chat path or an external benchmark harness (e.g. lm-evaluation-harness) for evaluation.
2. Pin/consult a newer LlamaFactory version's docs for the supported evaluation alternative.

Example fix

# before
llamafactory-cli eval eval_config.yaml

# after
# use an external evaluator, e.g.
lm_eval --model hf --model_args pretrained=...,dtype=bfloat16 --tasks mmlu

if command == "eval":
    raise SystemExit("'eval' is deprecated in LlamaFactory; use the API/chat path or lm-evaluation-harness")

When it happens

Trigger: Executing llamafactory-cli eval with any arguments.

Common situations: Following outdated tutorials or docs that reference the eval subcommand; scripts written against older LlamaFactory versions.

Related errors

• Invalid role
• SGLang only supports n=1.
• SGLang engine does not support `get_scores`.
• vLLM engine does not support `get_scores`.
• Please launch distributed training with `llamafactory-cli` o

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/32d3f1726559ca4c. Report an issue: GitHub.