hiyouga/LlamaFactory · warning · ValueError

Please make sure eval_dataset be provided or val_size >1e-6

Error message

Please make sure eval_dataset be provided or val_size >1e-6

What it means

Raised in parser.py:483 when do_eval, do_predict, or predict_with_generate is requested but no evaluation source exists: data_args.eval_dataset is None AND data_args.val_size < 1e-6. Evaluation needs either a named eval dataset or a split carved from the training set.

Source

Thrown at src/llamafactory/hparams/parser.py:483

        ):
            raise ValueError("PPO only accepts wandb, tensorboard, or trackio logger.")

    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:
        raise ValueError("Please launch distributed training with `llamafactory-cli` or `torchrun`.")

    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:
        raise ValueError("Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.")

    if training_args.max_steps == -1 and data_args.streaming:
        raise ValueError("Please specify `max_steps` in streaming mode.")

    if training_args.do_train and data_args.dataset is None:
        raise ValueError("Please specify dataset for training.")

    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (
        data_args.eval_dataset is None and data_args.val_size < 1e-6
    ):
        raise ValueError("Please make sure eval_dataset be provided or val_size >1e-6")

    if training_args.predict_with_generate:
        if is_deepspeed_zero3_enabled():
            raise ValueError("`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.")

        if finetuning_args.compute_accuracy:
            raise ValueError("Cannot use `predict_with_generate` and `compute_accuracy` together.")

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Add `val_size: 0.1` (or any fraction/count > 1e-6) to split part of the training set for eval
2. Or specify `eval_dataset: <name>` pointing to a dataset_info.json entry
3. If you did not intend evaluation, remove/disable do_eval, do_predict and predict_with_generate

Example fix

# before (YAML)
do_eval: true
# no eval_dataset, no val_size

# after
do_eval: true
val_size: 0.1

needs_eval = any(config.get(k) for k in ("do_eval", "do_predict", "predict_with_generate"))
if needs_eval and not config.get("eval_dataset") and float(config.get("val_size", 0)) < 1e-6:
    raise SystemExit("Set eval_dataset or val_size > 0 for evaluation")

When it happens

Trigger: Config with `val_size: 0` (or unset) plus `do_eval: true` / `predict_with_generate: true` and no `eval_dataset:` key; running an eval-only job whose dataset field for eval was misspelled.

Common situations: Enabling evaluation in an SFT config copied from a LoRA example that never defined val_size; running prediction benchmarks with predict_with_generate but forgetting to set aside validation data.

Related errors

• Please specify `max_steps` in streaming mode.
• Please specify dataset for training.
• Cannot use `predict_with_generate` and `compute_accuracy` to
• The model does not have a submodule named '{submodule_name}'
• Unsupported model type: {getattr(config, 'model_type')}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/ec07b6d10445ab47. Report an issue: GitHub.