{"record":{"id":"ec07b6d10445ab47","repo":"hiyouga/LlamaFactory","slug":"please-make-sure-eval-dataset-be-provided-or-val-s","errorCode":null,"errorMessage":"Please make sure eval_dataset be provided or val_size >1e-6","messagePattern":"Please make sure eval_dataset be provided or val_size >1e-6","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"warning","filePath":"src/llamafactory/hparams/parser.py","lineNumber":483,"sourceCode":"        ):\n            raise ValueError(\"PPO only accepts wandb, tensorboard, or trackio logger.\")\n\n    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:\n        raise ValueError(\"Please launch distributed training with `llamafactory-cli` or `torchrun`.\")\n\n    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:\n        raise ValueError(\"Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.\")\n\n    if training_args.max_steps == -1 and data_args.streaming:\n        raise ValueError(\"Please specify `max_steps` in streaming mode.\")\n\n    if training_args.do_train and data_args.dataset is None:\n        raise ValueError(\"Please specify dataset for training.\")\n\n    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (\n        data_args.eval_dataset is None and data_args.val_size < 1e-6\n    ):\n        raise ValueError(\"Please make sure eval_dataset be provided or val_size >1e-6\")\n\n    if training_args.predict_with_generate:\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.\")\n\n        if finetuning_args.compute_accuracy:\n            raise ValueError(\"Cannot use `predict_with_generate` and `compute_accuracy` together.\")\n\n    if training_args.do_train and model_args.quantization_device_map == \"auto\":\n        raise ValueError(\"Cannot use device map for quantized models in training.\")\n\n    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.\")\n\n    if finetuning_args.pure_bf16:\n        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):\n            raise ValueError(\"This device does not support `pure_bf16`.\")\n","sourceCodeStart":465,"sourceCodeEnd":501,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L465-L501","documentation":"Raised in parser.py:483 when do_eval, do_predict, or predict_with_generate is requested but no evaluation source exists: data_args.eval_dataset is None AND data_args.val_size < 1e-6. Evaluation needs either a named eval dataset or a split carved from the training set.","triggerScenarios":"Config with `val_size: 0` (or unset) plus `do_eval: true` / `predict_with_generate: true` and no `eval_dataset:` key; running an eval-only job whose dataset field for eval was misspelled.","commonSituations":"Enabling evaluation in an SFT config copied from a LoRA example that never defined val_size; running prediction benchmarks with predict_with_generate but forgetting to set aside validation data.","solutions":["Add `val_size: 0.1` (or any fraction/count > 1e-6) to split part of the training set for eval","Or specify `eval_dataset: <name>` pointing to a dataset_info.json entry","If you did not intend evaluation, remove/disable do_eval, do_predict and predict_with_generate"],"exampleFix":"# before (YAML)\ndo_eval: true\n# no eval_dataset, no val_size\n\n# after\ndo_eval: true\nval_size: 0.1","handlingStrategy":"validation","validationCode":"needs_eval = any(config.get(k) for k in (\"do_eval\", \"do_predict\", \"predict_with_generate\"))\nif needs_eval and not config.get(\"eval_dataset\") and float(config.get(\"val_size\", 0)) < 1e-6:\n    raise SystemExit(\"Set eval_dataset or val_size > 0 for evaluation\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Default val_size: 0.1 in eval-enabled templates","Pair do_eval/predict_with_generate edits with an eval data source in the same commit"],"tags":["evaluation","dataset","val-size","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}