{"record":{"id":"c8e0fb15fe08feff","repo":"hiyouga/LlamaFactory","slug":"cannot-specify-val-size-if-eval-dataset-is-not-c8e0fb","errorCode":null,"errorMessage":"Cannot specify `val_size` if `eval_dataset` is not None.","messagePattern":"Cannot specify `val_size` if `eval_dataset` is not None\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/data_args.py","lineNumber":163,"sourceCode":"    )\n\n    def __post_init__(self):\n        def split_arg(arg):\n            if isinstance(arg, str):\n                return [item.strip() for item in arg.split(\",\")]\n            return arg\n\n        self.dataset = split_arg(self.dataset)\n        self.eval_dataset = split_arg(self.eval_dataset)\n\n        if self.media_dir is None:\n            self.media_dir = self.dataset_dir\n\n        if self.dataset is None and self.val_size > 1e-6:\n            raise ValueError(\"Cannot specify `val_size` if `dataset` is None.\")\n\n        if self.eval_dataset is not None and self.val_size > 1e-6:\n            raise ValueError(\"Cannot specify `val_size` if `eval_dataset` is not None.\")\n\n        if self.interleave_probs is not None:\n            if self.mix_strategy == \"concat\":\n                raise ValueError(\"`interleave_probs` is only valid for interleaved mixing.\")\n\n            self.interleave_probs = list(map(float, split_arg(self.interleave_probs)))\n            if self.dataset is not None and len(self.dataset) != len(self.interleave_probs):\n                raise ValueError(\"The length of dataset and interleave probs should be identical.\")\n\n            if self.eval_dataset is not None and len(self.eval_dataset) != len(self.interleave_probs):\n                raise ValueError(\"The length of eval dataset and interleave probs should be identical.\")\n\n        if self.streaming and self.val_size > 1e-6 and self.val_size < 1:\n            raise ValueError(\"Streaming mode should have an integer val size.\")\n\n        if self.streaming and self.max_samples is not None:\n            raise ValueError(\"`max_samples` is incompatible with `streaming`.\")\n","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/data_args.py#L145-L181","documentation":"val_size splits the training dataset into train/val parts, which is redundant and ambiguous when an explicit eval_dataset is already provided. DataArguments.post_init raises ValueError when eval_dataset is not None and val_size > 1e-6, forcing you to choose one validation source.","triggerScenarios":"YAML with both eval_dataset: <name> and val_size: 0.05; dataset_info.json defining evaluation data while the config still carries a val_size fraction from an earlier setup.","commonSituations":"Iterating on configs: first using val_size for a quick split, later adding a proper eval_dataset but forgetting val_size; merging config snippets from examples that each used a different validation approach.","solutions":["Delete val_size (or set 0) and keep eval_dataset for validation.","Or remove eval_dataset and keep val_size if you want a random split from training data."],"exampleFix":"# before\ndataset: alpaca_demo\neval_dataset: mmlu_demo\nval_size: 0.1\n\n# after\ndataset: alpaca_demo\neval_dataset: mmlu_demo","handlingStrategy":"validation","validationCode":"assert not (data_args.eval_dataset is not None and data_args.val_size > 1e-6), \\\n    \"choose eval_dataset OR val_size, not both\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pick exactly one validation mechanism per config: eval_dataset or val_size.","Lint YAML configs for the dataset/eval_dataset/val_size combination before launching jobs."],"tags":["config","data-args","validation-split"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}