{"record":{"id":"f1b5ef29e6422d18","repo":"hiyouga/LlamaFactory","slug":"the-length-of-eval-dataset-and-interleave-probs-sh","errorCode":null,"errorMessage":"The length of eval dataset and interleave probs should be identical.","messagePattern":"The length of eval dataset and interleave probs should be identical\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/data_args.py","lineNumber":174,"sourceCode":"        if self.media_dir is None:\n            self.media_dir = self.dataset_dir\n\n        if self.dataset is None and self.val_size > 1e-6:\n            raise ValueError(\"Cannot specify `val_size` if `dataset` is None.\")\n\n        if self.eval_dataset is not None and self.val_size > 1e-6:\n            raise ValueError(\"Cannot specify `val_size` if `eval_dataset` is not None.\")\n\n        if self.interleave_probs is not None:\n            if self.mix_strategy == \"concat\":\n                raise ValueError(\"`interleave_probs` is only valid for interleaved mixing.\")\n\n            self.interleave_probs = list(map(float, split_arg(self.interleave_probs)))\n            if self.dataset is not None and len(self.dataset) != len(self.interleave_probs):\n                raise ValueError(\"The length of dataset and interleave probs should be identical.\")\n\n            if self.eval_dataset is not None and len(self.eval_dataset) != len(self.interleave_probs):\n                raise ValueError(\"The length of eval dataset and interleave probs should be identical.\")\n\n        if self.streaming and self.val_size > 1e-6 and self.val_size < 1:\n            raise ValueError(\"Streaming mode should have an integer val size.\")\n\n        if self.streaming and self.max_samples is not None:\n            raise ValueError(\"`max_samples` is incompatible with `streaming`.\")\n\n        if self.mask_history and self.train_on_prompt:\n            raise ValueError(\"`mask_history` is incompatible with `train_on_prompt`.\")\n\n        if self.neat_packing:\n            self.packing = True\n\n        if self.packing:\n            self.cutoff_len -= 1  # avoid pad_to_multiple_of, needs improve\n\n    def to_dict(self) -> dict[str, Any]:\n        return asdict(self)","sourceCodeStart":156,"sourceCodeEnd":192,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/data_args.py#L156-L192","documentation":"Symmetric to the training-dataset check: when interleave_probs is set and eval_dataset is provided, the number of evaluation datasets must equal the number of probabilities. DataArguments.post_init raises ValueError when len(eval_dataset) != len(interleave_probs).","triggerScenarios":"Two eval datasets (eval_dataset: e1,e2) with three probabilities matching a three-dataset training list; adding eval datasets without extending/adjusting the shared interleave_probs list (note: one list serves both).","commonSituations":"Reusing a single interleave_probs for different train/eval dataset counts; trimming eval datasets while keeping training probabilities unchanged.","solutions":["Align counts: either give each eval dataset its own probability by matching lengths, or restructure so train and eval use the same number of interleaved datasets.","If eval datasets shouldn't be interleaved, reconsider using interleave_probs at all (it applies to both lists).","Validate lengths programmatically before training (see validationCode)."],"exampleFix":"# before\ndataset: d1,d2,d3\neval_dataset: e1,e2\ninterleave_probs: 0.5,0.3,0.2\n\n# after\ndataset: d1,d2\neval_dataset: e1,e2\ninterleave_probs: 0.5,0.5","handlingStrategy":"validation","validationCode":"n_eval = len(data_args.eval_dataset) if data_args.eval_dataset else 0\nif data_args.interleave_probs is not None and data_args.eval_dataset is not None:\n    assert n_eval == len(data_args.interleave_probs), \\\n        f\"{n_eval} eval datasets vs {len(data_args.interleave_probs)} probs\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Remember one interleave_probs list must match BOTH dataset and eval_dataset counts.","Automate a config sanity check that compares all three lengths before submitting training jobs."],"tags":["config","data-args","mixing"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}