hiyouga/LlamaFactory · error · RuntimeError

Cannot find valid samples, check `data/README.md` for the da

Error message

Cannot find valid samples, check `data/README.md` for the data format.

What it means

Same StopIteration path as the pt variant, but for sft/rm/ppo/kto stages: after .filter() over the tokenized dataset, not a single valid sample remains, so printing the data example fails. 'Valid' means the sample passed all stage-specific filters (non-empty labels, matching pair lengths, acceptable token counts), and the message directs you to the documented data format.

Source

Thrown at src/llamafactory/data/loader.py:271

        )

    dataset = dataset.map(
        dataset_processor.preprocess_dataset,
        batched=True,
        batch_size=data_args.preprocessing_batch_size,
        remove_columns=column_names,
        **kwargs,
    )

    if training_args.should_log:
        try:
            print("eval example:" if is_eval else "training example:")
            dataset_processor.print_data_example(next(iter(dataset)))
        except StopIteration:
            if stage == "pt":
                raise RuntimeError("Cannot find sufficient samples, consider increasing dataset size.")
            else:
                raise RuntimeError("Cannot find valid samples, check `data/README.md` for the data format.")

    return dataset


def get_dataset(
    template: "Template",
    model_args: "ModelArguments",
    data_args: "DataArguments",
    training_args: "Seq2SeqTrainingArguments",
    stage: Literal["pt", "sft", "rm", "ppo", "kto"],
    tokenizer: "PreTrainedTokenizer",
    processor: Optional["ProcessorMixin"] = None,
) -> "DatasetModule":
    r"""Get the train dataset and optionally gets the evaluation dataset."""
    # Load tokenized dataset if path exists
    if data_args.tokenized_path is not None:
        if has_tokenized_data(data_args.tokenized_path):
            logger.warning_rank0("Loading dataset from disk will ignore other data arguments.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Read data/README.md and align your dataset fields with the expected format for the stage, including the columns mapping in dataset_info.json.
2. Verify each sample yields trainable tokens: for sft the output/response field must be non-empty; for rm both chosen and rejected must be present.
3. Raise cutoff_len / remove max_samples so samples survive filtering.
4. Test with a tiny known-good dataset (e.g. the built-in alpaca demo) to isolate whether the template or the data is at fault.

Example fix

# before (dataset_info.json, field names mismatch)
"columns": {"prompt": "instruction", "query": "input", "response": "answer"}

# after (match actual JSONL keys)
"columns": {"prompt": "instruction", "query": "input", "response": "output"}

def sft_rows_trainable(rows: list[dict], resp_key: str) -> list[int]:
    return [i for i, r in enumerate(rows) if not (r.get(resp_key) or "").strip()]  # indices that will be dropped

def row_has_response(row: dict, response_key: str) -> bool:
    return bool((row.get(response_key) or "").strip())

When it happens

Trigger: Alpaca-format rows where output is empty and the formatter yields no trainable tokens; conversations whose messages do not match the columns mapping; rm pairs where chosen/rejected tokenize to equal length 0; max_samples/cutoff_len filtering everything; wrong template producing empty encoded results.

Common situations: Columns mapping in dataset_info.json not matching the actual JSONL field names; datasets where all answers live in a field mapped to None; template/tags misconfiguration stripping all content; every sample exceeding cutoff_len.

Related errors

• Cannot find sufficient samples, consider increasing dataset
• No valid messages or chosen_messages/rejected_messages found
• The model does not have a submodule named '{submodule_name}'
• Unsupported model type: {getattr(config, 'model_type')}.
• Stage does not supported: {stage}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/3a128d8000ffc7ea. Report an issue: GitHub.