hiyouga/LlamaFactory · error · RuntimeError

Cannot find sufficient samples, consider increasing dataset

Error message

Cannot find sufficient samples, consider increasing dataset size.

What it means

RuntimeError raised after tokenization when iterating the preprocessed pt (pretrain) dataset yields StopIteration on the very first sample: the dataset became empty after filtering (e.g. max_samples, cutoff_len dropping everything, or token counts exceeding limits). For the pretraining stage the message points at insufficient raw sample count.

Source

Thrown at src/llamafactory/data/loader.py:269

            load_from_cache_file=(not data_args.overwrite_cache) or (training_args.local_process_index != 0),
            desc="Running tokenizer on dataset",
        )

    dataset = dataset.map(
        dataset_processor.preprocess_dataset,
        batched=True,
        batch_size=data_args.preprocessing_batch_size,
        remove_columns=column_names,
        **kwargs,
    )

    if training_args.should_log:
        try:
            print("eval example:" if is_eval else "training example:")
            dataset_processor.print_data_example(next(iter(dataset)))
        except StopIteration:
            if stage == "pt":
                raise RuntimeError("Cannot find sufficient samples, consider increasing dataset size.")
            else:
                raise RuntimeError("Cannot find valid samples, check `data/README.md` for the data format.")

    return dataset


def get_dataset(
    template: "Template",
    model_args: "ModelArguments",
    data_args: "DataArguments",
    training_args: "Seq2SeqTrainingArguments",
    stage: Literal["pt", "sft", "rm", "ppo", "kto"],
    tokenizer: "PreTrainedTokenizer",
    processor: Optional["ProcessorMixin"] = None,
) -> "DatasetModule":
    r"""Get the train dataset and optionally gets the evaluation dataset."""
    # Load tokenized dataset if path exists
    if data_args.tokenized_path is not None:

View on GitHub (pinned to f28afaf635)

Solutions

Increase max_samples or remove it so the pt dataset retains samples after preprocessing.
Check that the dataset's text column is non-empty and matches the columns mapping in dataset_info.json.
Review cutoff_len and packing settings — make sure sequences survive the length filtering.
Inspect the tokenized dataset length (or enable logging) to confirm samples survive before the print_data_example call.

Example fix

# before
max_samples: 2
cutoff_len: 8

# after
max_samples: 1000
cutoff_len: 2048

Defensive patterns

Strategy: validation

Validate before calling

def pt_sample_count_ok(rows: list[dict], max_samples: int | None) -> bool:
    usable = [r for r in rows if r.get("text", "").strip()]
    n = len(usable) if max_samples is None else min(max_samples, len(usable))
    return n > 0

Prevention

Smoke-test with max_samples large enough to survive filtering (>=10).
Check the text column is non-empty and correctly mapped.
Keep cutoff_len at a realistic value (>=512) for pt runs.

When it happens

Trigger: stage: pt with max_samples set smaller than the number of samples filtered out, cutoff_len so short that every concatenated sequence is discarded, or a corpus whose documents all fail preprocessing (e.g. empty texts).

Common situations: Smoke-testing with max_samples: 4 while a tokenized buffer/packing step consumes them; pretraining a corpus with very short lines and a large cutoff; empty or whitespace-only documents in the text field.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/a12bae1d1e5c4554. Report an issue: GitHub.