FoundationAgents/MetaGPT · error · ValueError

Invalid split {split} for dataset {dataset_name_or_path}

Error message

Invalid split {split} for dataset {dataset_name_or_path}

What it means

ValueError from load_hf_dataset in swe_agent_utils: the requested split name (default 'test') is not present in the loaded Hugging Face dataset (which may be a DatasetDict of train/validation/test, or a cached on-disk variant with different splits). The function caches datasets under cache_dir and reloads with load_from_disk, so a stale cache can also carry different splits than the remote dataset.

Source

Thrown at metagpt/tools/swe_agent_commands/swe_agent_utils.py:27

    recording = False
    for line in command_output.split("\n"):
        if line.startswith("diff --git"):
            recording = True
        if recording:
            patch_lines.append(line)
    return "\n".join(patch_lines)


def load_hf_dataset(dataset_name_or_path: str, cache_dir, split: str = "test", existing_ids: list = []):
    data_dir = cache_dir / dataset_name_or_path
    if Path(data_dir).exists():
        dataset = load_from_disk(data_dir)
    else:
        dataset = load_dataset(dataset_name_or_path)
        dataset.save_to_disk(data_dir)
    print(dataset)
    if split not in dataset:
        raise ValueError(f"Invalid split {split} for dataset {dataset_name_or_path}")
    dataset = dataset[split]
    np.array(list(map(len, dataset["instance_id"])))

    if existing_ids:
        dataset = dataset.filter(
            lambda x: x["instance_id"] not in existing_ids,
            desc="Filtering out existing ids",
            load_from_cache_file=False,
        )

    return dataset

View on GitHub (pinned to 11cdf466d0)

Solutions

1. Print/inspect the available splits (the function already print(dataset)) and use one of them, typically 'test'
2. If the cache is stale, delete cache_dir/<dataset_name> so it re-downloads fresh
3. Pass the correct split explicitly instead of relying on the 'test' default

Example fix

# before
ds = load_hf_dataset("princeton-nlp/SWE-bench_Lite", cache_dir=cache, split="validation")
# after
from datasets import load_dataset
print(load_dataset("princeton-nlp/SWE-bench_Lite").keys())  # e.g. dict_keys(['test'])
ds = load_hf_dataset("princeton-nlp/SWE-bench_Lite", cache_dir=cache, split="test")

from datasets import load_from_disk, load_dataset
from pathlib import Path
cache = Path(cache_dir) / dataset_name_or_path
ds = load_from_disk(cache) if cache.exists() else load_dataset(dataset_name_or_path)
if split not in ds:
    raise SystemExit(f"splits available: {list(ds.keys())}")

def has_split(dataset, split: str) -> bool:
    return split in dataset  # works for DatasetDict

try:
    ds = load_hf_dataset(name, cache_dir, split=split)
except ValueError:
    from datasets import load_dataset
    split = next(iter(load_dataset(name).keys()))  # pick first available split
    ds = load_hf_dataset(name, cache_dir, split=split)

When it happens

Trigger: load_hf_dataset('princeton-nlp/SWE-bench_Lite', cache_dir, split='validation') when the dataset only has 'test' and 'dev'; a cached copy on disk saved with different/renamed splits.

Common situations: Dataset revisions renaming splits (e.g. 'validation' -> 'dev'); code written against SWE-bench full reused for Lite/Verified variants; stale on-disk cache in cache_dir predating a split rename.

AI-assisted analysis of FoundationAgents/MetaGPT@11cdf466d0 (2026-08-14). Data as JSON: /api/errors/68090c0b67d4f21e. Report an issue: GitHub.