FoundationAgents/MetaGPT · error · ValueError

Unsupported dataset: {dataset}

Error message

Unsupported dataset: {dataset}

What it means

AFlow's Evaluator.graph_evaluate dispatches on a fixed dataset_configs mapping ({GSM8K, MATH, HumanEval, HotpotQA, MBPP, DROP}) and raises ValueError for any dataset name not present. The string must match a key exactly (case-sensitive); an unknown or misspelled dataset never reaches benchmark construction.

Source

Thrown at metagpt/ext/aflow/scripts/evaluator.py:40

    Complete the evaluation for different datasets here
    """

    def __init__(self, eval_path: str):
        self.eval_path = eval_path
        self.dataset_configs: Dict[DatasetType, BaseBenchmark] = {
            "GSM8K": GSM8KBenchmark,
            "MATH": MATHBenchmark,
            "HumanEval": HumanEvalBenchmark,
            "HotpotQA": HotpotQABenchmark,
            "MBPP": MBPPBenchmark,
            "DROP": DROPBenchmark,
        }

    async def graph_evaluate(
        self, dataset: DatasetType, graph, params: dict, path: str, is_test: bool = False
    ) -> Tuple[float, float, float]:
        if dataset not in self.dataset_configs:
            raise ValueError(f"Unsupported dataset: {dataset}")

        data_path = self._get_data_path(dataset, is_test)
        benchmark_class = self.dataset_configs[dataset]
        benchmark = benchmark_class(name=dataset, file_path=data_path, log_path=path)

        # Use params to configure the graph and benchmark
        configured_graph = await self._configure_graph(dataset, graph, params)
        if is_test:
            va_list = None  # For test data, generally use None to test all
        else:
            va_list = None  # Use None to test all Validation data, or set va_list (e.g., [1, 2, 3]) to use partial data
        return await benchmark.run_evaluation(configured_graph, va_list)

    async def _configure_graph(self, dataset, graph, params: dict):
        # Here you can configure the graph based on params
        # For example: set LLM configuration, dataset configuration, etc.
        dataset_config = params.get("dataset", {})
        llm_config = params.get("llm_config", {})

View on GitHub (pinned to 11cdf466d0)

Solutions

1. Use one of the exact supported keys: "GSM8K", "MATH", "HumanEval", "HotpotQA", "MBPP", "DROP" — match casing exactly.
2. Check what your version supports by printing evaluator.dataset_configs / DatasetType members before running.
3. To add a benchmark, register a Benchmark class in dataset_configs and provide the corresponding data path; do not bypass the check with a raw string.
4. Upgrade metagpt if you expect a dataset that newer aflow versions added.

Example fix

# before
await evaluator.graph_evaluate("gsm8k", graph, params, path)  # ValueError: Unsupported dataset: gsm8k

# after
await evaluator.graph_evaluate("GSM8K", graph, params, path)

SUPPORTED_DATASETS = {"GSM8K", "MATH", "HumanEval", "HotpotQA", "MBPP", "DROP"}

if dataset not in SUPPORTED_DATASETS:
    raise ValueError(f"dataset must be one of {sorted(SUPPORTED_DATASETS)}; got {dataset!r}")
await evaluator.graph_evaluate(dataset, graph, params, path)

def is_supported_aflow_dataset(dataset: str) -> bool:
    """True when aflow's evaluator has a benchmark registered under this exact key."""
    return dataset in {"GSM8K", "MATH", "HumanEval", "HotpotQA", "MBPP", "DROP"}

try:
    scores = await evaluator.graph_evaluate(dataset, graph, params, path)
except ValueError as e:
    if "Unsupported dataset" in str(e):
        raise ValueError(f"{dataset!r} not supported; choose from GSM8K/MATH/HumanEval/HotpotQA/MBPP/DROP") from e
    raise

When it happens

Trigger: Calling graph_evaluate(dataset=DatasetType.<X>, ...) or the aflow CLI with --dataset set to something like "gsm8k" (lowercase), "TriviaQA", or any benchmark outside the six supported ones.

Common situations: Case mismatch (gsm8k vs GSM8K); passing a dataset supported elsewhere in metagpt but not wired into aflow's evaluator config; typo in scripts/prompts; using a DatasetType enum member from a newer version than this evaluator supports.

Related errors

• please check MilvusConnection, uri must be set.
• Workflow file not found: {graph_path}
• Item list is empty.
• Score list is empty.
• Dataset {dataset_name} not found in config file. Available d

AI-assisted analysis of FoundationAgents/MetaGPT@11cdf466d0 (2026-08-14). Data as JSON: /api/errors/f52311d8e4f84555. Report an issue: GitHub.