{"record":{"id":"f52311d8e4f84555","repo":"FoundationAgents/MetaGPT","slug":"unsupported-dataset-dataset","errorCode":null,"errorMessage":"Unsupported dataset: {dataset}","messagePattern":"Unsupported dataset: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"metagpt/ext/aflow/scripts/evaluator.py","lineNumber":40,"sourceCode":"    Complete the evaluation for different datasets here\n    \"\"\"\n\n    def __init__(self, eval_path: str):\n        self.eval_path = eval_path\n        self.dataset_configs: Dict[DatasetType, BaseBenchmark] = {\n            \"GSM8K\": GSM8KBenchmark,\n            \"MATH\": MATHBenchmark,\n            \"HumanEval\": HumanEvalBenchmark,\n            \"HotpotQA\": HotpotQABenchmark,\n            \"MBPP\": MBPPBenchmark,\n            \"DROP\": DROPBenchmark,\n        }\n\n    async def graph_evaluate(\n        self, dataset: DatasetType, graph, params: dict, path: str, is_test: bool = False\n    ) -> Tuple[float, float, float]:\n        if dataset not in self.dataset_configs:\n            raise ValueError(f\"Unsupported dataset: {dataset}\")\n\n        data_path = self._get_data_path(dataset, is_test)\n        benchmark_class = self.dataset_configs[dataset]\n        benchmark = benchmark_class(name=dataset, file_path=data_path, log_path=path)\n\n        # Use params to configure the graph and benchmark\n        configured_graph = await self._configure_graph(dataset, graph, params)\n        if is_test:\n            va_list = None  # For test data, generally use None to test all\n        else:\n            va_list = None  # Use None to test all Validation data, or set va_list (e.g., [1, 2, 3]) to use partial data\n        return await benchmark.run_evaluation(configured_graph, va_list)\n\n    async def _configure_graph(self, dataset, graph, params: dict):\n        # Here you can configure the graph based on params\n        # For example: set LLM configuration, dataset configuration, etc.\n        dataset_config = params.get(\"dataset\", {})\n        llm_config = params.get(\"llm_config\", {})","sourceCodeStart":22,"sourceCodeEnd":58,"githubUrl":"https://github.com/FoundationAgents/MetaGPT/blob/11cdf466d042aece04fc6cfd13b28e1a70341b1f/metagpt/ext/aflow/scripts/evaluator.py#L22-L58","documentation":"AFlow's Evaluator.graph_evaluate dispatches on a fixed dataset_configs mapping ({GSM8K, MATH, HumanEval, HotpotQA, MBPP, DROP}) and raises ValueError for any dataset name not present. The string must match a key exactly (case-sensitive); an unknown or misspelled dataset never reaches benchmark construction.","triggerScenarios":"Calling graph_evaluate(dataset=DatasetType.<X>, ...) or the aflow CLI with --dataset set to something like \"gsm8k\" (lowercase), \"TriviaQA\", or any benchmark outside the six supported ones.","commonSituations":"Case mismatch (gsm8k vs GSM8K); passing a dataset supported elsewhere in metagpt but not wired into aflow's evaluator config; typo in scripts/prompts; using a DatasetType enum member from a newer version than this evaluator supports.","solutions":["Use one of the exact supported keys: \"GSM8K\", \"MATH\", \"HumanEval\", \"HotpotQA\", \"MBPP\", \"DROP\" — match casing exactly.","Check what your version supports by printing evaluator.dataset_configs / DatasetType members before running.","To add a benchmark, register a Benchmark class in dataset_configs and provide the corresponding data path; do not bypass the check with a raw string.","Upgrade metagpt if you expect a dataset that newer aflow versions added."],"exampleFix":"# before\nawait evaluator.graph_evaluate(\"gsm8k\", graph, params, path)  # ValueError: Unsupported dataset: gsm8k\n\n# after\nawait evaluator.graph_evaluate(\"GSM8K\", graph, params, path)","handlingStrategy":"validation","validationCode":"SUPPORTED_DATASETS = {\"GSM8K\", \"MATH\", \"HumanEval\", \"HotpotQA\", \"MBPP\", \"DROP\"}\n\nif dataset not in SUPPORTED_DATASETS:\n    raise ValueError(f\"dataset must be one of {sorted(SUPPORTED_DATASETS)}; got {dataset!r}\")\nawait evaluator.graph_evaluate(dataset, graph, params, path)","typeGuard":"def is_supported_aflow_dataset(dataset: str) -> bool:\n    \"\"\"True when aflow's evaluator has a benchmark registered under this exact key.\"\"\"\n    return dataset in {\"GSM8K\", \"MATH\", \"HumanEval\", \"HotpotQA\", \"MBPP\", \"DROP\"}","tryCatchPattern":"try:\n    scores = await evaluator.graph_evaluate(dataset, graph, params, path)\nexcept ValueError as e:\n    if \"Unsupported dataset\" in str(e):\n        raise ValueError(f\"{dataset!r} not supported; choose from GSM8K/MATH/HumanEval/HotpotQA/MBPP/DROP\") from e\n    raise","preventionTips":["Validate the dataset string against the supported set before starting long aflow runs (fail in seconds, not hours).","Use the exact casing of the config keys; aflow keys are case-sensitive.","On upgrades, re-read evaluator.dataset_configs to pick up newly added benchmarks."],"tags":["python","aflow","dataset","validation","benchmark"],"backgroundTag":null,"analyzedSha":"11cdf466d042aece04fc6cfd13b28e1a70341b1f","analyzedAt":"2026-08-14T23:20:02.994Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}