{"record":{"id":"bb64819eeeafdef8","repo":"FoundationAgents/MetaGPT","slug":"unsupported-metric-eval-metric","errorCode":null,"errorMessage":"Unsupported metric: {eval_metric}","messagePattern":"Unsupported metric: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"metagpt/ext/sela/runner/autosklearn.py","lineNumber":63,"sourceCode":"                metric=self.create_autosklearn_scorer(eval_metric),\n                memory_limit=8192,\n                tmp_folder=\"AutosklearnModels/as-{}-{}\".format(\n                    self.state[\"task\"], datetime.now().strftime(\"%y%m%d_%H%M\")\n                ),\n                n_jobs=-1,\n            )\n        elif eval_metric in [\"f1\", \"f1 weighted\"]:\n            automl = autosklearn.classification.AutoSklearnClassifier(\n                time_left_for_this_task=self.time_limit,\n                metric=self.create_autosklearn_scorer(eval_metric),\n                memory_limit=8192,\n                tmp_folder=\"AutosklearnModels/as-{}-{}\".format(\n                    self.state[\"task\"], datetime.now().strftime(\"%y%m%d_%H%M\")\n                ),\n                n_jobs=-1,\n            )\n        else:\n            raise ValueError(f\"Unsupported metric: {eval_metric}\")\n        automl.fit(X_train, y_train)\n\n        dev_preds = automl.predict(dev_data)\n        test_preds = automl.predict(test_data)\n\n        return {\"test_preds\": test_preds, \"dev_preds\": dev_preds}\n\n\nclass AutoSklearnRunner(CustomRunner):\n    result_path: str = \"results/autosklearn\"\n\n    def __init__(self, args, **kwargs):\n        super().__init__(args, **kwargs)\n        self.framework = ASRunner(self.state)\n\n    async def run_experiment(self):\n        result = self.framework.run()\n        user_requirement = self.state[\"requirement\"]","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/FoundationAgents/MetaGPT/blob/11cdf466d042aece04fc6cfd13b28e1a70341b1f/metagpt/ext/sela/runner/autosklearn.py#L45-L81","documentation":"Raised by the AutoSklearn custom runner when the dataset config's metric is neither 'rmse' (AutoSklearnRegressor) nor 'f1'/'f1 weighted' (AutoSklearnClassifier). The autosklearn backend only implements those branches, so e.g. 'roc_auc' or 'log rmse' datasets cannot be run in this mode even though evaluate_score supports them.","triggerScenarios":"Running with --exp_mode autosklearn on a task whose dataset_config['metric'] is 'roc_auc', 'f1 binary' (note: not plain 'f1'), or 'log rmse'.","commonSituations":"Using the autosklearn baseline on a task whose metric was auto-derived (e.g. 'f1 binary' for a 2-class dataset, which this runner does not accept).","solutions":["Run autosklearn mode only on tasks with metric 'rmse', 'f1', or 'f1 weighted'","Override the metric in the dataset config to 'f1' or 'f1 weighted' for binary tasks","Use a different exp_mode (e.g. mcts/custom) for metrics the autosklearn runner does not support"],"exampleFix":"# before\nmetric: f1 binary\n\n# after\nmetric: f1 weighted","handlingStrategy":"validation","validationCode":"metric = state[\"dataset_config\"][\"metric\"]\nassert metric in {\"rmse\", \"f1\", \"f1 weighted\"}, f\"autosklearn runner cannot handle {metric}\"","typeGuard":"def autosklearn_supports(metric: str) -> bool:\n    return metric in {\"rmse\", \"f1\", \"f1 weighted\"}","tryCatchPattern":null,"preventionTips":["Map derived metrics (e.g. 'f1 binary') to 'f1' before running the autosklearn baseline","Prefer mcts/custom modes for metrics outside the autosklearn branch set"],"tags":["sela","autosklearn","metric","config"],"backgroundTag":null,"analyzedSha":"11cdf466d042aece04fc6cfd13b28e1a70341b1f","analyzedAt":"2026-08-14T23:20:02.994Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}