{"record":{"id":"2f6ed95adf194c7a","repo":"FoundationAgents/MetaGPT","slug":"metric-metric-not-supported","errorCode":null,"errorMessage":"Metric {metric} not supported","messagePattern":"Metric (.+?) not supported","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"metagpt/ext/sela/evaluation/evaluation.py","lineNumber":26,"sourceCode":"    if metric == \"accuracy\":\n        return accuracy_score(gt, pred)\n    elif metric == \"f1\":\n        unique_classes = sorted(list(np.unique(gt)))\n        if 1 in unique_classes and 0 in unique_classes:\n            pos_label = 1\n        else:\n            pos_label = unique_classes[0] if len(unique_classes) == 2 else None\n        return f1_score(gt, pred, pos_label=pos_label)\n    elif metric == \"f1 weighted\":\n        return f1_score(gt, pred, average=\"weighted\")\n    elif metric == \"roc_auc\":\n        return roc_auc_score(gt, pred)\n    elif metric == \"rmse\":\n        return mean_squared_error(gt, pred, squared=False)\n    elif metric == \"log rmse\":\n        return mean_squared_error(np.log1p(gt), np.log1p(pred), squared=False)\n    else:\n        raise ValueError(f\"Metric {metric} not supported\")\n\n\ndef node_evaluate_score_sela(node):\n    preds = node.get_and_move_predictions(\"test\")[\"target\"]\n    gt = node.get_gt(\"test\")[\"target\"]\n    metric = node.state[\"dataset_config\"][\"metric\"]\n    return evaluate_score(preds, gt, metric)\n\n\ndef node_evaluate_score_mlebench(node):\n    # TODO\n    from mlebench.grade import grade_csv\n    from mlebench.registry import registry\n\n    competition_id = node.state[\"task\"]\n    data_dir = Path(node.state[\"custom_dataset_dir\"]).parent.parent.parent  # prepared/public/../../../\n    pred_path = node.get_predictions_path(\"test\")\n    new_registry = registry.set_data_dir(data_dir)","sourceCodeStart":8,"sourceCodeEnd":44,"githubUrl":"https://github.com/FoundationAgents/MetaGPT/blob/11cdf466d042aece04fc6cfd13b28e1a70341b1f/metagpt/ext/sela/evaluation/evaluation.py#L8-L44","documentation":"Raised by SELA's evaluate_score when the `metric` string from the dataset config is not one of the supported names ('f1 binary', 'f1 weighted', 'roc_auc', 'rmse', 'log rmse', plus the f1/accuracy branches above). The evaluator dispatches on exact string match, so any other spelling fails.","triggerScenarios":"node.state['dataset_config']['metric'] contains an unrecognized value such as 'f1 macro', 'accuracy_score', 'F1 Weighted' (wrong case), or a custom metric name.","commonSituations":"Hand-edited datasets.yaml with a metric name that does not match exactly; metrics produced by get_metric of a different pipeline; case/spacing typos.","solutions":["Set metric to one of the exact supported strings: 'f1 binary', 'f1 weighted', 'roc_auc', 'rmse', 'log rmse'","If you use ExpDataset.get_metric() to fill the config, the value is guaranteed valid — prefer that","Add an elif branch in evaluate_score if you genuinely need a new metric"],"exampleFix":"# before\nmetric: f1 macro\n\n# after\nmetric: f1 weighted","handlingStrategy":"validation","validationCode":"SUPPORTED = {\"f1\", \"f1 binary\", \"f1 weighted\", \"roc_auc\", \"rmse\", \"log rmse\"}\nassert metric in SUPPORTED, f\"unsupported metric {metric}\"","typeGuard":"def is_supported_metric(metric: str) -> bool:\n    return metric in {\"f1\", \"f1 binary\", \"f1 weighted\", \"roc_auc\", \"rmse\", \"log rmse\"}","tryCatchPattern":null,"preventionTips":["Let ExpDataset.get_metric() derive the metric instead of hand-writing it","Use exact lowercase spellings with single spaces"],"tags":["sela","metric","evaluation","config"],"backgroundTag":null,"analyzedSha":"11cdf466d042aece04fc6cfd13b28e1a70341b1f","analyzedAt":"2026-08-14T23:20:02.994Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}