FoundationAgents/MetaGPT · error · ValueError

Metric {metric} not supported

Error message

Metric {metric} not supported

What it means

Raised by SELA's evaluate_score when the `metric` string from the dataset config is not one of the supported names ('f1 binary', 'f1 weighted', 'roc_auc', 'rmse', 'log rmse', plus the f1/accuracy branches above). The evaluator dispatches on exact string match, so any other spelling fails.

Source

Thrown at metagpt/ext/sela/evaluation/evaluation.py:26

    if metric == "accuracy":
        return accuracy_score(gt, pred)
    elif metric == "f1":
        unique_classes = sorted(list(np.unique(gt)))
        if 1 in unique_classes and 0 in unique_classes:
            pos_label = 1
        else:
            pos_label = unique_classes[0] if len(unique_classes) == 2 else None
        return f1_score(gt, pred, pos_label=pos_label)
    elif metric == "f1 weighted":
        return f1_score(gt, pred, average="weighted")
    elif metric == "roc_auc":
        return roc_auc_score(gt, pred)
    elif metric == "rmse":
        return mean_squared_error(gt, pred, squared=False)
    elif metric == "log rmse":
        return mean_squared_error(np.log1p(gt), np.log1p(pred), squared=False)
    else:
        raise ValueError(f"Metric {metric} not supported")


def node_evaluate_score_sela(node):
    preds = node.get_and_move_predictions("test")["target"]
    gt = node.get_gt("test")["target"]
    metric = node.state["dataset_config"]["metric"]
    return evaluate_score(preds, gt, metric)


def node_evaluate_score_mlebench(node):
    # TODO
    from mlebench.grade import grade_csv
    from mlebench.registry import registry

    competition_id = node.state["task"]
    data_dir = Path(node.state["custom_dataset_dir"]).parent.parent.parent  # prepared/public/../../../
    pred_path = node.get_predictions_path("test")
    new_registry = registry.set_data_dir(data_dir)

View on GitHub (pinned to 11cdf466d0)

Solutions

1. Set metric to one of the exact supported strings: 'f1 binary', 'f1 weighted', 'roc_auc', 'rmse', 'log rmse'
2. If you use ExpDataset.get_metric() to fill the config, the value is guaranteed valid — prefer that
3. Add an elif branch in evaluate_score if you genuinely need a new metric

Example fix

# before
metric: f1 macro

# after
metric: f1 weighted

SUPPORTED = {"f1", "f1 binary", "f1 weighted", "roc_auc", "rmse", "log rmse"}
assert metric in SUPPORTED, f"unsupported metric {metric}"

def is_supported_metric(metric: str) -> bool:
    return metric in {"f1", "f1 binary", "f1 weighted", "roc_auc", "rmse", "log rmse"}

When it happens

Trigger: node.state['dataset_config']['metric'] contains an unrecognized value such as 'f1 macro', 'accuracy_score', 'F1 Weighted' (wrong case), or a custom metric name.

Common situations: Hand-edited datasets.yaml with a metric name that does not match exactly; metrics produced by get_metric of a different pipeline; case/spacing typos.

Related errors

• Unsupported metric: {eval_metric}
• Dataset {dataset_name} not found in config file. Available d
• Dataset {task_name} not found in config file. Available data
• Number of classes {num_classes} not supported
• Target column not provided

AI-assisted analysis of FoundationAgents/MetaGPT@11cdf466d0 (2026-08-14). Data as JSON: /api/errors/2f6ed95adf194c7a. Report an issue: GitHub.