FoundationAgents/MetaGPT · error · ValueError

Unsupported metric: {eval_metric}

Error message

Unsupported metric: {eval_metric}

What it means

Raised by the AutoSklearn custom runner when the dataset config's metric is neither 'rmse' (AutoSklearnRegressor) nor 'f1'/'f1 weighted' (AutoSklearnClassifier). The autosklearn backend only implements those branches, so e.g. 'roc_auc' or 'log rmse' datasets cannot be run in this mode even though evaluate_score supports them.

Source

Thrown at metagpt/ext/sela/runner/autosklearn.py:63

                metric=self.create_autosklearn_scorer(eval_metric),
                memory_limit=8192,
                tmp_folder="AutosklearnModels/as-{}-{}".format(
                    self.state["task"], datetime.now().strftime("%y%m%d_%H%M")
                ),
                n_jobs=-1,
            )
        elif eval_metric in ["f1", "f1 weighted"]:
            automl = autosklearn.classification.AutoSklearnClassifier(
                time_left_for_this_task=self.time_limit,
                metric=self.create_autosklearn_scorer(eval_metric),
                memory_limit=8192,
                tmp_folder="AutosklearnModels/as-{}-{}".format(
                    self.state["task"], datetime.now().strftime("%y%m%d_%H%M")
                ),
                n_jobs=-1,
            )
        else:
            raise ValueError(f"Unsupported metric: {eval_metric}")
        automl.fit(X_train, y_train)

        dev_preds = automl.predict(dev_data)
        test_preds = automl.predict(test_data)

        return {"test_preds": test_preds, "dev_preds": dev_preds}


class AutoSklearnRunner(CustomRunner):
    result_path: str = "results/autosklearn"

    def __init__(self, args, **kwargs):
        super().__init__(args, **kwargs)
        self.framework = ASRunner(self.state)

    async def run_experiment(self):
        result = self.framework.run()
        user_requirement = self.state["requirement"]

View on GitHub (pinned to 11cdf466d0)

Solutions

1. Run autosklearn mode only on tasks with metric 'rmse', 'f1', or 'f1 weighted'
2. Override the metric in the dataset config to 'f1' or 'f1 weighted' for binary tasks
3. Use a different exp_mode (e.g. mcts/custom) for metrics the autosklearn runner does not support

Example fix

# before
metric: f1 binary

# after
metric: f1 weighted

metric = state["dataset_config"]["metric"]
assert metric in {"rmse", "f1", "f1 weighted"}, f"autosklearn runner cannot handle {metric}"

def autosklearn_supports(metric: str) -> bool:
    return metric in {"rmse", "f1", "f1 weighted"}

When it happens

Trigger: Running with --exp_mode autosklearn on a task whose dataset_config['metric'] is 'roc_auc', 'f1 binary' (note: not plain 'f1'), or 'log rmse'.

Common situations: Using the autosklearn baseline on a task whose metric was auto-derived (e.g. 'f1 binary' for a 2-class dataset, which this runner does not accept).

Related errors

• Metric {metric} not supported
• Dataset {dataset_name} not found in config file. Available d
• Dataset {task_name} not found in config file. Available data
• Number of classes {num_classes} not supported
• Target column not provided

AI-assisted analysis of FoundationAgents/MetaGPT@11cdf466d0 (2026-08-14). Data as JSON: /api/errors/bb64819eeeafdef8. Report an issue: GitHub.