microsoft/qlib · error · ValueError

unknown metric `%s`

Error message

unknown metric `%s`

What it means

Thrown by TransformerModel.metric_fn, which scores each epoch for early stopping. Supported metric values are '' and 'loss' (negative MSE on the finite-label mask); anything else raises the first time validation runs.

Source

Thrown at qlib/contrib/model/pytorch_transformer.py:102

    def mse(self, pred, label):
        loss = (pred.float() - label.float()) ** 2
        return torch.mean(loss)

    def loss_fn(self, pred, label):
        mask = ~torch.isnan(label)

        if self.loss == "mse":
            return self.mse(pred[mask], label[mask])

        raise ValueError("unknown loss `%s`" % self.loss)

    def metric_fn(self, pred, label):
        mask = torch.isfinite(label)

        if self.metric in ("", "loss"):
            return -self.loss_fn(pred[mask], label[mask])

        raise ValueError("unknown metric `%s`" % self.metric)

    def train_epoch(self, x_train, y_train):
        x_train_values = x_train.values
        y_train_values = np.squeeze(y_train.values)

        self.model.train()

        indices = np.arange(len(x_train_values))
        np.random.shuffle(indices)

        for i in range(len(indices))[:: self.batch_size]:
            if len(indices) - i < self.batch_size:
                break

            feature = torch.from_numpy(x_train_values[indices[i : i + self.batch_size]]).float().to(self.device)
            label = torch.from_numpy(y_train_values[indices[i : i + self.batch_size]]).float().to(self.device)

            pred = self.model(feature)

View on GitHub (pinned to 79633dd950)

Solutions

  1. Use metric='loss' or '' in the TransformerModel constructor.
  2. Keep per-model hyperparameter dicts instead of one shared dict so unsupported metric names don't leak between models.
  3. Subclass and override metric_fn (e.g. IC on the finite mask) if a different early-stopping criterion is needed.

Example fix

# before
shared_kwargs = {"loss": "mse", "metric": "ic"}
model = TransformerModel(**shared_kwargs)

# after
model = TransformerModel(..., loss="mse", metric="loss")
Defensive patterns

Strategy: validation

Validate before calling

assert model_kwargs.get("metric", "") in ("", "loss"), "TransformerModel metric must be '' or 'loss'"

Try / catch

try:
    model.fit(dataset, evals_result)
except ValueError as e:
    if "unknown metric" in str(e):
        model_kwargs["metric"] = "loss"
        model = TransformerModel(**model_kwargs)
        model.fit(dataset, evals_result)
    else:
        raise

Prevention

When it happens

Trigger: TransformerModel(..., metric='ic'|'auc'|anything) followed by fit(); the validation pass calls metric_fn and hits the raise.

Common situations: Sharing one hyperparameter dict across several qlib models where 'ic' is valid elsewhere; typos like 'Loss'.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/7455374fe11220e3. Report an issue: GitHub.