microsoft/qlib · error · ValueError

unknown metric `%s`

Error message

unknown metric `%s`

What it means

LOCALTransformerModel.metric_fn() computes the validation score used for early stopping and best-epoch tracking. Only self.metric in ('', 'loss') is supported: it returns the negated masked loss (higher is better). Any other metric string raises ValueError("unknown metric `%s`") during validation in fit(), not at construction time.

Source

Thrown at qlib/contrib/model/pytorch_localformer_ts.py:103

    def mse(self, pred, label):
        loss = (pred.float() - label.float()) ** 2
        return torch.mean(loss)

    def loss_fn(self, pred, label):
        mask = ~torch.isnan(label)

        if self.loss == "mse":
            return self.mse(pred[mask], label[mask])

        raise ValueError("unknown loss `%s`" % self.loss)

    def metric_fn(self, pred, label):
        mask = torch.isfinite(label)

        if self.metric in ("", "loss"):
            return -self.loss_fn(pred[mask], label[mask])

        raise ValueError("unknown metric `%s`" % self.metric)

    def train_epoch(self, data_loader):
        self.model.train()

        for data in data_loader:
            feature = data[:, :, 0:-1].to(self.device)
            label = data[:, -1, -1].to(self.device)

            pred = self.model(feature.float())  # .float()
            loss = self.loss_fn(pred, label)

            self.train_optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_value_(self.model.parameters(), 3.0)
            self.train_optimizer.step()

    def test_epoch(self, data_loader):
        self.model.eval()

View on GitHub (pinned to 79633dd950)

Solutions

  1. Set metric='' or metric='loss' (both mean: use negative MSE as the score) in the model kwargs.
  2. To use a custom metric, subclass and override metric_fn(self, pred, label); return a scalar where higher is better, and mask non-finite labels with torch.isfinite(label).
  3. Note the sign convention: scores are maximized (best_score starts at -np.inf), so return negative values for losses.

Example fix

# before
model = LOCALTransformerModel(..., metric="ic")
model.fit(dataset)  # ValueError: unknown metric `ic` at validation

# after
model = LOCALTransformerModel(..., metric="loss")
model.fit(dataset)
Defensive patterns

Strategy: validation

Validate before calling

metric = "loss"
assert metric in ("", "loss"), "LOCALTransformerModel supports only metric='' or 'loss'"
model = LOCALTransformerModel(..., metric=metric)

Type guard

def is_supported_metric(name: str) -> bool:
    return isinstance(name, str) and name in ("", "loss")

Try / catch

try:
    model.fit(dataset)
except ValueError as e:
    if "unknown metric" in str(e):
        raise ValueError("Set metric='' or 'loss' (negative MSE as score)") from e
    raise

Prevention

When it happens

Trigger: Calling model.fit(...) with metric set to anything besides '' or 'loss' — e.g. 'ic', 'mse', 'accuracy'. The error is raised on the first validation pass after the first training epoch.

Common situations: Porting 'ic'-style metric configs from other qlib models (e.g. pytorch_nn or GBDT workflows where IC is common); assuming standard sklearn metric names work; leaving a metric from a copied YAML that this model family does not implement.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/57305c0c84855e1a. Report an issue: GitHub.