What causes "unknown metric `%s`" in microsoft/qlib?

Use metric='' or metric='loss' in the model config.

microsoft/qlib · error · ValueError

unknown metric `%s`

Error message

unknown metric `%s`

What it means

The LSTM model's metric_fn() supplies the validation score driving early stopping and checkpoint selection. Only self.metric in ('', 'loss') is valid and yields the negated masked MSE (higher = better). Any other string raises ValueError("unknown metric `%s`") on the first validation pass inside fit().

Source

Thrown at qlib/contrib/model/pytorch_lstm.py:150

    def mse(self, pred, label):
        loss = (pred - label) ** 2
        return torch.mean(loss)

    def loss_fn(self, pred, label):
        mask = ~torch.isnan(label)

        if self.loss == "mse":
            return self.mse(pred[mask], label[mask])

        raise ValueError("unknown loss `%s`" % self.loss)

    def metric_fn(self, pred, label):
        mask = torch.isfinite(label)

        if self.metric in ("", "loss"):
            return -self.loss_fn(pred[mask], label[mask])

        raise ValueError("unknown metric `%s`" % self.metric)

    def train_epoch(self, x_train, y_train):
        x_train_values = x_train.values
        y_train_values = np.squeeze(y_train.values)

        self.lstm_model.train()

        indices = np.arange(len(x_train_values))
        np.random.shuffle(indices)

        for i in range(len(indices))[:: self.batch_size]:
            if len(indices) - i < self.batch_size:
                break

            feature = torch.from_numpy(x_train_values[indices[i : i + self.batch_size]]).float().to(self.device)
            label = torch.from_numpy(y_train_values[indices[i : i + self.batch_size]]).float().to(self.device)

            pred = self.lstm_model(feature)

View on GitHub (pinned to 79633dd950)

Solutions

Use metric='' or metric='loss' in the model config.
Override metric_fn(self, pred, label) in a subclass for custom scores; mask with torch.isfinite(label) and return higher-is-better scalars.
Remember early stopping selects the maximum score, so losses must be negated.

Example fix

# before
model = LSTMModel(..., metric="ic")
model.fit(dataset)  # ValueError: unknown metric `ic`

# after
model = LSTMModel(..., metric="loss")
model.fit(dataset)

Defensive patterns

Strategy: validation

Validate before calling

assert metric in ("", "loss"), "LSTMModel supports only metric='' or 'loss'"
model = LSTMModel(..., metric=metric)

Type guard

def is_supported_metric(name: str) -> bool:
    return name in ("", "loss")

Try / catch

try:
    model.fit(dataset)
except ValueError as e:
    if "unknown metric" in str(e):
        raise ValueError("metric must be '' or 'loss'") from e
    raise

Prevention

This model family scores validation by negative loss only; don't copy IC metrics over.
Custom metric_fn overrides must mask non-finite labels and return higher-is-better values.
Validate metric up front — the error otherwise appears only after the first training epoch.

When it happens

Trigger: model.fit(...) with metric set to 'ic', 'mse', 'mae', 'acc', etc. — anything outside ('', 'loss'). Raises after the first training epoch when validation runs.

Common situations: Copying 'metric: ic' from Alpha158 benchmark configs used with other models; assuming the metric vocabulary is shared across all qlib contrib models.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/4d01d64389e21667. Report an issue: GitHub.