microsoft/qlib · error · ValueError

unknown metric `%s`

Error message

unknown metric `%s`

What it means

The TS LSTM metric_fn() only supports self.metric in ('', 'loss'), returning -loss_fn (with weight=None, i.e. unweighted) as the higher-is-better validation score. Any other metric string raises ValueError("unknown metric `%s`") during the first validation pass in fit(), after the first epoch of training has already run.

Source

Thrown at qlib/contrib/model/pytorch_lstm_ts.py:158

    def loss_fn(self, pred, label, weight):
        mask = ~torch.isnan(label)

        if weight is None:
            weight = torch.ones_like(label)

        if self.loss == "mse":
            return self.mse(pred[mask], label[mask], weight[mask])

        raise ValueError("unknown loss `%s`" % self.loss)

    def metric_fn(self, pred, label):
        mask = torch.isfinite(label)

        if self.metric in ("", "loss"):
            return -self.loss_fn(pred[mask], label[mask], weight=None)

        raise ValueError("unknown metric `%s`" % self.metric)

    def train_epoch(self, data_loader):
        self.LSTM_model.train()

        for data, weight in data_loader:
            feature = data[:, :, 0:-1].to(self.device)
            label = data[:, -1, -1].to(self.device)

            pred = self.LSTM_model(feature.float())
            loss = self.loss_fn(pred, label, weight.to(self.device))

            self.train_optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_value_(self.LSTM_model.parameters(), 3.0)
            self.train_optimizer.step()

    def test_epoch(self, data_loader):
        self.LSTM_model.eval()

View on GitHub (pinned to 79633dd950)

Solutions

  1. Set metric='' or metric='loss'.
  2. Override metric_fn(self, pred, label) in a subclass for custom scores; note the existing signature passes weight=None through loss_fn, so custom metrics should mask non-finite labels themselves.
  3. Keep the higher-is-better convention so early stopping and best-epoch logic behave correctly.

Example fix

# before
model = LSTMModel(..., metric="rank_ic")
model.fit(dataset)  # ValueError: unknown metric `rank_ic`

# after
model = LSTMModel(..., metric="loss")
model.fit(dataset)
Defensive patterns

Strategy: validation

Validate before calling

assert metric in ("", "loss"), "TS LSTM supports only metric='' or 'loss'"
model = LSTMModel(..., metric=metric)

Type guard

def is_supported_metric(name: str) -> bool:
    return name in ("", "loss")

Try / catch

try:
    model.fit(dataset)
except ValueError as e:
    if "unknown metric" in str(e):
        raise ValueError("metric must be '' or 'loss'") from e
    raise

Prevention

When it happens

Trigger: model.fit(...) with metric set to any string other than '' or 'loss' — 'ic', 'rank_ic', 'mse', etc.

Common situations: Benchmark configs written for models that accept IC-style metrics; users assuming the metric kwarg mirrors qlib's signal analysis metrics.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/dbf6b688545de650. Report an issue: GitHub.