microsoft/qlib · error · ValueError

unknown metric `%s`

Error message

unknown metric `%s`

What it means

IGMTFModel.metric_fn supports metric='ic' (Pearson correlation of prediction vs label over finite labels) and metric ('', 'loss') which is intended to mean 'use negative loss as the score'. Note the second check compares with == against a tuple, so the strings '' and 'loss' never actually match — a known qlib bug — meaning anything other than 'ic' always raises.

Source

Thrown at qlib/contrib/model/pytorch_igmtf.py:169

            return self.mse(pred[mask], label[mask])

        raise ValueError("unknown loss `%s`" % self.loss)

    def metric_fn(self, pred, label):
        mask = torch.isfinite(label)

        if self.metric == "ic":
            x = pred[mask]
            y = label[mask]

            vx = x - torch.mean(x)
            vy = y - torch.mean(y)
            return torch.sum(vx * vy) / (torch.sqrt(torch.sum(vx**2)) * torch.sqrt(torch.sum(vy**2)))

        if self.metric == ("", "loss"):
            return -self.loss_fn(pred[mask], label[mask])

        raise ValueError("unknown metric `%s`" % self.metric)

    def get_daily_inter(self, df, shuffle=False):
        # organize the train data into daily batches
        daily_count = df.groupby(level=0, group_keys=False).size().values
        daily_index = np.roll(np.cumsum(daily_count), 1)
        daily_index[0] = 0
        if shuffle:
            # shuffle data
            daily_shuffle = list(zip(daily_index, daily_count))
            np.random.shuffle(daily_shuffle)
            daily_index, daily_count = zip(*daily_shuffle)
        return daily_index, daily_count

    def get_train_hidden(self, x_train):
        x_train_values = x_train.values
        daily_index, daily_count = self.get_daily_inter(x_train, shuffle=True)
        self.igmtf_model.eval()
        train_hidden = []

View on GitHub (pinned to 79633dd950)

Solutions

  1. Set metric='ic' for IGMTFModel
  2. If you want loss-as-metric, patch metric_fn locally to use `if self.metric in ('', 'loss')` (this is the upstream bug) or subclass and override metric_fn

Example fix

# before
IGMTFModel(metric="loss")  # raises: code compares self.metric == ("", "loss")

# after
IGMTFModel(metric="ic")
# or patch upstream bug:
#   if self.metric in ("", "loss"): return -self.loss_fn(pred[mask], label[mask])
Defensive patterns

Strategy: validation

Validate before calling

assert metric == "ic", "IGMTFModel metric_fn only works with 'ic' (the ('', 'loss') branch is buggy upstream)"

Type guard

def is_supported_igmtf_metric(name):
    return name == "ic"

Try / catch

try:
    model.fit(dataset)
except ValueError as e:
    if "unknown metric" in str(e):
        model.metric = "ic"
        model.fit(dataset)
    else:
        raise

Prevention

When it happens

Trigger: Calling fit() with self.metric set to anything except 'ic'. Due to the tuple-comparison bug, even metric='' or metric='loss' (the values other qlib models accept) raise this error in IGMTFModel.

Common situations: Using hyperparameter defaults copied from pytorch_gru/ALSTM workflows where metric='' is common; expecting '' to mean 'loss' as in other qlib models and hitting the buggy tuple comparison.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/071930cff50ba8ed. Report an issue: GitHub.