microsoft/qlib · error · ValueError

unknown metric `%s`

Error message

unknown metric `%s`

What it means

Raised by HIST.metric_fn when self.metric is not "ic". Note the second branch contains a genuine bug: it tests `self.metric == ("", "loss")` — a string compared to a tuple, which is always False — so the intended ""/"loss" aliases never match and also raise this error. Effectively only metric="ic" works; any other value (including "" or "loss") hits the raise.

Source

Thrown at qlib/contrib/model/pytorch_hist.py:176

            return self.mse(pred[mask], label[mask])

        raise ValueError("unknown loss `%s`" % self.loss)

    def metric_fn(self, pred, label):
        mask = torch.isfinite(label)

        if self.metric == "ic":
            x = pred[mask]
            y = label[mask]

            vx = x - torch.mean(x)
            vy = y - torch.mean(y)
            return torch.sum(vx * vy) / (torch.sqrt(torch.sum(vx**2)) * torch.sqrt(torch.sum(vy**2)))

        if self.metric == ("", "loss"):
            return -self.loss_fn(pred[mask], label[mask])

        raise ValueError("unknown metric `%s`" % self.metric)

    def get_daily_inter(self, df, shuffle=False):
        # organize the train data into daily batches
        daily_count = df.groupby(level=0, group_keys=False).size().values
        daily_index = np.roll(np.cumsum(daily_count), 1)
        daily_index[0] = 0
        if shuffle:
            # shuffle data
            daily_shuffle = list(zip(daily_index, daily_count))
            np.random.shuffle(daily_shuffle)
            daily_index, daily_count = zip(*daily_shuffle)
        return daily_index, daily_count

    def train_epoch(self, x_train, y_train, stock_index):
        stock2concept_matrix = np.load(self.stock2concept)
        x_train_values = x_train.values
        y_train_values = np.squeeze(y_train.values)
        stock_index = stock_index.values

View on GitHub (pinned to 79633dd950)

Solutions

  1. Set metric="ic" — the only value that works in the shipped code.
  2. If you need loss-based validation, subclass HIST and fix metric_fn: change the comparison to `if self.metric in ("", "loss"):` and return -self.loss_fn(...).
  3. When filing/patching upstream, note the tuple-equality bug at pytorch_hist.py:172.

Example fix

# before (upstream, buggy)
if self.metric == ("", "loss"):  # always False
    return -self.loss_fn(pred[mask], label[mask])

# after (subclass patch)
if self.metric in ("", "loss"):
    return -self.loss_fn(pred[mask], label[mask])
Defensive patterns

Strategy: validation

Validate before calling

assert params.get("metric") == "ic", "HIST effectively supports only metric='ic' (the ''/'loss' branch is broken upstream: string == tuple comparison)"

Type guard

def is_supported_metric(metric: str) -> bool:
    return metric == "ic"

Try / catch

try:
    model.fit(dataset)
except ValueError as e:
    if "unknown metric" in str(e):
        params["metric"] = "ic"
        model = HIST(**params)
        model.fit(dataset)
    else:
        raise

Prevention

When it happens

Trigger: HIST(metric="") or HIST(metric="loss") (raises due to the tuple-comparison bug), or any value other than "ic". Fires on the first validation batch of fit().

Common situations: Setting metric="loss" expecting loss-based early stopping and being surprised it raises; configs copied from other models; users unaware "ic" is the only working metric in this class.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/8700e25a24aa4eb5. Report an issue: GitHub.