microsoft/qlib · error · ValueError

unknown metric `%s`

Error message

unknown metric `%s`

What it means

GATsModel.metric_fn accepts only the empty string or 'loss' as self.metric, returning the negated training loss as the early-stopping score. Unlike ALSTM it has no 'mse' branch, so metric='mse' is also rejected. Any unmatched value raises ValueError on the first validation pass inside fit().

Source

Thrown at qlib/contrib/model/pytorch_gats.py:162

    def mse(self, pred, label):
        loss = (pred - label) ** 2
        return torch.mean(loss)

    def loss_fn(self, pred, label):
        mask = ~torch.isnan(label)

        if self.loss == "mse":
            return self.mse(pred[mask], label[mask])

        raise ValueError("unknown loss `%s`" % self.loss)

    def metric_fn(self, pred, label):
        mask = torch.isfinite(label)

        if self.metric in ("", "loss"):
            return -self.loss_fn(pred[mask], label[mask])

        raise ValueError("unknown metric `%s`" % self.metric)

    def get_daily_inter(self, df, shuffle=False):
        # organize the train data into daily batches
        daily_count = df.groupby(level=0, group_keys=False).size().values
        daily_index = np.roll(np.cumsum(daily_count), 1)
        daily_index[0] = 0
        if shuffle:
            # shuffle data
            daily_shuffle = list(zip(daily_index, daily_count))
            np.random.shuffle(daily_shuffle)
            daily_index, daily_count = zip(*daily_shuffle)
        return daily_index, daily_count

    def train_epoch(self, x_train, y_train):
        x_train_values = x_train.values
        y_train_values = np.squeeze(y_train.values)
        self.GAT_model.train()

View on GitHub (pinned to 79633dd950)

Solutions

  1. Set metric='' or 'loss' (both mean loss-based scoring) in the GATsModel config.
  2. Remove metric='mse' from configs ported from ALSTM workflows.
  3. Subclass GATsModel and add metric branches in metric_fn if you need another metric.

Example fix

# before
model = GATsModel(metric='mse')

# after
model = GATsModel(metric='loss')
Defensive patterns

Strategy: validation

Validate before calling

assert model.metric in ('', 'loss'), f"GATsModel metric must be '' or 'loss', got {model.metric!r}"

Type guard

def is_supported_gats_metric(metric: str) -> bool:
    return metric in ('', 'loss')

Try / catch

try:
    model.fit(dataset)
except ValueError as e:
    if 'unknown metric' in str(e):
        model.metric = 'loss'
        model.fit(dataset)
    else:
        raise

Prevention

When it happens

Trigger: Constructing GATsModel(metric='mse') or metric='ic' and calling fit(); the first validation epoch calls metric_fn which raises.

Common situations: Assuming the metric options are identical across qlib PyTorch models (ALSTM supports 'mse', GATs does not); copy-pasting model hyperparameter blocks between workflow configs.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/92c36ad4ed6baef4. Report an issue: GitHub.