microsoft/qlib · error · ValueError

the history of distribution data is not long enough.

Error message

the history of distribution data is not long enough.

What it means

MetaDatasetDS builds meta tasks from a rolling IC history: it needs at least self.step * self.hist_step_n available rows of IC data to look back hist_step_n windows of step days. If ic_df_avail has fewer rows than that, the distribution history is too short and it raises ValueError.

Source

Thrown at qlib/contrib/meta/data_selection/dataset.py:384

        def mask_overlap(s):
            """
            mask overlap information
            data after self.name[end] with self.trunc_days that contains future info are also considered as overlap info

            Approximately the diagnal + horizon length of data are masked.
            """
            start, end = s.name
            end = get_date_by_shift(trading_date=end, shift=self.trunc_days - 1, future=True)
            return s.mask((s.index >= start) & (s.index <= end))

        ic_df_avail = ic_df_avail.apply(mask_overlap)  # apply to each col

        # 2) filter the info with too long periods
        total_len = self.step * self.hist_step_n
        if ic_df_avail.shape[0] >= total_len:
            return ic_df_avail.iloc[-total_len:]
        else:
            raise ValueError("the history of distribution data is not long enough.")

    def _prepare_seg(self, segment: Text) -> List[MetaTask]:
        if isinstance(self.segments, float):
            train_task_n = int(len(self.meta_task_l) * self.segments)
            if segment == "train":
                train_tasks = self.meta_task_l[:train_task_n]
                get_module_logger("MetaDatasetDS").info(f"The first train meta task: {train_tasks[0]}")
                return train_tasks
            elif segment == "test":
                test_tasks = self.meta_task_l[train_task_n:]
                get_module_logger("MetaDatasetDS").info(f"The first test meta task: {test_tasks[0]}")
                return test_tasks
            else:
                raise NotImplementedError(f"This type of input is not supported")
        elif isinstance(self.segments, str):
            train_tasks = []
            test_tasks = []
            for t in self.meta_task_l:

View on GitHub (pinned to 79633dd950)

Solutions

  1. Move train_start later so at least step * hist_step_n days of IC history exist before it.
  2. Reduce hist_step_n or step so the required history fits what you have.
  3. Regenerate the underlying IC records over a longer rolling backtest window.

Example fix

// before
mds = MetaDatasetDS(task_tpl=tpl, train_start="2010-01-01", step=20, hist_step_n=10)  # < 200 prior days

// after
mds = MetaDatasetDS(task_tpl=tpl, train_start="2011-06-01", step=20, hist_step_n=10)  # >= 200 prior days
Defensive patterns

Strategy: validation

Validate before calling

required = step * hist_step_n
avail = len(ic_df)  # rows before train_start
if avail < required:
    raise ValueError(f"need {required} prior IC days, have {avail}; lower hist_step_n/step or move train_start")

Try / catch

try:
    mds = MetaDatasetDS(task_tpl=tpl, ...)
except ValueError as e:
    if "not long enough" in str(e):
        mds = MetaDatasetDS(task_tpl=tpl, hist_step_n=hist_step_n // 2, ...)
    else:
        raise

Prevention

When it happens

Trigger: Constructing MetaDatasetDS with a training start such that the IC series before it is shorter than step * hist_step_n; or large step/hist_step_n hyperparameters relative to the backtest period.

Common situations: Setting train_start too early in the data; increasing hist_step_n for a longer lookback without extending the preceding evaluation period; using a short alpha158/alpha360 run whose IC history covers few dates.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/9897cf401e9d626c. Report an issue: GitHub.