{"record":{"id":"9897cf401e9d626c","repo":"microsoft/qlib","slug":"the-history-of-distribution-data-is-not-long-enoug","errorCode":null,"errorMessage":"the history of distribution data is not long enough.","messagePattern":"the history of distribution data is not long enough\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"qlib/contrib/meta/data_selection/dataset.py","lineNumber":384,"sourceCode":"        def mask_overlap(s):\n            \"\"\"\n            mask overlap information\n            data after self.name[end] with self.trunc_days that contains future info are also considered as overlap info\n\n            Approximately the diagnal + horizon length of data are masked.\n            \"\"\"\n            start, end = s.name\n            end = get_date_by_shift(trading_date=end, shift=self.trunc_days - 1, future=True)\n            return s.mask((s.index >= start) & (s.index <= end))\n\n        ic_df_avail = ic_df_avail.apply(mask_overlap)  # apply to each col\n\n        # 2) filter the info with too long periods\n        total_len = self.step * self.hist_step_n\n        if ic_df_avail.shape[0] >= total_len:\n            return ic_df_avail.iloc[-total_len:]\n        else:\n            raise ValueError(\"the history of distribution data is not long enough.\")\n\n    def _prepare_seg(self, segment: Text) -> List[MetaTask]:\n        if isinstance(self.segments, float):\n            train_task_n = int(len(self.meta_task_l) * self.segments)\n            if segment == \"train\":\n                train_tasks = self.meta_task_l[:train_task_n]\n                get_module_logger(\"MetaDatasetDS\").info(f\"The first train meta task: {train_tasks[0]}\")\n                return train_tasks\n            elif segment == \"test\":\n                test_tasks = self.meta_task_l[train_task_n:]\n                get_module_logger(\"MetaDatasetDS\").info(f\"The first test meta task: {test_tasks[0]}\")\n                return test_tasks\n            else:\n                raise NotImplementedError(f\"This type of input is not supported\")\n        elif isinstance(self.segments, str):\n            train_tasks = []\n            test_tasks = []\n            for t in self.meta_task_l:","sourceCodeStart":366,"sourceCodeEnd":402,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/qlib/contrib/meta/data_selection/dataset.py#L366-L402","documentation":"MetaDatasetDS builds meta tasks from a rolling IC history: it needs at least self.step * self.hist_step_n available rows of IC data to look back hist_step_n windows of step days. If ic_df_avail has fewer rows than that, the distribution history is too short and it raises ValueError.","triggerScenarios":"Constructing MetaDatasetDS with a training start such that the IC series before it is shorter than step * hist_step_n; or large step/hist_step_n hyperparameters relative to the backtest period.","commonSituations":"Setting train_start too early in the data; increasing hist_step_n for a longer lookback without extending the preceding evaluation period; using a short alpha158/alpha360 run whose IC history covers few dates.","solutions":["Move train_start later so at least step * hist_step_n days of IC history exist before it.","Reduce hist_step_n or step so the required history fits what you have.","Regenerate the underlying IC records over a longer rolling backtest window."],"exampleFix":"// before\nmds = MetaDatasetDS(task_tpl=tpl, train_start=\"2010-01-01\", step=20, hist_step_n=10)  # < 200 prior days\n\n// after\nmds = MetaDatasetDS(task_tpl=tpl, train_start=\"2011-06-01\", step=20, hist_step_n=10)  # >= 200 prior days","handlingStrategy":"validation","validationCode":"required = step * hist_step_n\navail = len(ic_df)  # rows before train_start\nif avail < required:\n    raise ValueError(f\"need {required} prior IC days, have {avail}; lower hist_step_n/step or move train_start\")","typeGuard":null,"tryCatchPattern":"try:\n    mds = MetaDatasetDS(task_tpl=tpl, ...)\nexcept ValueError as e:\n    if \"not long enough\" in str(e):\n        mds = MetaDatasetDS(task_tpl=tpl, hist_step_n=hist_step_n // 2, ...)\n    else:\n        raise","preventionTips":["Compute step * hist_step_n and check IC-history length before constructing MetaDatasetDS.","Start meta training at least that many days after your IC backtest begins.","Run a longer rolling backtest to build up IC history."],"tags":["qlib","meta-learning","data-selection","data-coverage"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}