{"record":{"id":"51af7ad87bd5f6f7","repo":"microsoft/qlib","slug":"most-of-samples-are-dropped-please-check-this-tas","errorCode":null,"errorMessage":"Most of samples are dropped. Please check this task: {task}","messagePattern":"Most of samples are dropped\\. Please check this task: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"qlib/contrib/meta/data_selection/dataset.py","lineNumber":161,"sourceCode":"            please refer to the docs of _prepare_meta_ipt for detailed explanation.\n        \"\"\"\n        super().__init__(task, meta_info)\n        self.fill_method = fill_method\n\n        time_perf = self._get_processed_meta_info()\n        self.processed_meta_input = {\"time_perf\": time_perf}\n        # FIXME: memory issue in this step\n        if mode == MetaTask.PROC_MODE_FULL:\n            # process metainfo_\n            ds = self.get_dataset()\n\n            # these three lines occupied 70% of the time of initializing MetaTaskDS\n            d_train, d_test = ds.prepare([\"train\", \"test\"], col_set=[\"feature\", \"label\"])\n            prev_size = d_test.shape[0]\n            d_train = d_train.dropna(axis=0)\n            d_test = d_test.dropna(axis=0)\n            if prev_size == 0 or d_test.shape[0] / prev_size <= 0.1:\n                raise ValueError(f\"Most of samples are dropped. Please check this task: {task}\")\n\n            assert (\n                d_test.groupby(\"datetime\", group_keys=False).size().shape[0] >= 5\n            ), \"In this segment, this trading dates is less than 5, you'd better check the data.\"\n\n            sample_time_belong = np.zeros((d_train.shape[0], time_perf.shape[1]))\n            for i, col in enumerate(time_perf.columns):\n                # these two lines of code occupied 20% of the time of initializing MetaTaskDS\n                slc = slice(*d_train.index.slice_locs(start=col[0], end=col[1]))\n                sample_time_belong[slc, i] = 1.0\n\n            # If you want that last month also belongs to the last time_perf\n            # Assumptions: the latest data has similar performance like the last month\n            sample_time_belong[sample_time_belong.sum(axis=1) != 1, -1] = 1.0\n\n            self.processed_meta_input.update(\n                dict(\n                    X=d_train[\"feature\"],","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/qlib/contrib/meta/data_selection/dataset.py#L143-L179","documentation":"When a MetaTaskDS (meta-learning dataset for data selection) processes a task in PROC_MODE_FULL, it drops NaN rows from the prepared train/test data. If the test segment loses more than 90% of its rows (or was empty to begin with), the task is judged unusable and this ValueError is raised, naming the offending task.","triggerScenarios":"MetaTaskDS init or prepare where d_test.shape[0]/prev_size <= 0.1 after dropna — e.g. a task whose label window extends past available data, so almost every row has NaN labels.","commonSituations":"Task templates with test segments near the end of the available data (label horizon overruns the calendar); misconfigured segments in the task dict; instrument universes whose data ends before the segment; handlers producing mostly-NaN features.","solutions":["Shorten or shift the task's test segment so the label horizon (e.g. learned days) fits inside available data.","Check the underlying DataHandler output for the offending task: prepare the segment and inspect df.isna().mean() to see whether features or labels produce the NaNs.","Widen the data coverage (more dates/instruments) for the universe the task uses.","Catch this per-task and skip the bad task when building a MetaDatasetDS over many tasks."],"exampleFix":"// before\ntask = {\"dataset\": {\"kwargs\": {\"segments\": {\"test\": (\"2019-01-01\", \"2020-12-31\")}}}}  # labels run past data end\n\n// after\ntask = {\"dataset\": {\"kwargs\": {\"segments\": {\"test\": (\"2019-01-01\", \"2020-06-30\")}}}}  # label horizon fits","handlingStrategy":"try-catch","validationCode":"d_test = task_handler.prepare(\"test\", col_set=[\"feature\", \"label\"])\nprev = d_test.shape[0]\nsurv = d_test.dropna().shape[0]\nif prev == 0 or surv / prev <= 0.1:\n    logger.warning(\"task %s drops most samples; adjust segments\", task)","typeGuard":null,"tryCatchPattern":"good_tasks = []\nfor task in tasks:\n    try:\n        mtds = MetaTaskDS(task=task, ...)\n        good_tasks.append(mtds)\n    except ValueError as e:\n        if \"Most of samples are dropped\" in str(e):\n            logger.warning(\"skipping bad task: %s\", task)\n            continue\n        raise","preventionTips":["Keep label horizons strictly inside available data for every segment.","Dry-run dropna() coverage checks per task before building the meta dataset.","Wrap per-task construction in try/except when iterating over many rolling tasks."],"tags":["qlib","meta-learning","data-selection","nan","data-quality"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}