{"record":{"id":"117dcd627bcbbac4","repo":"microsoft/qlib","slug":"must-specify-the-path-to-save-the-dataset","errorCode":null,"errorMessage":"Must specify the path to save the dataset.","messagePattern":"Must specify the path to save the dataset\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"qlib/contrib/data/highfreq_provider.py","lineNumber":128,"sourceCode":"            custom_ops=[DayLast, FFillNan, BFillNan, Date, Select, IsNull, IsInf, Cut],\n            expression_cache=None,\n            **qlib_conf,\n        )\n\n    def _prepare_calender_cache(self):\n        \"\"\"preload the calendar for cache\"\"\"\n\n        # This code used the copy-on-write feature of Linux\n        # to avoid calculating the calendar multiple times in the subprocess.\n        # This code may accelerate, but may be not useful on Windows and Mac Os\n        Cal.calendar(freq=self.freq)\n        get_calendar_day(freq=self.freq)\n\n    def _gen_dataframe(self, config, datasets=[\"train\", \"valid\", \"test\"]):\n        try:\n            path = config.pop(\"path\")\n        except KeyError as e:\n            raise ValueError(\"Must specify the path to save the dataset.\") from e\n        if os.path.isfile(path):\n            start = time.time()\n            self.logger.info(f\"[{__name__}]Dataset exists, load from disk.\")\n\n            # res = dataset.prepare(['train', 'valid', 'test'])\n            with open(path, \"rb\") as f:\n                data = pkl.load(f)\n            if isinstance(data, dict):\n                res = [data[i] for i in datasets]\n            else:\n                res = data.prepare(datasets)\n            self.logger.info(f\"[{__name__}]Data loaded, time cost: {time.time() - start:.2f}\")\n        else:\n            if not os.path.exists(os.path.dirname(path)):\n                os.makedirs(os.path.dirname(path))\n            self.logger.info(f\"[{__name__}]Generating dataset\")\n            start_time = time.time()\n            self._prepare_calender_cache()","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/microsoft/qlib/blob/79633dd9506ea689e5400dea0197717b5b3d74b7/qlib/contrib/data/highfreq_provider.py#L110-L146","documentation":"Raised by HighfreqDatasetProvider._gen_dataframe when the dataset handler config dict passed in has no 'path' key. The method does config.pop('path') to both retrieve the pickle location and remove it from the config before instantiating the handler via init_instance_by_config. A missing 'path' raises KeyError, which is re-raised as ValueError with this message.","triggerScenarios":"Calling provider._gen_dataframe(config) (directly or through the high-frequency data workflow) with a config dict whose 'kwargs' lack 'path', e.g. {'class': 'DataHandlerLP', 'kwargs': {...}} without a 'path' entry.","commonSituations":"Porting a standard qlib workflow config to the highfreq provider and forgetting that this provider requires an explicit dataset dump path; typos like 'save_path' or 'dump_path' instead of 'path'; building config programmatically and omitting the key.","solutions":["Add a 'path' entry pointing to the .pkl file to the dataset config dict before calling _gen_dataframe, e.g. config['kwargs']['path'] = '/data/dataset.pkl'.","Check the config for the key before calling: if 'path' not in config: raise a clear error at the caller level.","Make sure 'path' is at the level _gen_dataframe expects (the top of the config dict it receives, since it pops from that dict directly)."],"exampleFix":"// before\nconfig = {\"class\": \"DataHandlerLP\", \"kwargs\": {\"instruments\": insts}}\nprovider._gen_dataframe(config)\n\n// after\nconfig = {\"class\": \"DataHandlerLP\", \"kwargs\": {\"instruments\": insts}, \"path\": \"/data/highfreq/dataset.pkl\"}\nprovider._gen_dataframe(config)","handlingStrategy":"validation","validationCode":"if \"path\" not in config:\n    raise ValueError(\"highfreq dataset config requires 'path' (pickle dump location)\")\nprovider._gen_dataframe(config)","typeGuard":"def has_dataset_path(config: dict) -> bool:\n    return isinstance(config, dict) and isinstance(config.get(\"path\"), str) and len(config[\"path\"]) > 0","tryCatchPattern":"try:\n    res = provider._gen_dataframe(config)\nexcept ValueError as e:\n    if \"Must specify the path\" in str(e):\n        raise ValueError(f\"dataset config missing 'path': {config}\") from e\n    raise","preventionTips":["Always build highfreq dataset configs through a helper that injects 'path'.","Use absolute paths so generation and reload agree across working directories.","Validate required config keys at load time, not at generation time."],"tags":["qlib","highfreq","config","dataset"],"backgroundTag":null,"analyzedSha":"79633dd9506ea689e5400dea0197717b5b3d74b7","analyzedAt":"2026-08-15T07:01:27.511Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}