microsoft/qlib · error · ValueError

Must specify the path to save the dataset.

Error message

Must specify the path to save the dataset.

What it means

Raised by HighfreqDatasetProvider._gen_dataframe when the dataset handler config dict passed in has no 'path' key. The method does config.pop('path') to both retrieve the pickle location and remove it from the config before instantiating the handler via init_instance_by_config. A missing 'path' raises KeyError, which is re-raised as ValueError with this message.

Source

Thrown at qlib/contrib/data/highfreq_provider.py:128

            custom_ops=[DayLast, FFillNan, BFillNan, Date, Select, IsNull, IsInf, Cut],
            expression_cache=None,
            **qlib_conf,
        )

    def _prepare_calender_cache(self):
        """preload the calendar for cache"""

        # This code used the copy-on-write feature of Linux
        # to avoid calculating the calendar multiple times in the subprocess.
        # This code may accelerate, but may be not useful on Windows and Mac Os
        Cal.calendar(freq=self.freq)
        get_calendar_day(freq=self.freq)

    def _gen_dataframe(self, config, datasets=["train", "valid", "test"]):
        try:
            path = config.pop("path")
        except KeyError as e:
            raise ValueError("Must specify the path to save the dataset.") from e
        if os.path.isfile(path):
            start = time.time()
            self.logger.info(f"[{__name__}]Dataset exists, load from disk.")

            # res = dataset.prepare(['train', 'valid', 'test'])
            with open(path, "rb") as f:
                data = pkl.load(f)
            if isinstance(data, dict):
                res = [data[i] for i in datasets]
            else:
                res = data.prepare(datasets)
            self.logger.info(f"[{__name__}]Data loaded, time cost: {time.time() - start:.2f}")
        else:
            if not os.path.exists(os.path.dirname(path)):
                os.makedirs(os.path.dirname(path))
            self.logger.info(f"[{__name__}]Generating dataset")
            start_time = time.time()
            self._prepare_calender_cache()

View on GitHub (pinned to 79633dd950)

Solutions

1. Add a 'path' entry pointing to the .pkl file to the dataset config dict before calling _gen_dataframe, e.g. config['kwargs']['path'] = '/data/dataset.pkl'.
2. Check the config for the key before calling: if 'path' not in config: raise a clear error at the caller level.
3. Make sure 'path' is at the level _gen_dataframe expects (the top of the config dict it receives, since it pops from that dict directly).

Example fix

// before
config = {"class": "DataHandlerLP", "kwargs": {"instruments": insts}}
provider._gen_dataframe(config)

// after
config = {"class": "DataHandlerLP", "kwargs": {"instruments": insts}, "path": "/data/highfreq/dataset.pkl"}
provider._gen_dataframe(config)

if "path" not in config:
    raise ValueError("highfreq dataset config requires 'path' (pickle dump location)")
provider._gen_dataframe(config)

def has_dataset_path(config: dict) -> bool:
    return isinstance(config, dict) and isinstance(config.get("path"), str) and len(config["path"]) > 0

try:
    res = provider._gen_dataframe(config)
except ValueError as e:
    if "Must specify the path" in str(e):
        raise ValueError(f"dataset config missing 'path': {config}") from e
    raise

When it happens

Trigger: Calling provider._gen_dataframe(config) (directly or through the high-frequency data workflow) with a config dict whose 'kwargs' lack 'path', e.g. {'class': 'DataHandlerLP', 'kwargs': {...}} without a 'path' entry.

Common situations: Porting a standard qlib workflow config to the highfreq provider and forgetting that this provider requires an explicit dataset dump path; typos like 'save_path' or 'dump_path' instead of 'path'; building config programmatically and omitting the key.

Related errors

• invalid memory_mode `{self.memory_mode}`
• Empty data from dataset, please check your dataset config.
• Empty data from dataset, please check your dataset config.
• Empty data from dataset, please check your dataset config.
• freq(={freq}) missing group(={_gp})

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/117dcd627bcbbac4. Report an issue: GitHub.