microsoft/qlib · error · ValueError

proc_func is not supported by the storage {type(data_storage

Error message

proc_func is not supported by the storage {type(data_storage)}

What it means

`DataHandler.fetch(..., proc_func=f)` supports a post-fetch callback only when the underlying storage is a plain pd.DataFrame, where it can slice first and copy. When the handler stores data in a `BaseHandlerStorage` (e.g. HashingStockStorage), the fetch is delegated to the storage object which has no proc_func parameter, so passing one raises ValueError.

Source

Thrown at qlib/data/dataset/handler.py:315

            try:
                selector = slice(*selector)
            except ValueError:
                get_module_logger("DataHandlerLP").info(f"Fail to converting to query to slice. It will used directly")

        if isinstance(data_storage, pd.DataFrame):
            data_df = data_storage
            if proc_func is not None:
                # FIXME: fetching by time first will be more friendly to `proc_func`
                # Copy in case of `proc_func` changing the data inplace....
                data_df = proc_func(fetch_df_by_index(data_df, selector, level, fetch_orig=self.fetch_orig).copy())
                data_df = fetch_df_by_col(data_df, col_set)
            else:
                # Fetch column  first will be more friendly to SepDataFrame
                data_df = fetch_df_by_col(data_df, col_set)
                data_df = fetch_df_by_index(data_df, selector, level, fetch_orig=self.fetch_orig)
        elif isinstance(data_storage, BaseHandlerStorage):
            if proc_func is not None:
                raise ValueError(f"proc_func is not supported by the storage {type(data_storage)}")
            data_df = data_storage.fetch(selector=selector, level=level, col_set=col_set, fetch_orig=self.fetch_orig)
        else:
            raise TypeError(f"data_storage should be pd.DataFrame|HashingStockStorage, not {type(data_storage)}")

        if squeeze:
            # squeeze columns
            data_df = data_df.squeeze()
            # squeeze index
            if isinstance(selector, (str, pd.Timestamp)):
                data_df = data_df.reset_index(level=level, drop=True)
        return data_df

    def get_cols(self, col_set=DataHandlerABC.CS_ALL) -> list:
        """
        get the column names

        Parameters
        ----------

View on GitHub (pinned to 79633dd950)

Solutions

Drop proc_func and replicate its logic on the returned DataFrame: `df = handler.fetch(...); df = proc_func(df)`.
Or keep the handler on DataFrame storage (data_storage backed by pd.DataFrame) if proc_func is essential.
Push the transformation into processors configured on the handler instead.

Example fix

# before
df = handler.fetch(sel, proc_func=my_func)

# after
df = handler.fetch(sel)
df = my_func(df)

Defensive patterns

Strategy: type-guard

Validate before calling

from qlib.data.dataset.handler import BaseHandlerStorage

def fetch_safe(handler, selector, proc_func=None, **kw):
    storage = getattr(handler, '_data', None)
    if proc_func is not None and isinstance(storage, BaseHandlerStorage):
        return proc_func(handler.fetch(selector, **kw))
    return handler.fetch(selector, proc_func=proc_func, **kw)

Type guard

from qlib.data.dataset.handler import BaseHandlerStorage

def storage_supports_proc_func(handler) -> bool:
    return not isinstance(getattr(handler, '_data', None), BaseHandlerStorage)

Prevention

Prefer processors over proc_func for reproducible pipelines.
Apply custom transforms to the fetch() result, not inside fetch(), when using hashing storage.

When it happens

Trigger: Constructing/using a DataHandler whose `data_storage` is a HashingStockStorage (hashing-storage mode for large data) and calling `handler.fetch(..., proc_func=my_func)`.

Common situations: Workflows that add custom post-processing to fetch() and then switch the handler to hashing storage to save memory; copying example code from DataFrame-storage handlers into hashing-storage configs.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/74f1bf46e2a0aee2. Report an issue: GitHub.