microsoft/qlib · error · ValueError

Unrecognized data shape: {shape}

Error message

Unrecognized data shape: {shape}

What it means

In qlib/rl/data/pickle_styled.py:67, when process data is loaded from pickle files the column names are inferred from the array's feature dimension. Only shape 6 (high/low/open/close/vwap/volume), shape 5 (high/low/open/close/volume), and an 11/12-column order-book case (bid/ask levels) are recognized. Any other last-dimension size raises ValueError('Unrecognized data shape: <shape>').

Source

Thrown at qlib/rl/data/pickle_styled.py:67

            "$close",
            "$vwap",
            "$bid",
            "$ask",
            "$volume",
            "$bidV",
            "$bidV1",
            "$bidV3",
            "$bidV5",
            "$askV",
            "$askV1",
            "$askV3",
            "$askV5",
        ]
    if shape == 6:
        return ["$high", "$low", "$open", "$close", "$vwap", "$volume"]
    elif shape == 5:
        return ["$high", "$low", "$open", "$close", "$volume"]
    raise ValueError(f"Unrecognized data shape: {shape}")


def _find_pickle(filename_without_suffix: Path) -> Path:
    suffix_list = [".pkl", ".pkl.backtest"]
    paths: List[Path] = []
    for suffix in suffix_list:
        path = filename_without_suffix.parent / (filename_without_suffix.name + suffix)
        if path.exists():
            paths.append(path)
    if not paths:
        raise FileNotFoundError(f"No file starting with '{filename_without_suffix}' found")
    if len(paths) > 1:
        raise ValueError(f"Multiple paths are found with prefix '{filename_without_suffix}': {paths}")
    return paths[0]


@lru_cache(maxsize=10)  # 10 * 40M = 400MB
def _read_pickle(filename_without_suffix: Path) -> pd.DataFrame:

View on GitHub (pinned to 79633dd950)

Solutions

1. Regenerate the pickle data with an accepted schema: 6 features ($high,$low,$open,$close,$vwap,$volume), 5 features (no vwap), or the standard order-book layout
2. If you need custom columns, construct the DataFrame with explicit column names yourself instead of relying on shape inference
3. Check for accidental extra columns/rows introduced by your preprocessing

Example fix

# before
# pickle with 7 columns -> ValueError('Unrecognized data shape: (7,)')

# after
# keep the standard 6 columns when dumping
df = df[['$high', '$low', '$open', '$close', '$vwap', '$volume']]
df.to_pickle('data.pkl')

shape = arr.shape[-1]
assert shape in (5, 6) or shape >= 11, f'data feature dimension {shape} not recognized'

try:
    data = pickle_styled.load(path)
except ValueError as e:
    if 'Unrecognized data shape' in str(e):
        raise ValueError('regenerate pickle with standard 5/6/book columns') from e
    raise

When it happens

Trigger: Loading a pickle-styled process data file whose feature axis is not 5, 6, or the recognized book depth; hand-crafted pickles with extra columns or a different column order/count; data generated by a modified/different-versioned preprocessing script.

Common situations: See trigger scenarios.

Related errors

• No file starting with '{filename_without_suffix}' found
• Multiple paths are found with prefix '{filename_without_suff
• Order direction cannot be none when deal_price_type is not c
• Unsupported deal_price_type: {self.deal_price_type}
• This type of input {rtype} is not supported

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/d0a0d79895c5b689. Report an issue: GitHub.