pola-rs/polars · error · TypeError

cannot use `hive_partitions` with `use_pyarrow=True` Hint:

Error message

cannot use `hive_partitions` with `use_pyarrow=True`

Hint: Pass `pyarrow_options` instead with a 'partitioning' entry.

What it means

TypeError (not ValueError) raised by pl.read_parquet when use_pyarrow=True is combined with hive_schema. hive_schema declares the dtypes of hive-partitioned directory columns for the native reader; under the pyarrow engine, partitioning is configured differently, hence the hint in the message: pass a 'partitioning' entry inside pyarrow_options, which is forwarded to pyarrow.dataset.

Source

Thrown at py-polars/src/polars/io/parquet/functions.py:244

        issue_unstable_warning(msg)

    # Dispatch to pyarrow if requested
    if use_pyarrow:
        if n_rows is not None:
            msg = "`n_rows` cannot be used with `use_pyarrow=True`"
            raise ValueError(msg)
        if include_file_paths is not None:
            msg = "`include_file_paths` cannot be used with `use_pyarrow=True`"
            raise ValueError(msg)
        if schema is not None:
            msg = "`schema` cannot be used with `use_pyarrow=True`"
            raise ValueError(msg)
        if hive_schema is not None:
            msg = (
                "cannot use `hive_partitions` with `use_pyarrow=True`"
                "\n\nHint: Pass `pyarrow_options` instead with a 'partitioning' entry."
            )
            raise TypeError(msg)
        return _read_parquet_with_pyarrow(
            source,
            columns=columns,
            storage_options=storage_options,
            pyarrow_options=pyarrow_options,
            memory_map=memory_map,
            rechunk=rechunk,
        )

    if allow_missing_columns is not None:
        issue_deprecation_warning(
            "the parameter `allow_missing_columns` for `read_parquet` is deprecated. "
            "Use the parameter `missing_columns` instead and pass one of "
            "`('insert', 'raise')`.",
            version="1.30.0",
        )

        missing_columns = "insert" if allow_missing_columns else "raise"

View on GitHub (pinned to df599052da)

Solutions

Follow the hint: keep use_pyarrow=True, drop hive_schema, and pass pyarrow_options={'partitioning': ...} — either a pyarrow.dataset.Partitioning object or a schema/flavor, e.g. pyarrow_options={'partitioning': ds.partitioning(pa.schema([('year', pa.int32())]), flavor='hive')}.
Or stay on the native engine (no use_pyarrow) and keep hive_schema, which is exactly what it was designed for.
If partition keys are just mis-typed, also consider casting them after read: .with_columns(pl.col('year').cast(pl.Int32)).

Example fix

# before
pl.read_parquet('data/', use_pyarrow=True, hive_schema={'year': pl.Int32})

# after
import pyarrow.dataset as ds, pyarrow as pa
pl.read_parquet(
    'data/',
    use_pyarrow=True,
    pyarrow_options={'partitioning': ds.partitioning(pa.schema([('year', pa.int32())]), flavor='hive')},
)

Defensive patterns

Strategy: validation

Validate before calling

if hive_schema is not None and use_pyarrow:
    pyarrow_options = dict(pyarrow_options or {})
    import pyarrow as pa, pyarrow.dataset as ds
    pyarrow_options.setdefault(
        'partitioning',
        ds.partitioning(pa.schema([(k, v.to_arrow()) for k, v in hive_schema.items()]), flavor='hive'),
    )
    hive_schema = None
pl.read_parquet(path, use_pyarrow=use_pyarrow, hive_schema=hive_schema, pyarrow_options=pyarrow_options)

Try / catch

try:
    df = pl.read_parquet(path, use_pyarrow=True, hive_schema=hive_schema)
except TypeError as e:  # note: this guard raises TypeError, not ValueError
    if 'hive_partitions' in str(e):
        df = pl.read_parquet(path, hive_schema=hive_schema)  # native engine
    else:
        raise

Prevention

For hive datasets under pyarrow, configure partitioning via pyarrow_options['partitioning'] from the start.
Remember this specific check raises TypeError while the sibling checks raise ValueError.
Consider casting mis-typed partition keys after read instead of pinning an engine.

When it happens

Trigger: pl.read_parquet('hive_dir/', use_pyarrow=True, hive_schema={'year': pl.Int32}) — any non-None hive_schema with use_pyarrow=True.

Common situations: Reading hive-partitioned datasets where the native engine inferred partition keys as the wrong type (e.g. 'year' as String instead of Int), switching to use_pyarrow to fix it, and keeping hive_schema; or upgrading polars and hitting changed partition-inference defaults.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/ac345135a6d0fdf8. Report an issue: GitHub.