{"record":{"id":"ac345135a6d0fdf8","repo":"pola-rs/polars","slug":"cannot-use-hive-partitions-with-use-pyarrow-tru","errorCode":null,"errorMessage":"cannot use `hive_partitions` with `use_pyarrow=True`\n\nHint: Pass `pyarrow_options` instead with a 'partitioning' entry.","messagePattern":"cannot use `hive_partitions` with `use_pyarrow=True`\n\nHint: Pass `pyarrow_options` instead with a 'partitioning' entry\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/parquet/functions.py","lineNumber":244,"sourceCode":"        issue_unstable_warning(msg)\n\n    # Dispatch to pyarrow if requested\n    if use_pyarrow:\n        if n_rows is not None:\n            msg = \"`n_rows` cannot be used with `use_pyarrow=True`\"\n            raise ValueError(msg)\n        if include_file_paths is not None:\n            msg = \"`include_file_paths` cannot be used with `use_pyarrow=True`\"\n            raise ValueError(msg)\n        if schema is not None:\n            msg = \"`schema` cannot be used with `use_pyarrow=True`\"\n            raise ValueError(msg)\n        if hive_schema is not None:\n            msg = (\n                \"cannot use `hive_partitions` with `use_pyarrow=True`\"\n                \"\\n\\nHint: Pass `pyarrow_options` instead with a 'partitioning' entry.\"\n            )\n            raise TypeError(msg)\n        return _read_parquet_with_pyarrow(\n            source,\n            columns=columns,\n            storage_options=storage_options,\n            pyarrow_options=pyarrow_options,\n            memory_map=memory_map,\n            rechunk=rechunk,\n        )\n\n    if allow_missing_columns is not None:\n        issue_deprecation_warning(\n            \"the parameter `allow_missing_columns` for `read_parquet` is deprecated. \"\n            \"Use the parameter `missing_columns` instead and pass one of \"\n            \"`('insert', 'raise')`.\",\n            version=\"1.30.0\",\n        )\n\n        missing_columns = \"insert\" if allow_missing_columns else \"raise\"","sourceCodeStart":226,"sourceCodeEnd":262,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/io/parquet/functions.py#L226-L262","documentation":"TypeError (not ValueError) raised by pl.read_parquet when use_pyarrow=True is combined with hive_schema. hive_schema declares the dtypes of hive-partitioned directory columns for the native reader; under the pyarrow engine, partitioning is configured differently, hence the hint in the message: pass a 'partitioning' entry inside pyarrow_options, which is forwarded to pyarrow.dataset.","triggerScenarios":"pl.read_parquet('hive_dir/', use_pyarrow=True, hive_schema={'year': pl.Int32}) — any non-None hive_schema with use_pyarrow=True.","commonSituations":"Reading hive-partitioned datasets where the native engine inferred partition keys as the wrong type (e.g. 'year' as String instead of Int), switching to use_pyarrow to fix it, and keeping hive_schema; or upgrading polars and hitting changed partition-inference defaults.","solutions":["Follow the hint: keep use_pyarrow=True, drop hive_schema, and pass pyarrow_options={'partitioning': ...} — either a pyarrow.dataset.Partitioning object or a schema/flavor, e.g. pyarrow_options={'partitioning': ds.partitioning(pa.schema([('year', pa.int32())]), flavor='hive')}.","Or stay on the native engine (no use_pyarrow) and keep hive_schema, which is exactly what it was designed for.","If partition keys are just mis-typed, also consider casting them after read: .with_columns(pl.col('year').cast(pl.Int32))."],"exampleFix":"# before\npl.read_parquet('data/', use_pyarrow=True, hive_schema={'year': pl.Int32})\n\n# after\nimport pyarrow.dataset as ds, pyarrow as pa\npl.read_parquet(\n    'data/',\n    use_pyarrow=True,\n    pyarrow_options={'partitioning': ds.partitioning(pa.schema([('year', pa.int32())]), flavor='hive')},\n)","handlingStrategy":"validation","validationCode":"if hive_schema is not None and use_pyarrow:\n    pyarrow_options = dict(pyarrow_options or {})\n    import pyarrow as pa, pyarrow.dataset as ds\n    pyarrow_options.setdefault(\n        'partitioning',\n        ds.partitioning(pa.schema([(k, v.to_arrow()) for k, v in hive_schema.items()]), flavor='hive'),\n    )\n    hive_schema = None\npl.read_parquet(path, use_pyarrow=use_pyarrow, hive_schema=hive_schema, pyarrow_options=pyarrow_options)","typeGuard":null,"tryCatchPattern":"try:\n    df = pl.read_parquet(path, use_pyarrow=True, hive_schema=hive_schema)\nexcept TypeError as e:  # note: this guard raises TypeError, not ValueError\n    if 'hive_partitions' in str(e):\n        df = pl.read_parquet(path, hive_schema=hive_schema)  # native engine\n    else:\n        raise","preventionTips":["For hive datasets under pyarrow, configure partitioning via pyarrow_options['partitioning'] from the start.","Remember this specific check raises TypeError while the sibling checks raise ValueError.","Consider casting mis-typed partition keys after read instead of pinning an engine."],"tags":["parquet","pyarrow","hive-partitioning","parameter-conflict","io"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}