pola-rs/polars · error · ValueError

`include_file_paths` cannot be used with `use_pyarrow=True`

Error message

`include_file_paths` cannot be used with `use_pyarrow=True`

What it means

ValueError raised by pl.read_parquet when use_pyarrow=True is combined with include_file_paths. include_file_paths adds a column with each source file's path and is implemented only in polars' native multi-file reader; the pyarrow dispatch path (_read_parquet_with_pyarrow) has no equivalent bridge, so the combination is rejected before reading.

Source

Thrown at py-polars/src/polars/io/parquet/functions.py:235

    with `LazyFrame` s.

    """
    if schema is not None:
        msg = "the `schema` parameter of `read_parquet` is considered unstable."
        issue_unstable_warning(msg)

    if hive_schema is not None:
        msg = "the `hive_schema` parameter of `read_parquet` is considered unstable."
        issue_unstable_warning(msg)

    # Dispatch to pyarrow if requested
    if use_pyarrow:
        if n_rows is not None:
            msg = "`n_rows` cannot be used with `use_pyarrow=True`"
            raise ValueError(msg)
        if include_file_paths is not None:
            msg = "`include_file_paths` cannot be used with `use_pyarrow=True`"
            raise ValueError(msg)
        if schema is not None:
            msg = "`schema` cannot be used with `use_pyarrow=True`"
            raise ValueError(msg)
        if hive_schema is not None:
            msg = (
                "cannot use `hive_partitions` with `use_pyarrow=True`"
                "\n\nHint: Pass `pyarrow_options` instead with a 'partitioning' entry."
            )
            raise TypeError(msg)
        return _read_parquet_with_pyarrow(
            source,
            columns=columns,
            storage_options=storage_options,
            pyarrow_options=pyarrow_options,
            memory_map=memory_map,
            rechunk=rechunk,
        )

View on GitHub (pinned to df599052da)

Solutions

1. Remove use_pyarrow so the native engine runs — it supports include_file_paths natively.
2. Keep pyarrow and re-add provenance afterwards by reading files one by one and calling df.with_columns(pl.lit(path).alias(include_file_paths)), concatenating the results.
3. If pyarrow partitioning metadata is what you need, use the native reader's hive_partitioning support or pyarrow.dataset directly and construct the path column from fragment info.

Example fix

# before
pl.read_parquet('part-*.parquet', use_pyarrow=True, include_file_paths='src')

# after
pl.read_parquet('part-*.parquet', include_file_paths='src')  # native engine

PYARROW_INCOMPATIBLE = {'n_rows', 'include_file_paths', 'schema', 'hive_schema'}

def read_parquet_safe(path, *, use_pyarrow=False, **kwargs):
    if use_pyarrow:
        clash = PYARROW_INCOMPATIBLE & kwargs.keys()
        if clash:
            raise ValueError(f'remove {clash} or disable use_pyarrow')
    return pl.read_parquet(path, use_pyarrow=use_pyarrow, **kwargs)

try:
    df = pl.read_parquet(path, use_pyarrow=True, include_file_paths='src')
except ValueError as e:
    if 'include_file_paths' in str(e):
        df = pl.read_parquet(path, include_file_paths='src')  # native
    else:
        raise

When it happens

Trigger: pl.read_parquet(['a.parquet','b.parquet'], use_pyarrow=True, include_file_paths='source'). Any non-None include_file_paths with use_pyarrow=True triggers it.

Common situations: Reading a directory/glob of partitioned parquet files with the pyarrow engine while wanting provenance per row; migrating a working native-engine pipeline to use_pyarrow for compatibility and forgetting the path column is native-only.

Related errors

• `n_rows` cannot be used with `use_pyarrow=True`
• `schema` cannot be used with `use_pyarrow=True`
• cannot use `hive_partitions` with `use_pyarrow=True` Hint:
• write_parquet with `use_pyarrow=True` allows only boolean va
• pyarrow is required for converting a pandas dataframe to Pol

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/a36e30970fc2112b. Report an issue: GitHub.