pola-rs/polars · error · ValueError

write_parquet with `use_pyarrow=True` cannot be combined wit

Error message

write_parquet with `use_pyarrow=True` cannot be combined with `metadata`

What it means

Raised by DataFrame.write_parquet(use_pyarrow=True) when a `metadata` mapping is also supplied. Custom key-value file metadata is implemented by polars' native rust writer; the pyarrow delegation path has no plumbing for it, so the combination is rejected up front (before file creation) instead of silently dropping your metadata.

Source

Thrown at py-polars/src/polars/dataframe/frame.py:4312

        ... )
        """
        if compression is None:
            compression = "uncompressed"
        if isinstance(file, (str, Path)):
            if partition_by is not None or (
                pyarrow_options is not None and pyarrow_options.get("partition_cols")
            ):
                file = normalize_filepath(file, check_not_directory=False)
            else:
                file = normalize_filepath(file)

        if use_pyarrow:
            if statistics == "full" or isinstance(statistics, dict):
                msg = "write_parquet with `use_pyarrow=True` allows only boolean values for `statistics`"
                raise ValueError(msg)
            if metadata is not None:
                msg = "write_parquet with `use_pyarrow=True` cannot be combined with `metadata`"
                raise ValueError(msg)
            if mkdir:
                msg = "write_parquet with `use_pyarrow=True` cannot be combined with `mkdir`"
                raise ValueError(msg)

            tbl = self.to_arrow()
            data = {}

            for i, column in enumerate(tbl):
                # extract the name before casting
                name = f"column_{i}" if column._name is None else column._name

                data[name] = column

            tbl = pa.table(data)

            # do not remove this import!
            # needed below
            import pyarrow.parquet  # noqa: F401

View on GitHub (pinned to df599052da)

Solutions

Drop `use_pyarrow=True` so the native writer applies your metadata
Or keep use_pyarrow=True and stamp metadata afterwards with pyarrow: read the parquet file's metadata, rewrite via `pq.write_table(..., additional_metadata=...)` — or simply omit it
Restructure the shared writer so metadata is only passed on native-writer calls

Example fix

# before
df.write_parquet('f.parquet', use_pyarrow=True, metadata={'creator': 'etl'})

# after
df.write_parquet('f.parquet', metadata={'creator': 'etl'})  # native writer

Defensive patterns

Strategy: validation

Validate before calling

if use_pyarrow and metadata is not None:
    raise ValueError('metadata requires the native writer; drop use_pyarrow')
df.write_parquet(path, use_pyarrow=use_pyarrow, metadata=metadata)

Try / catch

try:
    df.write_parquet(path, use_pyarrow=True, metadata=metadata)
except ValueError as e:
    if 'cannot be combined with `metadata`' in str(e):
        df.write_parquet(path, metadata=metadata)  # native writer honors it
    else:
        raise

Prevention

Attach file metadata only through the native writer path
Split writer configs: pyarrow path (partition_by, pyarrow_options) vs native path (metadata, rich statistics)
Add a pre-flight check that rejects incompatible write_parquet kwarg combinations

When it happens

Trigger: `df.write_parquet('f.parquet', use_pyarrow=True, metadata={'creator': 'etl-7'})`. Typically appears when use_pyarrow is enabled for pyarrow_options/partition_by while a shared write helper always attaches metadata.

Common situations: ETL lineage pipelines stamping dataset metadata on every file; writers parameterized so that partitioned outputs force use_pyarrow while metadata stays on; refactors that moved a metadata dict into a generic kwargs passthrough.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/f265b5eaf80e025d. Report an issue: GitHub.