pola-rs/polars · error · ValueError

Cannot infer partition value from Parquet metadata for parti

Error message

Cannot infer partition value from Parquet metadata for partition field '{field.name}': {lower_value=}, {upper_value=}

What it means

The Iceberg sink infers each partition value from the min/max statistics in the Parquet file it just wrote. This error means the transformed lower and upper bounds differ, i.e. the file contains multiple distinct source values for one partition field, so no single partition value can be derived. The sink refuses to guess and raises ValueError instead of writing wrong partition metadata.

Source

Thrown at py-polars/src/polars/io/iceberg/_sink.py:106

        if aggregate is None:
            partition_values.append(None)
            continue

        source_type = schema.find_field(field.source_id).field_type
        transform = field.transform.transform(source_type)
        lower_value = transform(
            partition_record_value(field, aggregate.current_min, schema)
        )
        upper_value = transform(
            partition_record_value(field, aggregate.current_max, schema)
        )
        # A file can contain different source values in one transformed partition.
        if lower_value != upper_value:
            msg = (
                "Cannot infer partition value from Parquet metadata for partition "
                f"field '{field.name}': {lower_value=}, {upper_value=}"
            )
            raise ValueError(msg)
        partition_values.append(lower_value)

    return Record(*partition_values)


def _data_files_from_sink_metadata(
    table_metadata: TableMetadata,
    sinked_files: list[_IcebergSinkedFile],
    nested_source_ids: set[int],
) -> Iterable[DataFile]:
    from pyiceberg.io.pyarrow import (
        MetricModeTypes,
        MetricsMode,
        compute_statistics_plan,
        parquet_path_to_id_mapping,
    )
    from pyiceberg.utils.concurrent import ExecutorFactory

View on GitHub (pinned to fc24390824)

Solutions

Ensure the data being sunk is partitioned so that every file contains only rows belonging to one partition (e.g. sort or group the LazyFrame by the partition source column before sinking)
Verify the partition field's transform matches your intent — with `identity` transform a correctly-split file always yields equal lower/upper bounds
Check the metrics mode; if bounds are truncated or missing, `_partition_key_exprs` would fail earlier, so confirm statistics collection is intact for the source column
If the data legitimately spans partitions per file, use a different sink mode or restructure writes so each file maps to a single partition

Example fix

// before: unsorted input, one file spans multiple partition values
lf.sink_iceberg(table)
// after: sort by the partition source column so files are split per partition
lf.sort('category').sink_iceberg(table)

Defensive patterns

Strategy: validation

Validate before calling

import pyarrow.parquet as pq
md = pq.read_metadata(file_path)
# after sinking, per partition field, min/max statistics must be equal
# pre-check on data: each file must contain exactly one transform result
assert lf.select(pl.col(partition_col).n_unique()).collect().item() <= 1 per output file

Type guard

def single_partition_value(stats_min, stats_max, transform) -> bool:
    return transform(stats_min) == transform(stats_max)

Try / catch

try:
    lf.sink_iceberg(table)
except ValueError as e:
    if 'Cannot infer partition value' in str(e):
        lf.sort(partition_source_col).sink_iceberg(table)  # retry split per partition
    else:
        raise

Prevention

Sort or partition the LazyFrame by the partition source column before sinking so each file maps to one partition value
Prefer identity transforms when data may span multiple transform results per file
Verify table spec transforms match how your data is physically grouped

When it happens

Trigger: Calling `sink_iceberg` on a table where a partition field uses a transform (e.g. bucket or truncate) and the written file's rows map to more than one transform result — e.g. lower_value != upper_value in the Parquet column aggregates for that field's source_id.

Common situations: Writing a file that mixes rows from multiple bucket/truncate partitions because the input wasn't pre-grouped; a mis-declared partition transform where rows actually sharing one partition key produce differing bounds due to transform/aggregate type mismatch; re-sinking old data into a table whose spec changed.

Related errors

AI-assisted analysis of pola-rs/polars@fc24390824 (2026-09-02). Data as JSON: /api/errors/1b4a076dabea5154. Report an issue: GitHub.