{"record":{"id":"1b4a076dabea5154","repo":"pola-rs/polars","slug":"cannot-infer-partition-value-from-parquet-metadata","errorCode":null,"errorMessage":"Cannot infer partition value from Parquet metadata for partition field '{field.name}': {lower_value=}, {upper_value=}","messagePattern":"Cannot infer partition value from Parquet metadata for partition field '(.+?)': (.+?), (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/iceberg/_sink.py","lineNumber":106,"sourceCode":"        if aggregate is None:\n            partition_values.append(None)\n            continue\n\n        source_type = schema.find_field(field.source_id).field_type\n        transform = field.transform.transform(source_type)\n        lower_value = transform(\n            partition_record_value(field, aggregate.current_min, schema)\n        )\n        upper_value = transform(\n            partition_record_value(field, aggregate.current_max, schema)\n        )\n        # A file can contain different source values in one transformed partition.\n        if lower_value != upper_value:\n            msg = (\n                \"Cannot infer partition value from Parquet metadata for partition \"\n                f\"field '{field.name}': {lower_value=}, {upper_value=}\"\n            )\n            raise ValueError(msg)\n        partition_values.append(lower_value)\n\n    return Record(*partition_values)\n\n\ndef _data_files_from_sink_metadata(\n    table_metadata: TableMetadata,\n    sinked_files: list[_IcebergSinkedFile],\n    nested_source_ids: set[int],\n) -> Iterable[DataFile]:\n    from pyiceberg.io.pyarrow import (\n        MetricModeTypes,\n        MetricsMode,\n        compute_statistics_plan,\n        parquet_path_to_id_mapping,\n    )\n    from pyiceberg.utils.concurrent import ExecutorFactory\n","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/pola-rs/polars/blob/fc24390824009e8fc5b7870e256273c1b140a019/py-polars/src/polars/io/iceberg/_sink.py#L88-L124","documentation":"The Iceberg sink infers each partition value from the min/max statistics in the Parquet file it just wrote. This error means the transformed lower and upper bounds differ, i.e. the file contains multiple distinct source values for one partition field, so no single partition value can be derived. The sink refuses to guess and raises ValueError instead of writing wrong partition metadata.","triggerScenarios":"Calling `sink_iceberg` on a table where a partition field uses a transform (e.g. bucket or truncate) and the written file's rows map to more than one transform result — e.g. lower_value != upper_value in the Parquet column aggregates for that field's source_id.","commonSituations":"Writing a file that mixes rows from multiple bucket/truncate partitions because the input wasn't pre-grouped; a mis-declared partition transform where rows actually sharing one partition key produce differing bounds due to transform/aggregate type mismatch; re-sinking old data into a table whose spec changed.","solutions":["Ensure the data being sunk is partitioned so that every file contains only rows belonging to one partition (e.g. sort or group the LazyFrame by the partition source column before sinking)","Verify the partition field's transform matches your intent — with `identity` transform a correctly-split file always yields equal lower/upper bounds","Check the metrics mode; if bounds are truncated or missing, `_partition_key_exprs` would fail earlier, so confirm statistics collection is intact for the source column","If the data legitimately spans partitions per file, use a different sink mode or restructure writes so each file maps to a single partition"],"exampleFix":"// before: unsorted input, one file spans multiple partition values\nlf.sink_iceberg(table)\n// after: sort by the partition source column so files are split per partition\nlf.sort('category').sink_iceberg(table)","handlingStrategy":"validation","validationCode":"import pyarrow.parquet as pq\nmd = pq.read_metadata(file_path)\n# after sinking, per partition field, min/max statistics must be equal\n# pre-check on data: each file must contain exactly one transform result\nassert lf.select(pl.col(partition_col).n_unique()).collect().item() <= 1 per output file","typeGuard":"def single_partition_value(stats_min, stats_max, transform) -> bool:\n    return transform(stats_min) == transform(stats_max)","tryCatchPattern":"try:\n    lf.sink_iceberg(table)\nexcept ValueError as e:\n    if 'Cannot infer partition value' in str(e):\n        lf.sort(partition_source_col).sink_iceberg(table)  # retry split per partition\n    else:\n        raise","preventionTips":["Sort or partition the LazyFrame by the partition source column before sinking so each file maps to one partition value","Prefer identity transforms when data may span multiple transform results per file","Verify table spec transforms match how your data is physically grouped"],"tags":["iceberg","partitioning","parquet-statistics","polars"],"backgroundTag":"ambiguous-partition-value","analyzedSha":"fc24390824009e8fc5b7870e256273c1b140a019","analyzedAt":"2026-09-02T20:08:03.494Z","contentChangedAt":"2026-09-02T20:08:03.494Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}