pola-rs/polars · critical · ValueError

native sink row count {num_rows} does not match Parquet meta

Error message

native sink row count {num_rows} does not match Parquet metadata row count {parquet_metadata.num_rows} for '{file_path}'

What it means

After the native sink writes a Parquet file, polars cross-checks the row count it wrote (`num_rows`) against the row count reported by the Parquet file's footer metadata. A mismatch means the physical file does not match what the sink engine reported — a corruption/integrity failure in the output pipeline — so it raises ValueError instead of committing bad Iceberg DataFile metadata.

Source

Thrown at py-polars/src/polars/io/iceberg/_sink.py:177

    import pyarrow as pa
    import pyarrow.parquet as pq
    from pyiceberg.io.pyarrow import (
        MetricModeTypes,
        _check_pyarrow_schema_compatible,
        data_file_statistics_from_parquet_metadata,
    )
    from pyiceberg.manifest import DataFile, DataFileContent, FileFormat

    schema = table_metadata.schema()
    file_path, num_rows, num_bytes, parquet_metadata_bytes = sinked_file
    parquet_metadata = pq.read_metadata(pa.BufferReader(parquet_metadata_bytes))

    if parquet_metadata.num_rows != num_rows:
        msg = (
            f"native sink row count {num_rows} does not match Parquet metadata "
            f"row count {parquet_metadata.num_rows} for '{file_path}'"
        )
        raise ValueError(msg)

    _check_pyarrow_schema_compatible(schema, parquet_metadata.schema.to_arrow_schema())
    statistics = data_file_statistics_from_parquet_metadata(
        parquet_metadata=parquet_metadata,
        stats_columns=statistics_plan,
        parquet_column_mapping=parquet_column_mapping,
    )
    partition = _infer_partition_from_statistics(
        statistics, table_metadata.spec(), schema
    )
    serialized_statistics = statistics.to_serialized_dict()
    for source_id, metrics_mode in nested_metrics_modes.items():
        serialized_statistics["lower_bounds"].pop(source_id, None)
        serialized_statistics["upper_bounds"].pop(source_id, None)
        if metrics_mode is MetricModeTypes.NONE:
            serialized_statistics["value_counts"].pop(source_id, None)
            serialized_statistics["null_value_counts"].pop(source_id, None)
            serialized_statistics["nan_value_counts"].pop(source_id, None)

View on GitHub (pinned to fc24390824)

Solutions

Retry the sink with a fresh, unique file location to rule out a corrupted partial file
Ensure only one writer targets each output path — use unique paths or rely on the sink's default naming, never share paths across concurrent jobs
Inspect the offending Parquet file (read its footer metadata) to confirm the row count discrepancy and check storage/driver logs for upload errors
Update polars and pyiceberg; if reproducible on a single version, report it as a sink-integrity bug

Example fix

// before: concurrent jobs overwrite the same target path
lf.sink_iceberg(table)
// after: run one sink per unique target, no concurrent writers to the same path
# serialize writes or give each job its own location
lf.sink_iceberg(table)  # single writer for this table path

Defensive patterns

Strategy: validation

Validate before calling

import pyarrow.parquet as pq
md = pq.read_metadata(file_path)
assert md.num_rows == expected_rows, f"row count mismatch: {md.num_rows} vs {expected_rows}"

Try / catch

try:
    lf.sink_iceberg(table)
except ValueError as e:
    if 'does not match Parquet metadata row count' in str(e):
        remove_or_quarantine_partial_files(target_dir)
        lf.sink_iceberg(table)  # retry with clean, unique paths
    else:
        raise

Prevention

Never let concurrent writers target the same output file path
Check storage/driver health for the sink location (object store upload errors, NFS issues)
After failures, delete partial files before retrying
Keep polars/pyiceberg updated; reproducible mismatches are sink bugs

When it happens

Trigger: During `_data_file_from_sink_metadata`, when `parquet_metadata.num_rows != num_rows` for a file produced by `sink_iceberg`. Typically caused by a bug, a truncated/overwritten file on the object store, or concurrent writers touching the same path.

Common situations: Concurrent `sink_iceberg` jobs writing to the same file path; object-store flakiness truncating uploads; interrupted writes leaving partial files that a later run re-opens; filesystem/driver issues when sinking to local or NFS mounts.

Related errors

AI-assisted analysis of pola-rs/polars@fc24390824 (2026-09-02). Data as JSON: /api/errors/b40747d970e9f8db. Report an issue: GitHub.