{"record":{"id":"b40747d970e9f8db","repo":"pola-rs/polars","slug":"native-sink-row-count-num-rows-does-not-match-pa","errorCode":null,"errorMessage":"native sink row count {num_rows} does not match Parquet metadata row count {parquet_metadata.num_rows} for '{file_path}'","messagePattern":"native sink row count (.+?) does not match Parquet metadata row count (.+?) for '(.+?)'","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"py-polars/src/polars/io/iceberg/_sink.py","lineNumber":177,"sourceCode":"    import pyarrow as pa\n    import pyarrow.parquet as pq\n    from pyiceberg.io.pyarrow import (\n        MetricModeTypes,\n        _check_pyarrow_schema_compatible,\n        data_file_statistics_from_parquet_metadata,\n    )\n    from pyiceberg.manifest import DataFile, DataFileContent, FileFormat\n\n    schema = table_metadata.schema()\n    file_path, num_rows, num_bytes, parquet_metadata_bytes = sinked_file\n    parquet_metadata = pq.read_metadata(pa.BufferReader(parquet_metadata_bytes))\n\n    if parquet_metadata.num_rows != num_rows:\n        msg = (\n            f\"native sink row count {num_rows} does not match Parquet metadata \"\n            f\"row count {parquet_metadata.num_rows} for '{file_path}'\"\n        )\n        raise ValueError(msg)\n\n    _check_pyarrow_schema_compatible(schema, parquet_metadata.schema.to_arrow_schema())\n    statistics = data_file_statistics_from_parquet_metadata(\n        parquet_metadata=parquet_metadata,\n        stats_columns=statistics_plan,\n        parquet_column_mapping=parquet_column_mapping,\n    )\n    partition = _infer_partition_from_statistics(\n        statistics, table_metadata.spec(), schema\n    )\n    serialized_statistics = statistics.to_serialized_dict()\n    for source_id, metrics_mode in nested_metrics_modes.items():\n        serialized_statistics[\"lower_bounds\"].pop(source_id, None)\n        serialized_statistics[\"upper_bounds\"].pop(source_id, None)\n        if metrics_mode is MetricModeTypes.NONE:\n            serialized_statistics[\"value_counts\"].pop(source_id, None)\n            serialized_statistics[\"null_value_counts\"].pop(source_id, None)\n            serialized_statistics[\"nan_value_counts\"].pop(source_id, None)","sourceCodeStart":159,"sourceCodeEnd":195,"githubUrl":"https://github.com/pola-rs/polars/blob/fc24390824009e8fc5b7870e256273c1b140a019/py-polars/src/polars/io/iceberg/_sink.py#L159-L195","documentation":"After the native sink writes a Parquet file, polars cross-checks the row count it wrote (`num_rows`) against the row count reported by the Parquet file's footer metadata. A mismatch means the physical file does not match what the sink engine reported — a corruption/integrity failure in the output pipeline — so it raises ValueError instead of committing bad Iceberg DataFile metadata.","triggerScenarios":"During `_data_file_from_sink_metadata`, when `parquet_metadata.num_rows != num_rows` for a file produced by `sink_iceberg`. Typically caused by a bug, a truncated/overwritten file on the object store, or concurrent writers touching the same path.","commonSituations":"Concurrent `sink_iceberg` jobs writing to the same file path; object-store flakiness truncating uploads; interrupted writes leaving partial files that a later run re-opens; filesystem/driver issues when sinking to local or NFS mounts.","solutions":["Retry the sink with a fresh, unique file location to rule out a corrupted partial file","Ensure only one writer targets each output path — use unique paths or rely on the sink's default naming, never share paths across concurrent jobs","Inspect the offending Parquet file (read its footer metadata) to confirm the row count discrepancy and check storage/driver logs for upload errors","Update polars and pyiceberg; if reproducible on a single version, report it as a sink-integrity bug"],"exampleFix":"// before: concurrent jobs overwrite the same target path\nlf.sink_iceberg(table)\n// after: run one sink per unique target, no concurrent writers to the same path\n# serialize writes or give each job its own location\nlf.sink_iceberg(table)  # single writer for this table path","handlingStrategy":"validation","validationCode":"import pyarrow.parquet as pq\nmd = pq.read_metadata(file_path)\nassert md.num_rows == expected_rows, f\"row count mismatch: {md.num_rows} vs {expected_rows}\"","typeGuard":null,"tryCatchPattern":"try:\n    lf.sink_iceberg(table)\nexcept ValueError as e:\n    if 'does not match Parquet metadata row count' in str(e):\n        remove_or_quarantine_partial_files(target_dir)\n        lf.sink_iceberg(table)  # retry with clean, unique paths\n    else:\n        raise","preventionTips":["Never let concurrent writers target the same output file path","Check storage/driver health for the sink location (object store upload errors, NFS issues)","After failures, delete partial files before retrying","Keep polars/pyiceberg updated; reproducible mismatches are sink bugs"],"tags":["iceberg","data-integrity","parquet","polars"],"backgroundTag":"row-count-mismatch","analyzedSha":"fc24390824009e8fc5b7870e256273c1b140a019","analyzedAt":"2026-09-02T20:08:03.494Z","contentChangedAt":"2026-09-02T20:08:03.494Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}