{"record":{"id":"1627b443926d284a","repo":"apache/iceberg","slug":"using-full-compute-as-previous-statistics-file-is","errorCode":null,"errorMessage":"Using full compute as previous statistics file is corrupted for incremental compute.","messagePattern":"Using full compute as previous statistics file is corrupted for incremental compute\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java","lineNumber":122,"sourceCode":"\n    Collection<PartitionStatistics> stats;\n    PartitionStatisticsFile statisticsFile = latestStatsFile(table, snapshot.snapshotId());\n    if (statisticsFile == null) {\n      LOG.info(\n          \"Using full compute as previous statistics file is not present for incremental compute.\");\n      stats =\n          computeStats(table, snapshot.allManifests(table.io()), false /* incremental */).values();\n    } else {\n      if (statisticsFile.snapshotId() == snapshotId) {\n        // no-op\n        LOG.info(\"Returning existing statistics file for snapshot {}\", snapshotId);\n        return statisticsFile;\n      }\n\n      try {\n        stats = computeAndMergeStatsIncremental(table, snapshot, statisticsFile.snapshotId());\n      } catch (InvalidStatsFileException exception) {\n        LOG.warn(\n            \"Using full compute as previous statistics file is corrupted for incremental compute.\");\n        stats =\n            computeStats(table, snapshot.allManifests(table.io()), false /* incremental */)\n                .values();\n      }\n    }\n\n    if (stats.isEmpty()) {\n      // empty branch case\n      return null;\n    }\n\n    StructType partitionType = Partitioning.partitionType(table);\n    List<PartitionStatistics> sortedStats = sortStatsByPartition(stats, partitionType);\n    return writePartitionStatsFile(\n        table,\n        snapshot.snapshotId(),\n        PartitionStatistics.schema(partitionType, TableUtil.formatVersion(table)),","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java#L104-L140","documentation":"PartitionStatsHandler.computeAndWriteStatsFile tries to compute partition statistics incrementally by merging the previously written statistics file. If the previous statistics file is invalid (InvalidStatsFileException, e.g. unreadable or corrupt), it falls back to a full recompute from all manifests and logs this warning. The operation still succeeds; only efficiency is lost.","triggerScenarios":"Calling computeAndWriteStatsFile (or via Spark procedures writing partition stats) when the statistics file referenced by the previous snapshot's statistics list is corrupt, truncated, written by a buggy/older writer, or missing required content.","commonSituations":"Statistics files lost or truncated by incomplete uploads to object storage, metadata files copied between buckets without RewriteTablePathUtil, tables written by an older Iceberg version whose stats format differs, and interrupted write jobs that left a partial Parquet stats file registered.","solutions":["Let the fallback run: the partition stats file is recomputed fully and the next incremental compute will use the new valid file.","Check the previous statistics file (snapshot.statisticsFiles(io)) for readability/corruption and delete stale entries via removeStatistics / expireSnapshots.","Rewrite all historical statistics files with a full compute once to restore incremental merging.","Upgrade Iceberg if the corrupt file was produced by an older writer version."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"StatisticsFile prev = latestStatisticsFile(table, prevSnapshotId);\nboolean usable = prev != null && table.io().newInputFile(prev.path()).exists();","typeGuard":null,"tryCatchPattern":"try { computeIncremental(...); } catch (InvalidStatsFileException e) { computeFull(...); }","preventionTips":["Ensure stats file uploads complete atomically","Recompute full statistics after upgrading writer versions","Never copy statistics files without also fixing their paths"],"tags":["statistics","partition-stats","fallback","corruption"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}