apache/iceberg · error · UncheckedIOException

Fail to deserialize aggregated statistics

Error message

Fail to deserialize aggregated statistics

What it means

StatisticsUtil.deserializeCompletedStatistics throws this UncheckedIOException when even the v1 fallback deserialization (after changeSortKeySerializerVersion(1)) fails with an IOException, meaning the bytes are unreadable under any known statistics format version.

Solutions

1. Treat the state as unrecoverable: discard the checkpoint/savepoint and restart the job to regenerate shuffle statistics.
2. Verify checkpoint storage integrity (DFS corruption, incomplete upload) and check TaskManager logs for the original IOException cause.
3. Ensure all nodes run the identical iceberg-flink-runtime version; a mixed cluster can produce unreadable aggregates.
4. If the issue follows an upgrade, downgrade to the version that wrote the state, then migrate forward cleanly.

// verify checkpoint storage integrity before restore
FileSystem fs = FileSystem.get(checkpointUri, conf);
if (!fs.exists(new Path(checkpointUri, "_metadata"))) throw new IllegalStateException("incomplete checkpoint");

try {
  CompletedStatistics s = StatisticsUtil.deserializeCompletedStatistics(bytes, serializer);
} catch (UncheckedIOException e) {
  LOG.error("statistics unreadable under any format version; regenerating", e.getCause());
  // fall back to empty statistics / restart without state
}

When it happens

Trigger: Calling StatisticsUtil.deserializeCompletedStatistics where both the latest-version attempt (invalid result) and the v1 retry path throw IOException — bytes are corrupt or from an incompatible/unrecognized format.

Common situations: Truncated or corrupted checkpoint data; statistics bytes produced by a much older or newer format with no compatible reader; manual state surgery.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• Fail to deserialize aggregated statistics
• Fail to deserialize data statistics
• Fail to deserialize data statistics
• Failed to decode partition
• Failed to deserialize sort key sketch

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/3c7f7342c31e7d25. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:97

      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);
      if (!completedStatistics.isValid()) {
        throw new RuntimeException("Fail to deserialize aggregated statistics,change to v1");
      }

      return completedStatistics;
    } catch (Exception e) {
      try {
        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly
        // parse the checkpointData, so we need to first switch the version to v1. Once the state
        // data is successfully parsed, we need to switch the serialization version to the latest
        // version to parse the subsequent data passed from the TM.
        statisticsSerializer.changeSortKeySerializerVersion(1);
        DataInputDeserializer input = new DataInputDeserializer(bytes);
        CompletedStatistics deserialize = statisticsSerializer.deserialize(input);
        statisticsSerializer.changeSortKeySerializerVersionLatest();
        return deserialize;
      } catch (IOException ioException) {
        throw new UncheckedIOException("Fail to deserialize aggregated statistics", ioException);
      }
    }
  }

  static byte[] serializeGlobalStatistics(
      GlobalStatistics globalStatistics, TypeSerializer<GlobalStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(globalStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static GlobalStatistics deserializeGlobalStatistics(
      byte[] bytes, TypeSerializer<GlobalStatistics> statisticsSerializer) {
    try {

View on GitHub (pinned to 86d9c8fc54)