apache/iceberg · error · UncheckedIOException

Fail to deserialize aggregated statistics

Error message

Fail to deserialize aggregated statistics

What it means

StatisticsUtil.deserializeCompletedStatistics's final fallback: after retrying deserialization with sort-key serializer version 1, any remaining IOException becomes UncheckedIOException "Fail to deserialize aggregated statistics". This means the bytes are unreadable with both the latest and the v1 serializer — the payload is corrupt, truncated, or from an unrecognized format.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:97

      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);
      if (!completedStatistics.isValid()) {
        throw new RuntimeException("Fail to deserialize aggregated statistics,change to v1");
      }

      return completedStatistics;
    } catch (Exception e) {
      try {
        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly
        // parse the checkpointData, so we need to first switch the version to v1. Once the state
        // data is successfully parsed, we need to switch the serialization version to the latest
        // version to parse the subsequent data passed from the TM.
        statisticsSerializer.changeSortKeySerializerVersion(1);
        DataInputDeserializer input = new DataInputDeserializer(bytes);
        CompletedStatistics deserialize = statisticsSerializer.deserialize(input);
        statisticsSerializer.changeSortKeySerializerVersionLatest();
        return deserialize;
      } catch (IOException ioException) {
        throw new UncheckedIOException("Fail to deserialize aggregated statistics", ioException);
      }
    }
  }

  static byte[] serializeGlobalStatistics(
      GlobalStatistics globalStatistics, TypeSerializer<GlobalStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(globalStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static GlobalStatistics deserializeGlobalStatistics(
      byte[] bytes, TypeSerializer<GlobalStatistics> statisticsSerializer) {
    try {

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Verify the byte array actually contains completed statistics from the same serializer family (not global/other statistics).
  2. Restore from an intact checkpoint/savepoint; discard the corrupted state file.
  3. If corruption is systemic, disable restore and start fresh — sort-key statistics are advisory and rebuilt automatically by the sink.
Defensive patterns

Strategy: try-catch

Validate before calling

if (bytes == null || bytes.length == 0) { throw new IllegalArgumentException("No statistics bytes"); }

Try / catch

try {
  stats = StatisticsUtil.deserializeCompletedStatistics(bytes, serializer);
} catch (UncheckedIOException e) {
  LOG.warn("Statistics unreadable in both serializer versions; discarding", e);
  stats = null; // sink rebuilds statistics
}

Prevention

When it happens

Trigger: deserializeCompletedStatistics where the v1-fallback path's statisticsSerializer.deserialize throws IOException, e.g. truncated checkpoint bytes, wrong byte array passed in, or bytes from a format neither serializer version handles.

Common situations: Corrupted savepoint/checkpoint files; passing statistics bytes to the wrong deserializer (e.g. GlobalStatistics bytes fed to CompletedStatisticsSerializer); hand-migrated state.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/273f03546119d7d2. Report an issue: GitHub.