apache/iceberg · warning · RuntimeException

Fail to deserialize aggregated statistics,change to v1

Error message

Fail to deserialize aggregated statistics,change to v1

What it means

StatisticsUtil.deserializeCompletedStatistics throws a plain RuntimeException "Fail to deserialize aggregated statistics,change to v1" when bytes deserialized into a CompletedStatistics that reports !isValid(), signalling the payload was not written with the current (latest) sort-key serializer version. The catch block then retries the deserialization with sort key serializer version 1, so this exception is an internal control-flow signal; if it escapes, the fallback also failed.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:81

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(completedStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static CompletedStatistics deserializeCompletedStatistics(
      byte[] bytes, CompletedStatisticsSerializer statisticsSerializer) {
    try {
      DataInputDeserializer input = new DataInputDeserializer(bytes);
      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);
      if (!completedStatistics.isValid()) {
        throw new RuntimeException("Fail to deserialize aggregated statistics,change to v1");
      }

      return completedStatistics;
    } catch (Exception e) {
      try {
        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly
        // parse the checkpointData, so we need to first switch the version to v1. Once the state
        // data is successfully parsed, we need to switch the serialization version to the latest
        // version to parse the subsequent data passed from the TM.
        statisticsSerializer.changeSortKeySerializerVersion(1);
        DataInputDeserializer input = new DataInputDeserializer(bytes);
        CompletedStatistics deserialize = statisticsSerializer.deserialize(input);
        statisticsSerializer.changeSortKeySerializerVersionLatest();
        return deserialize;
      } catch (IOException ioException) {
        throw new UncheckedIOException("Fail to deserialize aggregated statistics", ioException);
      }
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. This should be transparently handled by the internal v1 fallback — if you see it escape, check that you are calling StatisticsUtil.deserializeCompletedStatistics (not custom deserialization) so the fallback path runs.
2. Upgrade both job stages to the same iceberg-flink-runtime version so v2 serialization is used end-to-end.
3. Start from a fresh checkpoint if the fallback also fails; statistics are advisory and will be rebuilt.

Example fix

// before: custom code without fallback
CompletedStatistics s = completedSerializer.deserialize(new DataInputDeserializer(bytes));
// after: use version-aware util
CompletedStatistics s = StatisticsUtil.deserializeCompletedStatistics(bytes, completedSerializer);

// Ensure bytes came from StatisticsUtil.serializeCompletedStatistics of a compatible version
if (bytes == null || bytes.length == 0) { return null; }

try {
  stats = StatisticsUtil.deserializeCompletedStatistics(bytes, serializer);
} catch (RuntimeException | UncheckedIOException e) {
  // v1 fallback failed too; rebuild statistics from scratch
  stats = null;
}

When it happens

Trigger: Deserializing completed statistics whose bytes were produced with the v1 SortKeySerializer (older Iceberg checkpoint) while the current job uses v2 — the validity check fails and the v1-fallback marker is thrown; if surrounding code doesn't catch it, state restoration aborts.

Common situations: Upgrading iceberg-flink-runtime while restoring from a savepoint written before the SortKeySerializer version bump; mixing old and new operator states during rescale.

Related errors

• Fail to deserialize aggregated statistics,change to v1
• Failed to deserialize sort key sketch
• Fail to deserialize data statistics
• Fail to deserialize aggregated statistics
• Failed to decode partition

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/2a2989252c53a6c3. Report an issue: GitHub.