apache/iceberg · error · RuntimeException

Fail to deserialize aggregated statistics,change to v1

Error message

Fail to deserialize aggregated statistics,change to v1

What it means

Thrown by StatisticsUtil.deserializeCompletedStatistics when the deserialized CompletedStatistics fails its isValid() check, indicating the payload was written by a newer serializer version (e.g. v2 sort keys) that this code path cannot interpret as valid. It signals fallback to v1 serialization is needed; the same message is also used when falling back. As a RuntimeException it propagates when the v1 fallback itself cannot be prepared.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:81

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(completedStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static CompletedStatistics deserializeCompletedStatistics(
      byte[] bytes, CompletedStatisticsSerializer statisticsSerializer) {
    try {
      DataInputDeserializer input = new DataInputDeserializer(bytes);
      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);
      if (!completedStatistics.isValid()) {
        throw new RuntimeException("Fail to deserialize aggregated statistics,change to v1");
      }

      return completedStatistics;
    } catch (Exception e) {
      try {
        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly
        // parse the checkpointData, so we need to first switch the version to v1. Once the state
        // data is successfully parsed, we need to switch the serialization version to the latest
        // version to parse the subsequent data passed from the TM.
        statisticsSerializer.changeSortKeySerializerVersion(1);
        DataInputDeserializer input = new DataInputDeserializer(bytes);
        CompletedStatistics deserialize = statisticsSerializer.deserialize(input);
        statisticsSerializer.changeSortKeySerializerVersionLatest();
        return deserialize;
      } catch (IOException ioException) {
        throw new UncheckedIOException("Fail to deserialize aggregated statistics", ioException);
      }
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Let the built-in fallback run: it retries with statisticsSerializer.changeSortKeySerializerVersion(1) and restores the latest version afterwards.
2. If this RuntimeException escapes, the v1 fallback also failed — restore with the original Iceberg version that wrote the checkpoint.
3. Migrate state by draining the job with the old version and restarting clean with the new version.

try {
  CompletedStatistics s = StatisticsUtil.deserializeCompletedStatistics(bytes, serializer);
} catch (RuntimeException e) {
  // both current and v1 fallback failed — restore state with the original Iceberg version
  LOG.error("Aggregated statistics unreadable in any version", e);
}

When it happens

Trigger: Restoring aggregated statistics bytes written with the latest sort-key serializer version into a job whose current serializer version disagrees, so the deserialized object is invalid; failures inside the primary deserialize path caught by the generic catch(Exception).

Common situations: Flink job upgrade where the checkpointed aggregated statistics were written with v2 sort-key serialization but restored with v1 settings, or rolling upgrades mixing task versions.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• The Avro schema is not a nullable type: ${schema}
• Fail to serialize at field: %s.
• Unrecognized version or corrupt state: <version>
• Failed to serialize PK index key
• Failed to encode partition

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/726084d16cf7f40e. Report an issue: GitHub.