apache/iceberg · error · UncheckedIOException

Fail to deserialize data statistics

Error message

Fail to deserialize data statistics

What it means

StatisticsUtil.deserializeDataStatistics wraps IOException from the TypeSerializer when deserializing a byte[] back into a DataStatistics object. It is thrown as UncheckedIOException because the input bytes could not be interpreted by the given serializer, typically indicating corrupt or incompatible data.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:59

  static byte[] serializeDataStatistics(
      DataStatistics dataStatistics, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataOutputSerializer out = new DataOutputSerializer(64);
    try {
      statisticsSerializer.serialize(dataStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize data statistics", e);
    }
  }

  static DataStatistics deserializeDataStatistics(
      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);
    try {
      return statisticsSerializer.deserialize(input);
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to deserialize data statistics", e);
    }
  }

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(completedStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static CompletedStatistics deserializeCompletedStatistics(
      byte[] bytes, CompletedStatisticsSerializer statisticsSerializer) {
    try {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Ensure the same TypeSerializer version that produced the bytes is used for deserialization (align Iceberg/Flink versions across the job).
2. Inspect e.getCause() for the exact deserialization offset failure to confirm byte corruption or version drift.
3. If restoring from an old savepoint, use the versioned migration path (CompletedStatisticsSerializer.changeSortKeySerializerVersion) or discard incompatible state.
4. Regenerate statistics by restarting the shuffle stage without restored state.

if (bytes == null || bytes.length == 0) throw new IllegalArgumentException("statistics bytes are empty");

try {
  DataStatistics stats = StatisticsUtil.deserializeDataStatistics(bytes, serializer);
} catch (UncheckedIOException e) {
  LOG.error("statistics deserialization failed; check version alignment", e.getCause());
  throw e;
}

When it happens

Trigger: Calling StatisticsUtil.deserializeDataStatistics(bytes, serializer) where the bytes were produced by a different serializer version or truncated/corrupted, so statisticsSerializer.deserialize(input) throws IOException.

Common situations: Restoring a Flink job checkpoint/savepoint from an older or newer Iceberg version; manually patched or truncated state bytes; network channel data mismatch after upgrade.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• Failed to deserialize sort key sketch
• Fail to deserialize data statistics
• Fail to deserialize aggregated statistics
• Failed to decode partition
• Failed to deserialize sort key sketch

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f4e5fda9f8a002a7. Report an issue: GitHub.