apache/iceberg · error · UncheckedIOException

Fail to deserialize data statistics

Error message

Fail to deserialize data statistics

What it means

Thrown by StatisticsUtil.deserializeDataStatistics when bytes cannot be decoded back into a DataStatistics via the given TypeSerializer; the IOException is wrapped as UncheckedIOException. This occurs when reading shuffle statistics persisted in Flink state or transmitted between operators. Mismatched serializer versions or corrupted bytes cause it.

Solutions

  1. Restore the job with the same Iceberg version that created the state, or migrate state deliberately.
  2. Confirm the serializer passed matches the one used at serialization time (statistics type must match).
  3. Discard stale state and restart the job so statistics are re-collected from scratch.

Example fix

// before: bytes from old job version
DataStatistics stats = StatisticsUtil.deserializeDataStatistics(bytes, serializer);
// after: null/empty guard and version-validated path
if (bytes == null || bytes.length == 0) return DataStatistics.empty();
DataStatistics stats = StatisticsUtil.deserializeDataStatistics(bytes, serializer);
Defensive patterns

Strategy: validation

Validate before calling

if (bytes == null || bytes.length == 0) { return null; } // skip empty payloads before deserializing

Type guard

boolean hasPayload(byte[] bytes) { return bytes != null && bytes.length > 0; }

Try / catch

try {
  DataStatistics s = StatisticsUtil.deserializeDataStatistics(bytes, serializer);
} catch (UncheckedIOException e) {
  LOG.warn("Cannot decode statistics; treating as empty", e);
  DataStatistics s = DataStatistics.empty();
}

Prevention

When it happens

Trigger: Calling deserializeDataStatistics on bytes produced by a different serializer version, truncated byte arrays, or deserializing statistics written by another Iceberg release.

Common situations: Job restore from checkpoints across Iceberg upgrades, mixing serializer versions between writer and reader subtasks during a rolling upgrade.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6c864b1765c02f3a. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:59

  static byte[] serializeDataStatistics(
      DataStatistics dataStatistics, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataOutputSerializer out = new DataOutputSerializer(64);
    try {
      statisticsSerializer.serialize(dataStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize data statistics", e);
    }
  }

  static DataStatistics deserializeDataStatistics(
      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);
    try {
      return statisticsSerializer.deserialize(input);
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to deserialize data statistics", e);
    }
  }

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(completedStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static CompletedStatistics deserializeCompletedStatistics(
      byte[] bytes, CompletedStatisticsSerializer statisticsSerializer) {
    try {

View on GitHub (pinned to 86d9c8fc54)