apache/iceberg · error · UncheckedIOException

Fail to serialize aggregated statistics

Error message

Fail to serialize aggregated statistics

What it means

Thrown by StatisticsUtil.serializeCompletedStatistics when serializing a CompletedStatistics (aggregated statistics from the coordinator) fails with an IOException. Completed statistics are persisted with a 1024-byte initial DataOutputSerializer and handed to state backends / committed metadata. The library wraps the IO failure as an UncheckedIOException.

Solutions

  1. Ensure the CompletedStatisticsSerializer variant matches the statistics version produced by the aggregator.
  2. Align Iceberg versions between checkpoint writing and restoring jobs.
  3. Verify the completedStatistics object is non-null and valid (isValid()) before serializing.

Example fix

// before
byte[] bytes = StatisticsUtil.serializeCompletedStatistics(completed, statisticsSerializer);
// after
if (completed == null || !completed.isValid()) {
  throw new IllegalStateException("Completed statistics missing or invalid");
}
byte[] bytes = StatisticsUtil.serializeCompletedStatistics(completed, statisticsSerializer);
Defensive patterns

Strategy: validation

Validate before calling

if (completedStatistics == null || !completedStatistics.isValid()) { throw new IllegalStateException("Completed statistics invalid before serialize"); }

Type guard

boolean canSerialize(CompletedStatistics cs) { return cs != null && cs.isValid(); }

Try / catch

try {
  byte[] bytes = StatisticsUtil.serializeCompletedStatistics(completed, serializer);
} catch (UncheckedIOException e) {
  LOG.error("Failed to persist aggregated statistics", e);
  throw e;
}

Prevention

When it happens

Trigger: Calling serializeCompletedStatistics with the wrong CompletedStatisticsSerializer for the concrete statistics class, or an internal serializer IOException while writing aggregated sketch/map statistics.

Common situations: Coordinator persisting aggregated statistics at checkpoint time; mismatched serializer after upgrading Iceberg versions where the statistics layout changed.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/4a62e7e2d9384347. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:71

  static DataStatistics deserializeDataStatistics(
      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);
    try {
      return statisticsSerializer.deserialize(input);
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to deserialize data statistics", e);
    }
  }

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(completedStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static CompletedStatistics deserializeCompletedStatistics(
      byte[] bytes, CompletedStatisticsSerializer statisticsSerializer) {
    try {
      DataInputDeserializer input = new DataInputDeserializer(bytes);
      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);
      if (!completedStatistics.isValid()) {
        throw new RuntimeException("Fail to deserialize aggregated statistics,change to v1");
      }

      return completedStatistics;
    } catch (Exception e) {
      try {
        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly
        // parse the checkpointData, so we need to first switch the version to v1. Once the state
        // data is successfully parsed, we need to switch the serialization version to the latest

View on GitHub (pinned to 86d9c8fc54)