apache/iceberg · error · UncheckedIOException

Fail to serialize aggregated statistics

Error message

Fail to serialize aggregated statistics

What it means

StatisticsUtil.serializeCompletedStatistics wraps IOException thrown while serializing a CompletedStatistics object (aggregated shuffle statistics sent from task to coordinator) into an UncheckedIOException. It signals that the CompletedStatisticsSerializer failed to write the aggregate payload.

Solutions

  1. Confirm CompletedStatistics and its inner DataStatistics match the serializer version (Sketch vs Map, sort key version).
  2. Align iceberg-flink-runtime version on all task and job manager nodes.
  3. Get the root cause from e.getCause() and fix or upgrade the serializer.
  4. Fall back to StatisticsType.Map statistics if sketch serialization keeps failing after version changes.
Defensive patterns

Strategy: try-catch

Validate before calling

Preconditions.checkArgument(completedStatistics != null, "completedStatistics must not be null");
Preconditions.checkArgument(statisticsSerializer != null, "statisticsSerializer must not be null");

Try / catch

try {
  byte[] bytes = StatisticsUtil.serializeCompletedStatistics(stats, serializer);
} catch (UncheckedIOException e) {
  LOG.error("completed statistics serialization failed", e.getCause());
  throw e;
}

Prevention

When it happens

Trigger: Calling StatisticsUtil.serializeCompletedStatistics(completedStatistics, statisticsSerializer) when statisticsSerializer.serialize() throws IOException, e.g. serializer version mismatch with the contained statistics entries or buffer allocation failure.

Common situations: Coordinator/task communication after an Iceberg upgrade where the SortKeySerializer version no longer matches; custom serializers not registered on all workers.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/e1572a028371d795. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:71

  static DataStatistics deserializeDataStatistics(
      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);
    try {
      return statisticsSerializer.deserialize(input);
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to deserialize data statistics", e);
    }
  }

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(completedStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static CompletedStatistics deserializeCompletedStatistics(
      byte[] bytes, CompletedStatisticsSerializer statisticsSerializer) {
    try {
      DataInputDeserializer input = new DataInputDeserializer(bytes);
      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);
      if (!completedStatistics.isValid()) {
        throw new RuntimeException("Fail to deserialize aggregated statistics,change to v1");
      }

      return completedStatistics;
    } catch (Exception e) {
      try {
        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly
        // parse the checkpointData, so we need to first switch the version to v1. Once the state
        // data is successfully parsed, we need to switch the serialization version to the latest

View on GitHub (pinned to 86d9c8fc54)