apache/iceberg · error · UncheckedIOException

Fail to serialize aggregated statistics

Error message

Fail to serialize aggregated statistics

What it means

StatisticsUtil.serializeCompletedStatistics serializes aggregated (completed) statistics via the provided serializer into bytes; IOException becomes UncheckedIOException "Fail to serialize aggregated statistics". Used when persisting/broadcasting final sort-key statistics after shuffle aggregation.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:71

  static DataStatistics deserializeDataStatistics(
      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);
    try {
      return statisticsSerializer.deserialize(input);
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to deserialize data statistics", e);
    }
  }

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);
      statisticsSerializer.serialize(completedStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize aggregated statistics", e);
    }
  }

  static CompletedStatistics deserializeCompletedStatistics(
      byte[] bytes, CompletedStatisticsSerializer statisticsSerializer) {
    try {
      DataInputDeserializer input = new DataInputDeserializer(bytes);
      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);
      if (!completedStatistics.isValid()) {
        throw new RuntimeException("Fail to deserialize aggregated statistics,change to v1");
      }

      return completedStatistics;
    } catch (Exception e) {
      try {
        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly
        // parse the checkpointData, so we need to first switch the version to v1. Once the state
        // data is successfully parsed, we need to switch the serialization version to the latest

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check the cause; increase task-manager memory or reduce sketch size/num samples configured for the sink.
2. Confirm the CompletedStatistics object is fully initialized (non-null sort key serializer) before serialization.
3. Retry after transient backpressure resolves.

Preconditions.checkNotNull(completedStatistics, "completedStatistics must not be null");

try {
  byte[] out = StatisticsUtil.serializeCompletedStatistics(stats, serializer);
} catch (UncheckedIOException e) {
  LOG.error("Failed to serialize aggregated statistics", e.getCause());
}

When it happens

Trigger: Calling serializeCompletedStatistics when the underlying CompletedStatisticsSerializer.serialize throws IOException — usually buffer growth failure or memory pressure, since the flow is in-memory.

Common situations: Very large sketches (many sort-key distributions) exhausting the DataOutputSerializer buffer under constrained task-manager memory.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Fail to serialize data statistics
• Failed to serialize sort key
• Unsupported data statistics type: ${statisticsType}
• Failed to serialize sort key
• Fail to deserialize aggregated statistics

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b663cc556896301f. Report an issue: GitHub.