apache/iceberg · error · UncheckedIOException

Fail to serialize data statistics

Error message

Fail to serialize data statistics

What it means

StatisticsUtil.serializeDataStatistics serializes a DataStatistics object through the provided Flink TypeSerializer into a byte array; any IOException becomes UncheckedIOException "Fail to serialize data statistics". It is a low-level utility used when shipping sort-key statistics through Flink operator state/broadcast.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:49

  static DataStatistics createTaskStatistics(
      StatisticsType type, int operatorParallelism, int numPartitions) {
    if (type == StatisticsType.Map) {
      return new MapDataStatistics();
    } else {
      return new SketchDataStatistics(
          SketchUtil.determineOperatorReservoirSize(operatorParallelism, numPartitions));
    }
  }

  static byte[] serializeDataStatistics(
      DataStatistics dataStatistics, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataOutputSerializer out = new DataOutputSerializer(64);
    try {
      statisticsSerializer.serialize(dataStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize data statistics", e);
    }
  }

  static DataStatistics deserializeDataStatistics(
      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);
    try {
      return statisticsSerializer.deserialize(input);
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to deserialize data statistics", e);
    }
  }

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check the IOException cause; increase task-manager heap/network memory if it is a buffer failure.
2. Ensure the serializer passed matches the DataStatistics implementation (SortKeyStatisticsSerializer for sort-key stats).
3. Retry the job; transient buffer issues usually do not recur after backpressure eases.

Preconditions.checkNotNull(dataStatistics, "dataStatistics must not be null");
Preconditions.checkNotNull(statisticsSerializer, "statisticsSerializer must not be null");

try {
  byte[] out = StatisticsUtil.serializeDataStatistics(stats, serializer);
} catch (UncheckedIOException e) {
  LOG.error("Failed to serialize data statistics", e.getCause());
}

When it happens

Trigger: Calling StatisticsUtil.serializeDataStatistics when the underlying TypeSerializer.serialize throws IOException — essentially only from stream/buffer failures or a malformed statistics object.

Common situations: Task-manager memory pressure during shuffle statistics collection; a custom/incorrect statistics serializer passed in.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Fail to serialize aggregated statistics
• Failed to serialize sort key
• Unsupported data statistics type: ${statisticsType}
• Failed to serialize sort key
• Fail to deserialize aggregated statistics

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/cb831bc5d3398241. Report an issue: GitHub.