apache/iceberg · error · UncheckedIOException

Fail to serialize data statistics

Error message

Fail to serialize data statistics

What it means

Thrown by StatisticsUtil.serializeDataStatistics when the Flink TypeSerializer cannot serialize a DataStatistics object to bytes, wrapping the IOException in an UncheckedIOException. Data statistics are serialized to be carried through Flink accumulators/state during the shuffle-based sort. Any IO failure in the serializer's internal write path triggers this.

Solutions

1. Verify the TypeSerializer passed in matches the concrete DataStatistics implementation being serialized.
2. Upgrade to a fixed Iceberg version if the serializer itself is buggy (check issue tracker).
3. Add a guard that statistics are non-null and of the expected class before calling serializeDataStatistics.

Example fix

// before
byte[] bytes = StatisticsUtil.serializeDataStatistics(stats, new LocalStatisticsSerializer());
// after
Preconditions.checkArgument(stats instanceof LocalStatistics,
    "Expected LocalStatistics, got %s", stats.getClass());
byte[] bytes = StatisticsUtil.serializeDataStatistics(stats, new LocalStatisticsSerializer());

if (dataStatistics == null || statisticsSerializer == null) { throw new IllegalArgumentException("statistics and serializer required"); }

boolean isSerializable(DataStatistics s, TypeSerializer<DataStatistics> ser) { return s != null && ser != null; }

try {
  byte[] bytes = StatisticsUtil.serializeDataStatistics(stats, serializer);
} catch (UncheckedIOException e) {
  LOG.error("Statistics serialization failed", e);
  throw e; // statistics loss is not recoverable locally
}

When it happens

Trigger: Calling serializeDataStatistics with a DataStatistics whose underlying serializer (e.g. the map-based LocalStatisticsSerializer) hits an IOException, typically from DataOutputSerializer write failures or a null/incompatible statistics object.

Common situations: Passing a DataStatistics of a type not matching the supplied TypeSerializer after a code change, or serializer bugs when emitting shuffle statistics from a downstream subtask to the coordinator.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Fail to deserialize data statistics
• Fail to serialize aggregated statistics
• Fail to serialize aggregated statistics
• Fail to serialize aggregated statistics
• Fail to serialize data statistics

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0cc9710bb81745f6. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:49

  static DataStatistics createTaskStatistics(
      StatisticsType type, int operatorParallelism, int numPartitions) {
    if (type == StatisticsType.Map) {
      return new MapDataStatistics();
    } else {
      return new SketchDataStatistics(
          SketchUtil.determineOperatorReservoirSize(operatorParallelism, numPartitions));
    }
  }

  static byte[] serializeDataStatistics(
      DataStatistics dataStatistics, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataOutputSerializer out = new DataOutputSerializer(64);
    try {
      statisticsSerializer.serialize(dataStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize data statistics", e);
    }
  }

  static DataStatistics deserializeDataStatistics(
      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);
    try {
      return statisticsSerializer.deserialize(input);
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to deserialize data statistics", e);
    }
  }

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);

View on GitHub (pinned to 86d9c8fc54)