apache/iceberg · error · UncheckedIOException

Fail to serialize data statistics

Error message

Fail to serialize data statistics

What it means

StatisticsUtil.serializeDataStatistics wraps any IOException thrown while serializing a DataStatistics object (used in Flink key shuffle downsketching) into an UncheckedIOException. This is an internal failure of the Flink TypeSerializer for the DataStatistics type; the library throws it because serialization failures cannot propagate as checked IOExceptions through non-declaring call sites.

Solutions

  1. Verify the TypeSerializer<DataStatistics> passed in matches the DataStatistics implementation currently in use (SketchDataStatistics vs MapDataStatistics) and its version.
  2. Check for Flink/Iceberg version mismatch between job manager and task manager classpaths; align all nodes to the same iceberg-flink-runtime version.
  3. Reproduce the wrapped cause via e.getCause() to see the underlying IOException and fix the serializer accordingly.
  4. If a job restore triggered it, restart the job without state or migrate state with a matching serializer version.

Example fix

// before
byte[] bytes = StatisticsUtil.serializeDataStatistics(stats, wrongSerializer);
// after
TypeSerializer<DataStatistics> serializer =
    new DataStatisticsSerializer(); // matches DataStatistics impl and version
byte[] bytes = StatisticsUtil.serializeDataStatistics(stats, serializer);
Defensive patterns

Strategy: try-catch

Validate before calling

if (dataStatistics == null) throw new IllegalArgumentException("dataStatistics must not be null");
// ensure serializer type matches the DataStatistics impl before calling
Preconditions.checkArgument(serializer instanceof DataStatisticsSerializer);

Type guard

boolean isCompatible(TypeSerializer<DataStatistics> s, DataStatistics d) {
  return s != null && d != null;
}

Try / catch

try {
  byte[] bytes = StatisticsUtil.serializeDataStatistics(stats, serializer);
} catch (UncheckedIOException e) {
  IOException cause = e.getCause();
  LOG.error("statistics serialization failed", cause);
  throw e; // or fall back to empty statistics
}

Prevention

When it happens

Trigger: Calling StatisticsUtil.serializeDataStatistics(dataStatistics, serializer) when the provided TypeSerializer<DataStatistics> throws IOException during serialize(), e.g. because the serializer's internal buffer cannot grow or the object graph is inconsistent with the serializer version.

Common situations: Job upgrade/downgrade where the serialized statistics type no longer matches the serializer handed to the utility; a custom or misregistered TypeSerializer that fails mid-serialization; JVM heap/buffer issues during state transfer in the shuffle.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0ad3c4f403b95d4d. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java:49

  static DataStatistics createTaskStatistics(
      StatisticsType type, int operatorParallelism, int numPartitions) {
    if (type == StatisticsType.Map) {
      return new MapDataStatistics();
    } else {
      return new SketchDataStatistics(
          SketchUtil.determineOperatorReservoirSize(operatorParallelism, numPartitions));
    }
  }

  static byte[] serializeDataStatistics(
      DataStatistics dataStatistics, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataOutputSerializer out = new DataOutputSerializer(64);
    try {
      statisticsSerializer.serialize(dataStatistics, out);
      return out.getCopyOfBuffer();
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to serialize data statistics", e);
    }
  }

  static DataStatistics deserializeDataStatistics(
      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {
    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);
    try {
      return statisticsSerializer.deserialize(input);
    } catch (IOException e) {
      throw new UncheckedIOException("Fail to deserialize data statistics", e);
    }
  }

  static byte[] serializeCompletedStatistics(
      CompletedStatistics completedStatistics,
      TypeSerializer<CompletedStatistics> statisticsSerializer) {
    try {
      DataOutputSerializer out = new DataOutputSerializer(1024);

View on GitHub (pinned to 86d9c8fc54)