apache/iceberg · error · IllegalArgumentException

Unsupported data statistics type:

Error message

Unsupported data statistics type: 

What it means

DataStatisticsSerializer.copy() deserializes statistics whose type was read from the wire (via statisticsTypeSerializer) and reconstructs either a MapDataStatistics or SketchDataStatistics. If the recorded StatisticsType is neither Map nor Sketch, the serializer cannot reconstruct the object and throws IllegalArgumentException. This guards against corrupted serialization output or a StatisticsType enum that the current code version does not handle.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsSerializer.java:91

  public DataStatistics copy(DataStatistics obj) {
    StatisticsType statisticsType = obj.type();
    if (statisticsType == StatisticsType.Map) {
      MapDataStatistics from = (MapDataStatistics) obj;
      Map<SortKey, Long> fromStats = (Map<SortKey, Long>) from.result();
      Map<SortKey, Long> toStats = Maps.newHashMap(fromStats);
      return new MapDataStatistics(toStats);
    } else if (statisticsType == StatisticsType.Sketch) {
      // because ReservoirItemsSketch doesn't expose enough public methods for cloning,
      // this implementation adopted the less efficient serialization and deserialization.
      SketchDataStatistics from = (SketchDataStatistics) obj;
      ReservoirItemsSketch<SortKey> fromStats = (ReservoirItemsSketch<SortKey>) from.result();
      byte[] bytes = fromStats.toByteArray(sketchSerializer);
      Memory memory = Memory.wrap(bytes);
      ReservoirItemsSketch<SortKey> toStats =
          ReservoirItemsSketch.heapify(memory, sketchSerializer);
      return new SketchDataStatistics(toStats);
    } else {
      throw new IllegalArgumentException("Unsupported data statistics type: " + statisticsType);
    }
  }

  @Override
  public DataStatistics copy(DataStatistics from, DataStatistics reuse) {
    // not much benefit to reuse
    return copy(from);
  }

  @Override
  public int getLength() {
    return -1;
  }

  @SuppressWarnings("unchecked")
  @Override
  public void serialize(DataStatistics obj, DataOutputView target) throws IOException {
    StatisticsType statisticsType = obj.type();

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Align the Iceberg flink runtime version between job submission and the cluster/checkpoint state so StatisticsType values match.
2. Discard old incompatible state (e.g. start with a fresh savepoint / allowNonRestoredState) if it contains unknown statistics types.
3. If you maintain a fork, add a branch handling the new StatisticsType in copy() (and serialize/deserialize) of DataStatisticsSerializer.

Example fix

// before: pass through unknown statistics
DataStatistics copied = serializer.copy(stats, null);
// after: guard before copy
if (stats.type() == StatisticsType.Map || stats.type() == StatisticsType.Sketch) {
  DataStatistics copied = serializer.copy(stats, null);
} else {
  DataStatistics copied = new MapDataStatistics(Collections.emptyMap());
}

if (stats.type() != StatisticsType.Map && stats.type() != StatisticsType.Sketch) {
  throw new IllegalStateException("Cannot copy statistics of type " + stats.type());
}

boolean copyable(DataStatistics s) {
  return s.type() == StatisticsType.Map || s.type() == StatisticsType.Sketch;
}

try {
  copied = serializer.copy(stats, null);
} catch (IllegalArgumentException e) {
  LOG.warn("Unsupported statistics type {}, using empty stats", stats.type(), e);
  copied = new MapDataStatistics(Collections.emptyMap());
}

When it happens

Trigger: Calling copy(DataStatistics, DataStatistics) on a DataStatistics object whose type() returns a StatisticsType value other than Map or Sketch (e.g. NONE or an enum value from a newer Iceberg version). The type is serialized/deserialized through DataStatisticsSerializer.serialize/deserialize before reaching copy's branch logic.

Common situations: Running a job where checkpoints/state were written by a newer Iceberg version that introduced a new StatisticsType; manually constructing a DataStatistics wrapper with an unexpected type; bugs in custom DataStatistics implementations whose type() lies about the concrete class.

Related errors

• Unsupported data statistics type: ${statisticsType}
• The Avro schema is not a nullable type: ${schema}
• Fail to serialize at field: %s.
• Unrecognized version or corrupt state: <version>
• Failed to serialize PK index key

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/def7223348d3dac5. Report an issue: GitHub.