apache/iceberg · error · IllegalArgumentException

Unsupported data statistics type: ${statisticsType}

Error message

Unsupported data statistics type: ${statisticsType}

What it means

DataStatisticsSerializer.copy delegates to serialize/deserialize after determining the StatisticsType. If the type is neither Map nor Sketch, copy throws IllegalArgumentException('Unsupported data statistics type: ...'). This indicates a DataStatistics implementation or type value the serializer does not know how to deep-copy.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsSerializer.java:91

  public DataStatistics copy(DataStatistics obj) {
    StatisticsType statisticsType = obj.type();
    if (statisticsType == StatisticsType.Map) {
      MapDataStatistics from = (MapDataStatistics) obj;
      Map<SortKey, Long> fromStats = (Map<SortKey, Long>) from.result();
      Map<SortKey, Long> toStats = Maps.newHashMap(fromStats);
      return new MapDataStatistics(toStats);
    } else if (statisticsType == StatisticsType.Sketch) {
      // because ReservoirItemsSketch doesn't expose enough public methods for cloning,
      // this implementation adopted the less efficient serialization and deserialization.
      SketchDataStatistics from = (SketchDataStatistics) obj;
      ReservoirItemsSketch<SortKey> fromStats = (ReservoirItemsSketch<SortKey>) from.result();
      byte[] bytes = fromStats.toByteArray(sketchSerializer);
      Memory memory = Memory.wrap(bytes);
      ReservoirItemsSketch<SortKey> toStats =
          ReservoirItemsSketch.heapify(memory, sketchSerializer);
      return new SketchDataStatistics(toStats);
    } else {
      throw new IllegalArgumentException("Unsupported data statistics type: " + statisticsType);
    }
  }

  @Override
  public DataStatistics copy(DataStatistics from, DataStatistics reuse) {
    // not much benefit to reuse
    return copy(from);
  }

  @Override
  public int getLength() {
    return -1;
  }

  @SuppressWarnings("unchecked")
  @Override
  public void serialize(DataStatistics obj, DataOutputView target) throws IOException {
    StatisticsType statisticsType = obj.type();

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Use only the built-in MapDataStatistics / SketchDataStatistics types with the shuffle sink
2. Ensure the same connector version is used across the whole job and when restoring from a savepoint
3. Check the statisticsType value in the message; if it is null or odd, trace where the DataStatistics was created
4. Remove any custom DataStatistics wrapper classes from the sink configuration

Example fix

// before: custom type
DataStatistics stats = new MyCustomStatistics();
// after: built-in types only
DataStatistics stats = new MapDataStatistics(map);

StatisticsType t = from != null ? from.type() : null;
if (t != StatisticsType.Map && t != StatisticsType.Sketch) {
  throw new IllegalArgumentException("Pre-check: unsupported statistics type " + t);
}

static boolean isSupportedStatistics(DataStatistics s) {
  return s instanceof MapDataStatistics || s instanceof SketchDataStatistics;
}

try {
  serializer.copy(stats, null);
} catch (IllegalArgumentException e) {
  // fall back to dropping statistics or rebuilding a supported instance
}

When it happens

Trigger: copy(from, reuse) is invoked by Flink on a DataStatistics whose type() returns a value outside {Map, Sketch} — e.g. a custom DataStatistics implementation passed into the statistics-based sink, or a type added in a newer connector version processed by an older serializer.

Common situations: Custom DataStatistics implementations registered into the shuffle sink; forward/backward version skew between job jars during a savepoint restore; corrupted type metadata in shipped state.

Related errors

• Invalid statistics type: %s. Should be Map or Sketch
• Unsupported data statistics type:
• Unknown type for int field. Type name: ${className}
• Unknown type for long field. Type name: ${className}
• Unknown type for timestamp_ns: ${class}

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/905de2a7d78a1e27. Report an issue: GitHub.