What causes "Failed to deserialize sort key sketch" in apache/iceberg?

Use the version-aware path (StatisticsUtil.deserializeCompletedStatistics) that falls back to sort key serializer version 1, rather than calling deserializeFromMemory with the default serializer.

apache/iceberg · error · UncheckedIOException

Failed to deserialize sort key sketch

Error message

Failed to deserialize sort key sketch

What it means

SortKeySketchSerializer.deserializeFromMemory wraps IOException while reading numItems sort keys back out of the sketch's memory (via the list serializer) into UncheckedIOException "Failed to deserialize sort key sketch". It indicates the serialized bytes in memory don't match what the current serializer expects.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java:113

    long offset = startingOffset;
    Util.checkBounds(offset, Integer.BYTES, mem.getCapacity());
    int numBytes = mem.getInt(offset);
    offset += Integer.BYTES;

    Util.checkBounds(offset, numBytes, mem.getCapacity());
    byte[] sortKeyBytes = new byte[numBytes];
    mem.getByteArray(offset, sortKeyBytes, 0, numBytes);
    input.setBuffer(sortKeyBytes);

    try {
      List<SortKey> sortKeys = listSerializer.deserialize(input);
      SortKey[] array = new SortKey[numItems];
      sortKeys.toArray(array);
      input.releaseArrays();
      return array;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to deserialize sort key sketch", e);
    }
  }

  @Override
  public int sizeOf(SortKey item) {
    return serializeToByteArray(item).length;
  }

  @Override
  public int sizeOf(Memory mem, long offset, int numItems) {
    Preconditions.checkArgument(mem != null, "Invalid input memory: null");
    if (numItems <= 0) {
      return 0;
    }

    Util.checkBounds(offset, Integer.BYTES, mem.getCapacity());
    int numBytes = mem.getInt(offset);
    return Integer.BYTES + numBytes;

View on GitHub (pinned to 86d9c8fc54)

Solutions

Use the version-aware path (StatisticsUtil.deserializeCompletedStatistics) that falls back to sort key serializer version 1, rather than calling deserializeFromMemory with the default serializer.
Align the serializer version with the data's version before deserializing.
Discard stale statistics state and rebuild from a fresh run if version provenance is unknown.

Example fix

// before: direct deserialize with latest serializer
SortKey[] keys = sketchSerializer.deserializeFromMemory(mem, offset, n);
// after: let StatisticsUtil handle version fallback
CompletedStatistics stats = StatisticsUtil.deserializeCompletedStatistics(bytes, completedSerializer);

Defensive patterns

Strategy: fallback

Validate before calling

Preconditions.checkNotNull(mem, "Invalid input memory: null");
Preconditions.checkArgument(numItems >= 0, "numItems must be non-negative");

Try / catch

try {
  SortKey[] keys = serializer.deserializeFromMemory(mem, offset, n);
} catch (UncheckedIOException e) {
  // fall back to v1 serializer or discard stale statistics
}

Prevention

Prefer StatisticsUtil.deserializeCompletedStatistics for version-aware reads
Track which serializer version wrote sketch bytes
Treat statistics state as disposable — it can be rebuilt

When it happens

Trigger: Calling deserializeFromMemory (e.g. during sketch deserialization in DataSketches-backed sort-key statistics) when bytes were written by an incompatible serializer version — notably the v1/v2 SortKeySerializer version change.

Common situations: Restoring a Flink job (or merging sketches) where statistics bytes were produced with the older v1 SortKeySerializer but are read with the v2 serializer.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/080555189226ca83. Report an issue: GitHub.