apache/iceberg · error · UncheckedIOException

Failed to deserialize sort key sketch

Error message

Failed to deserialize sort key sketch

What it means

SortKeySketchSerializer.deserializeFromMemory() wraps IOException from decoding sort keys out of the sketch's memory segment into an UncheckedIOException with message "Failed to deserialize sort key sketch". This happens when reading a reservoir sketch's sampled items and the byte layout or schema no longer matches the serializer.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java:113

    long offset = startingOffset;
    Util.checkBounds(offset, Integer.BYTES, mem.getCapacity());
    int numBytes = mem.getInt(offset);
    offset += Integer.BYTES;

    Util.checkBounds(offset, numBytes, mem.getCapacity());
    byte[] sortKeyBytes = new byte[numBytes];
    mem.getByteArray(offset, sortKeyBytes, 0, numBytes);
    input.setBuffer(sortKeyBytes);

    try {
      List<SortKey> sortKeys = listSerializer.deserialize(input);
      SortKey[] array = new SortKey[numItems];
      sortKeys.toArray(array);
      input.releaseArrays();
      return array;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to deserialize sort key sketch", e);
    }
  }

  @Override
  public int sizeOf(SortKey item) {
    return serializeToByteArray(item).length;
  }

  @Override
  public int sizeOf(Memory mem, long offset, int numItems) {
    Preconditions.checkArgument(mem != null, "Invalid input memory: null");
    if (numItems <= 0) {
      return 0;
    }

    Util.checkBounds(offset, Integer.BYTES, mem.getCapacity());
    int numBytes = mem.getInt(offset);
    return Integer.BYTES + numBytes;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Ensure all subtasks serialize sketches with the same schema and iceberg-flink-runtime version before merge.
2. Do not change the sort order between checkpoint and restore; take a new savepoint after schema evolution.
3. Catch UncheckedIOException around sketch deserialization and fall back to discarding stale statistics for one cycle.
4. Inspect getCause() to confirm whether it is a schema incompatibility or byte corruption.

Example fix

// before
SortKey[] keys = sketchSerializer.deserializeFromMemory(mem, offset, numItems);
// after
try {
  SortKey[] keys = sketchSerializer.deserializeFromMemory(mem, offset, numItems);
} catch (UncheckedIOException e) {
  LOG.warn("Discarding incompatible sketch statistics", e);
}

try {
  SortKey[] keys = sketchSerializer.deserializeFromMemory(mem, offset, numItems);
} catch (UncheckedIOException e) {
  LOG.warn("Discarding incompatible sketch statistics", e);
  return new SortKey[0]; // fall back to no statistics for this cycle
}

When it happens

Trigger: Deserializing sampled SortKey items from sketch Memory during sketch merge/read (restore from state, coordinator aggregation) when the item bytes were written with a different schema/sort order or are truncated/corrupted.

Common situations: Restoring sketches from a checkpoint after the table's sort order or field types changed; merging sketches produced by operators with mismatched serializer versions; corrupted state bytes.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• Failed to deserialize sort key sketch
• Fail to deserialize data statistics
• Fail to deserialize aggregated statistics
• Failed to decode partition
• Failed to serialize sort key

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/4cd068c78d1f71b7. Report an issue: GitHub.