apache/iceberg · error · UncheckedIOException

Failed to deserialize sort key sketch

Error message

Failed to deserialize sort key sketch

What it means

Thrown by SortKeySketchSerializer when deserializing a SortKey sketch from its byte representation fails with an IOException during Flink state/network transfer of shuffle statistics. It wraps the underlying serialization failure as an UncheckedIOException. The library throws it because sort-key sketches are exchanged between Flink subtasks and a corrupt or incompatible payload cannot be decoded.

Solutions

  1. Align the Iceberg version used to write the payload and the one deserializing it; restore checkpoints with the same version that produced them.
  2. Check SortKeySerializer version compatibility; if restoring from a lower version, switch the serializer to v1 as StatisticsUtil does via changeSortKeySerializerVersion(1).
  3. Re-run the shuffle from a fresh job (discard the old state) so sketches are rebuilt from input data.
  4. Inspect the cause (IOException) for corruption/truncation and re-transfer or re-serialize the sketch.

Example fix

// before: deserializing bytes written by an older serializer
SortKey[] keys = serializer.deserializeFromMemory(record);
// after: guard version before deserializing (as StatisticsUtil does)
serializer.changeSortKeySerializerVersion(1);
SortKey[] keys = serializer.deserializeFromMemory(record);
serializer.changeSortKeySerializerVersionLatest();
Defensive patterns

Strategy: try-catch

Validate before calling

if (bytes == null || bytes.length == 0) { throw new IllegalArgumentException("Empty sketch payload"); }

Type guard

boolean isSketchPayload(byte[] bytes) { return bytes != null && bytes.length > 0; }

Try / catch

try {
  SortKey[] keys = serializer.deserializeFromMemory(record);
} catch (UncheckedIOException e) {
  LOG.warn("Sketch deserialization failed, rebuilding statistics", e);
  // fall back to re-collecting statistics
}

Prevention

When it happens

Trigger: Calling deserializeFromMemory on bytes written by an incompatible SortKeySerializer version, truncated or corrupted Flink state / operator-channel payload, or reading a snapshot/checkpoint saved by a different Iceberg/Flink version whose sketch layout changed.

Common situations: Restoring a Flink job from a checkpoint/savepoint produced with an older Iceberg version, network-serialized statistics crossing serializer versions after an upgrade, or manual replay of statistics payloads stored out of band.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/7607d56732a996b2. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java:113

    long offset = startingOffset;
    Util.checkBounds(offset, Integer.BYTES, mem.getCapacity());
    int numBytes = mem.getInt(offset);
    offset += Integer.BYTES;

    Util.checkBounds(offset, numBytes, mem.getCapacity());
    byte[] sortKeyBytes = new byte[numBytes];
    mem.getByteArray(offset, sortKeyBytes, 0, numBytes);
    input.setBuffer(sortKeyBytes);

    try {
      List<SortKey> sortKeys = listSerializer.deserialize(input);
      SortKey[] array = new SortKey[numItems];
      sortKeys.toArray(array);
      input.releaseArrays();
      return array;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to deserialize sort key sketch", e);
    }
  }

  @Override
  public int sizeOf(SortKey item) {
    return serializeToByteArray(item).length;
  }

  @Override
  public int sizeOf(Memory mem, long offset, int numItems) {
    Preconditions.checkArgument(mem != null, "Invalid input memory: null");
    if (numItems <= 0) {
      return 0;
    }

    Util.checkBounds(offset, Integer.BYTES, mem.getCapacity());
    int numBytes = mem.getInt(offset);
    return Integer.BYTES + numBytes;

View on GitHub (pinned to 86d9c8fc54)