apache/iceberg · error · UncheckedIOException

Failed to deserialize sort key sketch

Error message

Failed to deserialize sort key sketch

What it means

UncheckedIOException thrown by SortKeySketchSerializer.deserializeFromMemory when the SortKey list inside a serialized sketch cannot be read from memory. It wraps the underlying IOException raised by Flink's list TypeSerializer, typically because the bytes are corrupt, truncated, or were written by an incompatible serializer version.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java:113

    long offset = startingOffset;
    Util.checkBounds(offset, Integer.BYTES, mem.getCapacity());
    int numBytes = mem.getInt(offset);
    offset += Integer.BYTES;

    Util.checkBounds(offset, numBytes, mem.getCapacity());
    byte[] sortKeyBytes = new byte[numBytes];
    mem.getByteArray(offset, sortKeyBytes, 0, numBytes);
    input.setBuffer(sortKeyBytes);

    try {
      List<SortKey> sortKeys = listSerializer.deserialize(input);
      SortKey[] array = new SortKey[numItems];
      sortKeys.toArray(array);
      input.releaseArrays();
      return array;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to deserialize sort key sketch", e);
    }
  }

  @Override
  public int sizeOf(SortKey item) {
    return serializeToByteArray(item).length;
  }

  @Override
  public int sizeOf(Memory mem, long offset, int numItems) {
    Preconditions.checkArgument(mem != null, "Invalid input memory: null");
    if (numItems <= 0) {
      return 0;
    }

    Util.checkBounds(offset, Integer.BYTES, mem.getCapacity());
    int numBytes = mem.getInt(offset);
    return Integer.BYTES + numBytes;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Verify the checkpoint/savepoint was created with the same (or compatible) Iceberg/Flink version as the restoring job
2. Regenerate the sketch bytes via SortKeySketchSerializer.serializeToByteArray instead of reusing old bytes
3. If restoring across versions, use StatisticsUtil.deserializeCompletedStatistics which falls back to sort key serializer v1
4. Inspect the wrapped IOException cause to identify exact offset/format mismatch

Example fix

// before
SortKey[] keys = serializer.deserializeFromMemory(legacyBytes);
// after
try {
  SortKey[] keys = serializer.deserializeFromMemory(legacyBytes);
} catch (UncheckedIOException e) {
  // fall back to v1 sort key serialization for old checkpoints
  sortKeySerializer.changeSortKeySerializerVersion(1);
  SortKey[] keys = serializer.deserializeFromMemory(legacyBytes);
  sortKeySerializer.changeSortKeySerializerVersionLatest();
}

if (bytes == null || bytes.length == 0) { throw new IllegalArgumentException("empty sketch bytes"); }

boolean isSortableSketchBytes(byte[] bytes) { return bytes != null && bytes.length > 0; }

try { SortKey[] keys = serializer.deserializeFromMemory(bytes); } catch (UncheckedIOException e) { /* fall back to v1 serializer or regenerate sketch */ }

When it happens

Trigger: Calling deserializeFromMemory on bytes that are truncated, corrupted, or produced by an older/newer SortKeySerializer version whose on-wire format no longer matches; also fails when numItems or the list payload does not match what the serializer expects.

Common situations: Restoring a Flink job from a checkpoint/savepoint taken with a different Iceberg version; corrupted or hand-edited state bytes; passing a byte array of the wrong payload to the memory deserializer.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• The Avro schema is not a nullable type: ${schema}
• Fail to serialize at field: %s.
• Unrecognized version or corrupt state: <version>
• Failed to serialize PK index key
• Failed to encode partition

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/980842a4094bfab3. Report an issue: GitHub.