apache/iceberg · error · UncheckedIOException

Failed to serialize sort key

Error message

Failed to serialize sort key

What it means

SortKeySketchSerializer.serializeToByteArray(SortKey) wraps any IOException from the underlying Flink TypeSerializer into an UncheckedIOException with message 'Failed to serialize sort key'. This happens while encoding a single SortKey for the reservoir sketch (called from sizeOf and sketch serialization); I/O failures at this layer indicate the DataOutputSerializer buffer could not accommodate the record.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java:69

  SortKeySketchSerializer(TypeSerializer<SortKey> itemSerializer) {
    this.itemSerializer = itemSerializer;
    this.listSerializer = new ListSerializer<>(itemSerializer);
    this.input = new DataInputDeserializer();
  }

  @Override
  public byte[] serializeToByteArray(SortKey item) {
    try {
      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE);
      itemSerializer.serialize(item, output);
      byte[] itemBytes = output.getSharedBuffer();
      int numBytes = output.length();
      byte[] out = new byte[numBytes + Integer.BYTES];
      ByteArrayUtil.copyBytes(itemBytes, 0, out, 4, numBytes);
      ByteArrayUtil.putIntLE(out, 0, numBytes);
      return out;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to serialize sort key", e);
    }
  }

  @Override
  public byte[] serializeToByteArray(SortKey[] items) {
    try {
      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE * items.length);
      listSerializer.serialize(Arrays.asList(items), output);
      byte[] itemsBytes = output.getSharedBuffer();
      int numBytes = output.length();
      byte[] out = new byte[Integer.BYTES + numBytes];
      ByteArrayUtil.putIntLE(out, 0, numBytes);
      System.arraycopy(itemsBytes, 0, out, Integer.BYTES, numBytes);
      return out;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to serialize sort key", e);
    }
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the wrapped cause (getCause()) to find the real IOException source and fix it there
2. Ensure the job has adequate task-manager heap; reduce sketch size or sort-key payload sizes
3. Verify the SortKeySerializer in use is not misconfigured (correct schema and sort order) and test serialization in isolation
4. Retry the failing job; transient OOM/buffer pressure is the usual trigger

try {
  byte[] bytes = sketchSerializer.serializeToByteArray(sortKey);
} catch (UncheckedIOException e) {
  LOG.error("Sort key serialization failed", e.getCause());
  throw e;
}

When it happens

Trigger: Invoked via sizeOf(item) or when the ReservoirItemsSketch serializes an item, and itemSerializer.serialize throws IOException (e.g. buffer growth failure in DataOutputSerializer, or a nested serializer raising IOException).

Common situations: Memory pressure during heavy shuffle traffic when the serializer buffer cannot be expanded; a custom/broken item serializer; extremely large sort-key values exceeding buffer limits.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Failed to serialize sort key
• Fail to serialize data statistics
• Fail to serialize aggregated statistics
• Failed to serialize PK index key
• Failed to encode partition

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/ae1358188e954e95. Report an issue: GitHub.