apache/iceberg · error · UncheckedIOException

Failed to serialize sort key

Error message

Failed to serialize sort key

What it means

SortKeySketchSerializer.serializeToByteArray(SortKey) wraps any IOException raised while serializing a single sort key into an UncheckedIOException with message "Failed to serialize sort key". This path is invoked by the sketch (QuantileSummary/KllFloatsSketch style) sizeOf calculation. A failure here means the underlying DataOutputSerializer stream failed, which is practically a serialization bug or OOM-level stream issue.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java:69

  SortKeySketchSerializer(TypeSerializer<SortKey> itemSerializer) {
    this.itemSerializer = itemSerializer;
    this.listSerializer = new ListSerializer<>(itemSerializer);
    this.input = new DataInputDeserializer();
  }

  @Override
  public byte[] serializeToByteArray(SortKey item) {
    try {
      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE);
      itemSerializer.serialize(item, output);
      byte[] itemBytes = output.getSharedBuffer();
      int numBytes = output.length();
      byte[] out = new byte[numBytes + Integer.BYTES];
      ByteArrayUtil.copyBytes(itemBytes, 0, out, 4, numBytes);
      ByteArrayUtil.putIntLE(out, 0, numBytes);
      return out;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to serialize sort key", e);
    }
  }

  @Override
  public byte[] serializeToByteArray(SortKey[] items) {
    try {
      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE * items.length);
      listSerializer.serialize(Arrays.asList(items), output);
      byte[] itemsBytes = output.getSharedBuffer();
      int numBytes = output.length();
      byte[] out = new byte[Integer.BYTES + numBytes];
      ByteArrayUtil.putIntLE(out, 0, numBytes);
      System.arraycopy(itemsBytes, 0, out, Integer.BYTES, numBytes);
      return out;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to serialize sort key", e);
    }
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check the wrapped IOException cause for the real failure (often a serializer/schema mismatch) and fix the sort-key schema alignment.
2. Ensure the SortKey instance is initialized for the current SortOrder before it reaches the sketch (use SortKey.create(sortOrder)).
3. Increase task-manager memory if the cause indicates buffer allocation failure.

Example fix

// before: SortKey created with stale sort order after schema change
SortKey key = new SortKey(oldSortOrder);
// after
SortKey key = SortKey.create(table.sortOrder());

Preconditions.checkNotNull(sortKey, "SortKey must not be null");
// ensure sortKey was created for the active sort order

try {
  byte[] bytes = serializer.serializeToByteArray(key);
} catch (UncheckedIOException e) {
  LOG.error("Sort key serialization failed", e.getCause());
}

When it happens

Trigger: sizeOf(SortKey) being called (e.g. by the sketch during statistics collection in SortKeySketch/SortKeySketchAggregator) when the underlying SortKeySerializer.serialize throws IOException.

Common situations: A SortKey whose internal state (null/invalid values vs the sort schema) makes serialization fail; stream buffer growth failure due to memory pressure during shuffle statistics collection.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Failed to serialize sort key
• Fail to serialize data statistics
• Fail to serialize aggregated statistics
• Failed to serialize PK index key
• Failed to encode partition

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b20563c594242a44. Report an issue: GitHub.