apache/iceberg · error · UncheckedIOException

Failed to serialize sort key

Error message

Failed to serialize sort key

What it means

SortKeySketchSerializer.serializeToByteArray(SortKey) wraps any IOException from writing the sort key into an UncheckedIOException with message "Failed to serialize sort key". This serializer feeds the reservoir-sketch sampling; a failure here means the underlying TypeSerializer could not encode the SortKey to the output stream.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java:69

  SortKeySketchSerializer(TypeSerializer<SortKey> itemSerializer) {
    this.itemSerializer = itemSerializer;
    this.listSerializer = new ListSerializer<>(itemSerializer);
    this.input = new DataInputDeserializer();
  }

  @Override
  public byte[] serializeToByteArray(SortKey item) {
    try {
      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE);
      itemSerializer.serialize(item, output);
      byte[] itemBytes = output.getSharedBuffer();
      int numBytes = output.length();
      byte[] out = new byte[numBytes + Integer.BYTES];
      ByteArrayUtil.copyBytes(itemBytes, 0, out, 4, numBytes);
      ByteArrayUtil.putIntLE(out, 0, numBytes);
      return out;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to serialize sort key", e);
    }
  }

  @Override
  public byte[] serializeToByteArray(SortKey[] items) {
    try {
      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE * items.length);
      listSerializer.serialize(Arrays.asList(items), output);
      byte[] itemsBytes = output.getSharedBuffer();
      int numBytes = output.length();
      byte[] out = new byte[Integer.BYTES + numBytes];
      ByteArrayUtil.putIntLE(out, 0, numBytes);
      System.arraycopy(itemsBytes, 0, out, Integer.BYTES, numBytes);
      return out;
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to serialize sort key", e);
    }
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Verify the sort key's schema matches the serializer's schema — rebuild the serializer after schema changes instead of reusing a stale one.
2. Check that sort-key columns are not dropped/retyped in an evolved schema before resuming from a savepoint.
3. Catch UncheckedIOException in sampling code to fail gracefully and skip statistics collection rather than kill the job, if statistics are optional.
4. Inspect getCause() for the underlying IOException to identify whether it is a schema or stream problem.

Example fix

// before
byte[] bytes = sketchSerializer.serializeToByteArray(sortKey);
// after
try {
  byte[] bytes = sketchSerializer.serializeToByteArray(sortKey);
} catch (UncheckedIOException e) {
  LOG.warn("Skipping sort key statistics: {}", e.getCause().getMessage());
}

try {
  byte[] bytes = sketchSerializer.serializeToByteArray(sortKey);
} catch (UncheckedIOException e) {
  LOG.warn("Skipping sort key: {}", e.getCause().getMessage());
}

When it happens

Trigger: Called during sketch sampling when sizeOf() or sketch update needs the byte representation and the underlying sort-key serialization throws IOException (e.g., serializer state invalid, incompatible schema after evolution, or I/O failure in the backing output).

Common situations: Schema evolution changed a sort-key column type so the cached serializer no longer matches live SortKey objects; null/incoherent SortKey fields; output stream failures in the operator.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Failed to serialize sort key
• Failed to serialize sort key
• Fail to serialize data statistics
• Fail to serialize aggregated statistics
• Failed to serialize PK index key

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/97954ee17fe2d289. Report an issue: GitHub.