{"record":{"id":"b20563c594242a44","repo":"apache/iceberg","slug":"failed-to-serialize-sort-key-b20563","errorCode":null,"errorMessage":"Failed to serialize sort key","messagePattern":"Failed to serialize sort key","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java","lineNumber":69,"sourceCode":"  SortKeySketchSerializer(TypeSerializer<SortKey> itemSerializer) {\n    this.itemSerializer = itemSerializer;\n    this.listSerializer = new ListSerializer<>(itemSerializer);\n    this.input = new DataInputDeserializer();\n  }\n\n  @Override\n  public byte[] serializeToByteArray(SortKey item) {\n    try {\n      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE);\n      itemSerializer.serialize(item, output);\n      byte[] itemBytes = output.getSharedBuffer();\n      int numBytes = output.length();\n      byte[] out = new byte[numBytes + Integer.BYTES];\n      ByteArrayUtil.copyBytes(itemBytes, 0, out, 4, numBytes);\n      ByteArrayUtil.putIntLE(out, 0, numBytes);\n      return out;\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to serialize sort key\", e);\n    }\n  }\n\n  @Override\n  public byte[] serializeToByteArray(SortKey[] items) {\n    try {\n      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE * items.length);\n      listSerializer.serialize(Arrays.asList(items), output);\n      byte[] itemsBytes = output.getSharedBuffer();\n      int numBytes = output.length();\n      byte[] out = new byte[Integer.BYTES + numBytes];\n      ByteArrayUtil.putIntLE(out, 0, numBytes);\n      System.arraycopy(itemsBytes, 0, out, Integer.BYTES, numBytes);\n      return out;\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to serialize sort key\", e);\n    }\n  }","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java#L51-L87","documentation":"SortKeySketchSerializer.serializeToByteArray(SortKey) wraps any IOException raised while serializing a single sort key into an UncheckedIOException with message \"Failed to serialize sort key\". This path is invoked by the sketch (QuantileSummary/KllFloatsSketch style) sizeOf calculation. A failure here means the underlying DataOutputSerializer stream failed, which is practically a serialization bug or OOM-level stream issue.","triggerScenarios":"sizeOf(SortKey) being called (e.g. by the sketch during statistics collection in SortKeySketch/SortKeySketchAggregator) when the underlying SortKeySerializer.serialize throws IOException.","commonSituations":"A SortKey whose internal state (null/invalid values vs the sort schema) makes serialization fail; stream buffer growth failure due to memory pressure during shuffle statistics collection.","solutions":["Check the wrapped IOException cause for the real failure (often a serializer/schema mismatch) and fix the sort-key schema alignment.","Ensure the SortKey instance is initialized for the current SortOrder before it reaches the sketch (use SortKey.create(sortOrder)).","Increase task-manager memory if the cause indicates buffer allocation failure."],"exampleFix":"// before: SortKey created with stale sort order after schema change\nSortKey key = new SortKey(oldSortOrder);\n// after\nSortKey key = SortKey.create(table.sortOrder());","handlingStrategy":"try-catch","validationCode":"Preconditions.checkNotNull(sortKey, \"SortKey must not be null\");\n// ensure sortKey was created for the active sort order","typeGuard":null,"tryCatchPattern":"try {\n  byte[] bytes = serializer.serializeToByteArray(key);\n} catch (UncheckedIOException e) {\n  LOG.error(\"Sort key serialization failed\", e.getCause());\n}","preventionTips":["Create SortKey instances via SortKey.create(table.sortOrder())","Refresh sort order references after schema changes","Watch task-manager memory during shuffle statistics collection"],"tags":["flink","serialization","unchecked-io","sort-key"],"backgroundTag":"json-serialization-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}