{"record":{"id":"e3500e23d0336447","repo":"apache/iceberg","slug":"failed-to-serialize-sort-key-e3500e","errorCode":null,"errorMessage":"Failed to serialize sort key","messagePattern":"Failed to serialize sort key","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java","lineNumber":69,"sourceCode":"  SortKeySketchSerializer(TypeSerializer<SortKey> itemSerializer) {\n    this.itemSerializer = itemSerializer;\n    this.listSerializer = new ListSerializer<>(itemSerializer);\n    this.input = new DataInputDeserializer();\n  }\n\n  @Override\n  public byte[] serializeToByteArray(SortKey item) {\n    try {\n      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE);\n      itemSerializer.serialize(item, output);\n      byte[] itemBytes = output.getSharedBuffer();\n      int numBytes = output.length();\n      byte[] out = new byte[numBytes + Integer.BYTES];\n      ByteArrayUtil.copyBytes(itemBytes, 0, out, 4, numBytes);\n      ByteArrayUtil.putIntLE(out, 0, numBytes);\n      return out;\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to serialize sort key\", e);\n    }\n  }\n\n  @Override\n  public byte[] serializeToByteArray(SortKey[] items) {\n    try {\n      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE * items.length);\n      listSerializer.serialize(Arrays.asList(items), output);\n      byte[] itemsBytes = output.getSharedBuffer();\n      int numBytes = output.length();\n      byte[] out = new byte[Integer.BYTES + numBytes];\n      ByteArrayUtil.putIntLE(out, 0, numBytes);\n      System.arraycopy(itemsBytes, 0, out, Integer.BYTES, numBytes);\n      return out;\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to serialize sort key\", e);\n    }\n  }","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java#L51-L87","documentation":"SortKeySketchSerializer.serializeToByteArray(SortKey) writes a single SortKey to a DataOutputSerializer to feed the Apache DataSketches reservoir. If the underlying write throws IOException (serialization backend failure, output buffer problem), it is wrapped in UncheckedIOException(\"Failed to serialize sort key\"). It converts checked I/O exceptions from the low-level serializer into runtime exceptions for the sketch API.","triggerScenarios":"The sketch calls serializeToByteArray during toByteArray/sizeOf while serializing a SortKey and DataOutputSerializer.write throws IOException — typically due to buffer growth failures (e.g. negative or overflowed sizes) or memory issues for very large field values.","commonSituations":"Extremely large string/binary sort key values causing huge intermediate buffers; JVM memory pressure / OOM conditions inside the output stream; custom types with faulty conversion producing oversized output.","solutions":["Inspect the IOException cause (UncheckedIOException.getCause()) to identify the underlying buffer/IO failure.","Reduce sort key size: avoid sorting on very large string/binary columns; prefix or hash long values before sorting.","Increase JVM heap/taskmanager memory if buffer growth failures stem from memory pressure.","Verify the table schema types of sort columns match expected primitives (no broken conversion logic)."],"exampleFix":"// before: sorting on huge binary column\nALTER TABLE t WRITE ORDERED BY payload;  -- binary, tens of MB\n// after\nALTER TABLE t WRITE ORDERED BY payload_hash;","handlingStrategy":"try-catch","validationCode":"// pre-check key size before handing to sketch\nif (estimatedSortKeySize(key) > MAX_KEY_BYTES) {\n  key = hashKey(key);\n}","typeGuard":null,"tryCatchPattern":"try {\n  byte[] bytes = serializer.serializeToByteArray(key);\n} catch (UncheckedIOException e) {\n  LOG.error(\"Sort key serialization failed\", e.getCause());\n  throw e;\n}","preventionTips":["Avoid very large string/binary sort columns.","Monitor taskmanager memory; buffer growth failures often correlate with pressure.","Read the cause chain of UncheckedIOException when debugging."],"tags":["flink","serialization","sort-key","io"],"backgroundTag":"json-serialization-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}