{"record":{"id":"ae1358188e954e95","repo":"apache/iceberg","slug":"failed-to-serialize-sort-key","errorCode":null,"errorMessage":"Failed to serialize sort key","messagePattern":"Failed to serialize sort key","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java","lineNumber":69,"sourceCode":"  SortKeySketchSerializer(TypeSerializer<SortKey> itemSerializer) {\n    this.itemSerializer = itemSerializer;\n    this.listSerializer = new ListSerializer<>(itemSerializer);\n    this.input = new DataInputDeserializer();\n  }\n\n  @Override\n  public byte[] serializeToByteArray(SortKey item) {\n    try {\n      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE);\n      itemSerializer.serialize(item, output);\n      byte[] itemBytes = output.getSharedBuffer();\n      int numBytes = output.length();\n      byte[] out = new byte[numBytes + Integer.BYTES];\n      ByteArrayUtil.copyBytes(itemBytes, 0, out, 4, numBytes);\n      ByteArrayUtil.putIntLE(out, 0, numBytes);\n      return out;\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to serialize sort key\", e);\n    }\n  }\n\n  @Override\n  public byte[] serializeToByteArray(SortKey[] items) {\n    try {\n      DataOutputSerializer output = new DataOutputSerializer(DEFAULT_SORT_KEY_SIZE * items.length);\n      listSerializer.serialize(Arrays.asList(items), output);\n      byte[] itemsBytes = output.getSharedBuffer();\n      int numBytes = output.length();\n      byte[] out = new byte[Integer.BYTES + numBytes];\n      ByteArrayUtil.putIntLE(out, 0, numBytes);\n      System.arraycopy(itemsBytes, 0, out, Integer.BYTES, numBytes);\n      return out;\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to serialize sort key\", e);\n    }\n  }","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java#L51-L87","documentation":"SortKeySketchSerializer.serializeToByteArray(SortKey) wraps any IOException from the underlying Flink TypeSerializer into an UncheckedIOException with message 'Failed to serialize sort key'. This happens while encoding a single SortKey for the reservoir sketch (called from sizeOf and sketch serialization); I/O failures at this layer indicate the DataOutputSerializer buffer could not accommodate the record.","triggerScenarios":"Invoked via sizeOf(item) or when the ReservoirItemsSketch serializes an item, and itemSerializer.serialize throws IOException (e.g. buffer growth failure in DataOutputSerializer, or a nested serializer raising IOException).","commonSituations":"Memory pressure during heavy shuffle traffic when the serializer buffer cannot be expanded; a custom/broken item serializer; extremely large sort-key values exceeding buffer limits.","solutions":["Inspect the wrapped cause (getCause()) to find the real IOException source and fix it there","Ensure the job has adequate task-manager heap; reduce sketch size or sort-key payload sizes","Verify the SortKeySerializer in use is not misconfigured (correct schema and sort order) and test serialization in isolation","Retry the failing job; transient OOM/buffer pressure is the usual trigger"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  byte[] bytes = sketchSerializer.serializeToByteArray(sortKey);\n} catch (UncheckedIOException e) {\n  LOG.error(\"Sort key serialization failed\", e.getCause());\n  throw e;\n}","preventionTips":["Check the wrapped IOException cause; it names the real failing component","Provision sufficient task-manager heap for shuffle sketch buffers","Keep sort-key payloads small (prefer transformed/primitive columns)","Catch UncheckedIOException rather than IOException at call sites"],"tags":["flink","serialization","unchecked-io"],"backgroundTag":"json-serialization-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}