{"record":{"id":"262f691997e9fb09","repo":"apache/iceberg","slug":"encoded-string-reached-maximum-length-262f69","errorCode":null,"errorMessage":"Encoded string reached maximum length: ","messagePattern":"Encoded string reached maximum length: ","errorType":"exception","errorClass":"UTFDataFormatException","httpStatus":null,"severity":"error","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/util/SerializerHelper.java","lineNumber":65,"sourceCode":"   *\n   * <p>See * <a href=\"https://issues.apache.org/jira/browse/FLINK-34228\">FLINK-34228</a> * <a\n   * href=\"https://github.com/apache/flink/pull/24191\">https://github.com/apache/flink/pull/24191</a>\n   *\n   * @param out the output stream to write the string to.\n   * @param str the string value to be written.\n   */\n  public static void writeLongUTF(DataOutputView out, String str) throws IOException {\n    int strlen = str.length();\n    long utflen = 0;\n    int ch;\n\n    /* use charAt instead of copying String to char array */\n    for (int i = 0; i < strlen; i++) {\n      ch = str.charAt(i);\n      utflen += getUTFBytesSize(ch);\n\n      if (utflen > Integer.MAX_VALUE) {\n        throw new UTFDataFormatException(\"Encoded string reached maximum length: \" + utflen);\n      }\n    }\n\n    if (utflen > Integer.MAX_VALUE - 4) {\n      throw new UTFDataFormatException(\"Encoded string is too long: \" + utflen);\n    }\n\n    out.writeInt((int) utflen);\n    writeUTFBytes(out, str, (int) utflen);\n  }\n\n  /**\n   * Similar to {@link DataInputDeserializer#readUTF()}. Except this supports larger payloads which\n   * is up to max integer value.\n   *\n   * <p>Note: This method can be removed when the method which does similar thing within the {@link\n   * DataOutputSerializer} already which does the same thing, so use that one instead once that is\n   * released on Flink version 1.20.","sourceCodeStart":47,"sourceCodeEnd":83,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/util/SerializerHelper.java#L47-L83","documentation":"SerializerHelper.writeLongUTF computes the UTF-8 encoded length of the string in a long accumulator while iterating characters; if the running length exceeds Integer.MAX_VALUE it throws UTFDataFormatException('Encoded string reached maximum length'). This guards the modified UTF-style writer, which ultimately writes the length as an int.","triggerScenarios":"Serializing a single String whose UTF-8 encoding exceeds ~2.1 GB (Integer.MAX_VALUE bytes) via writeLongUTF — practically only with enormous string values in Kafka/serializer payloads.","commonSituations":"Jobs that pack gigantic blobs (e.g. huge JSON, base64 payloads) into a single string column/attribute and then pass them through the custom UTF serializer.","solutions":["Split or chunk the payload across multiple records/fields before serialization","Compress the payload (e.g. gzip + base64) so encoded size stays under the limit","Store large payloads externally (object storage) and serialize only a reference"],"exampleFix":"// before\nString huge = buildGiantJson(); // > 2GB encoded\nhelper.writeLongUTF(out, huge);\n// after\nbyte[] compressed = gzip(huge.getBytes(StandardCharsets.UTF_8));\nout.writeInt(compressed.length);\nout.write(compressed);","handlingStrategy":"validation","validationCode":"long utfSize = 0;\nfor (int i = 0; i < str.length(); i++) { utfSize += utfByteLen(str.charAt(i)); }\nif (utfSize > Integer.MAX_VALUE) throw new IllegalArgumentException(\"String exceeds encodable size\");\n// or compute with StandardCharsets.UTF_8 encoder before writing","typeGuard":"static boolean isEncodable(String s) {\n  return s != null && s.getBytes(StandardCharsets.UTF_8).length < Integer.MAX_VALUE;\n}","tryCatchPattern":"try { helper.writeLongUTF(out, str); }\ncatch (UTFDataFormatException e) {\n  if (e.getMessage().contains(\"reached maximum length\") || e.getMessage().contains(\"too long\")) { chunkAndWrite(out, str); }\n  else throw e;\n}","preventionTips":["Keep single serialized strings well below 2GB encoded","Chunk or compress large payloads before serialization","Add size assertions on string fields before writing records"],"tags":["flink","serialization","utf","string-size-limit"],"backgroundTag":"payload-too-large","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}