{"record":{"id":"f6ca6a480b1eda3d","repo":"apache/hadoop","slug":"string-too-long","errorCode":null,"errorMessage":"string too long!","messagePattern":"string too long!","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/UTF8.java","lineNumber":105,"sourceCode":"  /** @return The number of bytes in the encoded string. */\n  public int getLength() {\n    return length;\n  }\n\n  /**\n   * Set to contain the contents of a string.\n   * @param string input string.\n   */\n  public void set(String string) {\n    if (string.length() > 0xffff/3) {             // maybe too long\n      LOG.warn(\"truncating long string: \" + string.length()\n               + \" chars, starting with \" + string.substring(0, 20));\n      string = string.substring(0, 0xffff/3);\n    }\n\n    length = utf8Length(string);                  // compute length\n    if (length > 0xffff)                          // double-check length\n      throw new RuntimeException(\"string too long!\");\n\n    if (bytes == null || length > bytes.length)   // grow buffer\n      bytes = new byte[length];\n\n    try {                                         // avoid sync'd allocations\n      DataOutputBuffer obuf = OBUF_FACTORY.get();\n      obuf.reset();\n      writeChars(obuf, string, 0, string.length());\n      System.arraycopy(obuf.getData(), 0, bytes, 0, length);\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    }\n  }\n\n  /**\n   * Set to contain the contents of a string.\n   * @param other input other.\n   */","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/UTF8.java#L87-L123","documentation":"org.apache.hadoop.io.UTF8 is the legacy UTF-8 Writable whose length field is an unsigned 16-bit short. set(String) pre-truncates input longer than 0xffff/3 = 21845 characters (with a WARN) and then encodes; if the encoded byte length still exceeds 0xffff it throws RuntimeException(\"string too long!\"). Because characters can take up to 3 bytes, even pre-truncated strings of CJK content can exceed the 64 KB byte bound.","triggerScenarios":"Calling UTF8.set(String) with a string whose UTF-8 encoding exceeds 65535 bytes — most easily reached with 3-byte-per-char content at or near the 21845-char pre-truncation point; long URLs, JSON blobs, or messages stored in legacy UTF8-typed fields.","commonSituations":"Legacy MapFile/UTF8-based schemas receiving modern long strings; East-Asian content where character-count guards pass but byte limits fail; migrating old Writable types that were sized for ASCII identifiers.","solutions":["Switch the field to org.apache.hadoop.io.Text, which stores the length as a VInt (up to ~2 GB) — Text is the supported replacement for UTF8.","If UTF8 must stay, cap the encoded size yourself before calling set(): ensure UTF8.utf8Length(s) <= 0xffff, truncating by bytes.","Redesign the storage so unbounded strings are not forced through a 16-bit-length container."],"exampleFix":"// before\nUTF8 u = new UTF8();\nu.set(hugeString);                       // RuntimeException: string too long!\n\n// after\nText t = new Text(hugeString);           // VInt length, no 64 KB bound","handlingStrategy":"validation","validationCode":"static boolean fitsLegacyUtf8(String s) {\n  return UTF8.utf8Length(s) <= 0xffff;  // 16-bit length field bound\n}\n\n// use before storing into a UTF8 field\nif (!fitsLegacyUtf8(s)) {\n  s = s.substring(0, Math.min(s.length(), 20000));  // byte-safe for CJK too\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer org.apache.hadoop.io.Text for any string that can approach 64 KB","If stuck with UTF8, check UTF8.utf8Length(s) <= 0xffff before set()","Truncate by encoded bytes, not by characters, when content is multi-byte"],"tags":["utf8","text","legacy-api","string-length","hadoop"],"backgroundTag":"string-too-long","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}