{"record":{"id":"d84372d6194bfc5d","repo":"apache/hadoop","slug":"illegal-unicode-codepoint-in-string","errorCode":null,"errorMessage":"Illegal Unicode Codepoint {} in string.","messagePattern":"Illegal Unicode Codepoint (.+?) in string\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java","lineNumber":264,"sourceCode":"    }\n    return new Buffer(barr);\n  }\n  \n  private static int utf8LenForCodePoint(final int cpt) throws IOException {\n    if (cpt >=0 && cpt <= 0x7F) {\n      return 1;\n    }\n    if (cpt >= 0x80 && cpt <= 0x07FF) {\n      return 2;\n    }\n    if ((cpt >= 0x0800 && cpt < 0xD800) ||\n        (cpt > 0xDFFF && cpt <= 0xFFFD)) {\n      return 3;\n    }\n    if (cpt >= 0x10000 && cpt <= 0x10FFFF) {\n      return 4;\n    }\n    throw new IOException(\"Illegal Unicode Codepoint \"+\n                          Integer.toHexString(cpt)+\" in string.\");\n  }\n  \n  private static final int B10 =    Integer.parseInt(\"10000000\", 2);\n  private static final int B110 =   Integer.parseInt(\"11000000\", 2);\n  private static final int B1110 =  Integer.parseInt(\"11100000\", 2);\n  private static final int B11110 = Integer.parseInt(\"11110000\", 2);\n  private static final int B11 =    Integer.parseInt(\"11000000\", 2);\n  private static final int B111 =   Integer.parseInt(\"11100000\", 2);\n  private static final int B1111 =  Integer.parseInt(\"11110000\", 2);\n  private static final int B11111 = Integer.parseInt(\"11111000\", 2);\n  \n  private static int writeUtf8(int cpt, final byte[] bytes, final int offset)\n    throws IOException {\n    if (cpt >=0 && cpt <= 0x7F) {\n      bytes[offset] = (byte) cpt;\n      return 1;\n    }","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java#L246-L282","documentation":"Utils computes UTF-8 byte length for a codepoint (used by toBinaryString when sizing the encoding of legacy record strings). Codepoints accepted are < 0x80, 0x80-0x07FF, 0x0800-0xD7FF, 0xE000-0xFFFD, and 0x10000-0x10FFFF. Anything else — lone surrogates (0xD800-0xDFFF), values above 0x10FFFF, or negative values — throws IOException 'Illegal Unicode Codepoint <hex> in string.'.","triggerScenarios":"Serializing a Java String that contains an unpaired surrogate char (half of a broken surrogate pair), which occurs when the string was built from data decoded with a charset mismatch or sliced mid-surrogate; the codepoint fails every valid range check.","commonSituations":"Strings decoded from bytes with ISO-8859-1/UTF-8 mismatch, substring() cuts splitting a surrogate pair, data read from sources that emit CESU-8 or raw surrogate code units, or upstream files already containing invalid UTF-8 that got laundered into Java chars.","solutions":["Sanitize strings before serialization: replace unpaired surrogates (Character.isHighSurrogate/isLowSurrogate without their counterpart) with U+FFFD","Fix the upstream decode: read the source bytes as UTF-8 (or the true charset) instead of constructing chars ad hoc","Avoid slicing strings on char boundaries where supplementary characters may exist; slice on codepoint boundaries","Locate the offending value by catching the exception and logging the string around the failing index"],"exampleFix":"// before\nString out = s; // s may contain unpaired surrogates\n// after\nStringBuilder sb = new StringBuilder(s.length());\nfor (int i = 0; i < s.length(); i++) {\n  char c = s.charAt(i);\n  if (Character.isSurrogate(c)) { sb.append('\\uFFFD'); }\n  else { sb.append(c); }\n}","handlingStrategy":"validation","validationCode":"static boolean allCodePointsEncodable(String s) {\n  for (int i = 0; i < s.length(); ) {\n    int cpt = s.codePointAt(i);\n    if (cpt >= 0xD800 && cpt <= 0xDFFF) return false;\n    if (cpt < 0 || cpt > 0x10FFFF) return false;\n    i += Character.charCount(cpt);\n  }\n  return true;\n}","typeGuard":null,"tryCatchPattern":"catch IOException from toBinaryString/record string writes; identify the surrogate-bearing field from the hex codepoint in the message and scrub or reject the value.","preventionTips":["Decode source bytes with the correct charset (usually UTF-8) so unpaired surrogates never enter Strings","Sanitize strings (replace unpaired surrogates with U+FFFD) at ingestion boundaries","Avoid substring() on data that may contain supplementary characters"],"tags":["hadoop-record","unicode","utf-8","serialization","deprecated"],"backgroundTag":"invalid-unicode-codepoint","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}