apache/hadoop · error · IOException

Illegal Unicode Codepoint {} in string.

Error message

Illegal Unicode Codepoint {} in string.

What it means

Utils computes UTF-8 byte length for a codepoint (used by toBinaryString when sizing the encoding of legacy record strings). Codepoints accepted are < 0x80, 0x80-0x07FF, 0x0800-0xD7FF, 0xE000-0xFFFD, and 0x10000-0x10FFFF. Anything else — lone surrogates (0xD800-0xDFFF), values above 0x10FFFF, or negative values — throws IOException 'Illegal Unicode Codepoint <hex> in string.'.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java:264

    }
    return new Buffer(barr);
  }
  
  private static int utf8LenForCodePoint(final int cpt) throws IOException {
    if (cpt >=0 && cpt <= 0x7F) {
      return 1;
    }
    if (cpt >= 0x80 && cpt <= 0x07FF) {
      return 2;
    }
    if ((cpt >= 0x0800 && cpt < 0xD800) ||
        (cpt > 0xDFFF && cpt <= 0xFFFD)) {
      return 3;
    }
    if (cpt >= 0x10000 && cpt <= 0x10FFFF) {
      return 4;
    }
    throw new IOException("Illegal Unicode Codepoint "+
                          Integer.toHexString(cpt)+" in string.");
  }
  
  private static final int B10 =    Integer.parseInt("10000000", 2);
  private static final int B110 =   Integer.parseInt("11000000", 2);
  private static final int B1110 =  Integer.parseInt("11100000", 2);
  private static final int B11110 = Integer.parseInt("11110000", 2);
  private static final int B11 =    Integer.parseInt("11000000", 2);
  private static final int B111 =   Integer.parseInt("11100000", 2);
  private static final int B1111 =  Integer.parseInt("11110000", 2);
  private static final int B11111 = Integer.parseInt("11111000", 2);
  
  private static int writeUtf8(int cpt, final byte[] bytes, final int offset)
    throws IOException {
    if (cpt >=0 && cpt <= 0x7F) {
      bytes[offset] = (byte) cpt;
      return 1;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Sanitize strings before serialization: replace unpaired surrogates (Character.isHighSurrogate/isLowSurrogate without their counterpart) with U+FFFD
2. Fix the upstream decode: read the source bytes as UTF-8 (or the true charset) instead of constructing chars ad hoc
3. Avoid slicing strings on char boundaries where supplementary characters may exist; slice on codepoint boundaries
4. Locate the offending value by catching the exception and logging the string around the failing index

Example fix

// before
String out = s; // s may contain unpaired surrogates
// after
StringBuilder sb = new StringBuilder(s.length());
for (int i = 0; i < s.length(); i++) {
  char c = s.charAt(i);
  if (Character.isSurrogate(c)) { sb.append('\uFFFD'); }
  else { sb.append(c); }
}

static boolean allCodePointsEncodable(String s) {
  for (int i = 0; i < s.length(); ) {
    int cpt = s.codePointAt(i);
    if (cpt >= 0xD800 && cpt <= 0xDFFF) return false;
    if (cpt < 0 || cpt > 0x10FFFF) return false;
    i += Character.charCount(cpt);
  }
  return true;
}

catch IOException from toBinaryString/record string writes; identify the surrogate-bearing field from the hex codepoint in the message and scrub or reject the value.

When it happens

Trigger: Serializing a Java String that contains an unpaired surrogate char (half of a broken surrogate pair), which occurs when the string was built from data decoded with a charset mismatch or sliced mid-surrogate; the codepoint fails every valid range check.

Common situations: Strings decoded from bytes with ISO-8859-1/UTF-8 mismatch, substring() cuts splitting a surrogate pair, data read from sources that emit CESU-8 or raw surrogate code units, or upstream files already containing invalid UTF-8 that got laundered into Java chars.

Related errors

• Illegal Unicode Codepoint {} in stream.
• Error serializing {}
• Invalid UTF-8 representation.
• Invalid UTF-8 byte {} at offset {} in length of {}
• string was too long to write! Expected less than or equal t

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d84372d6194bfc5d. Report an issue: GitHub.