apache/hadoop · error · IOException

Illegal Unicode Codepoint {} in stream.

Error message

Illegal Unicode Codepoint {} in stream.

What it means

Final validation in Utils.fromBinaryString(): after decoding a multi-byte sequence into codepoint cpt, isValidCodePoint(cpt) is checked. A structurally valid byte pattern that decodes into the surrogate range 0xD800-0xDFFF (or another disallowed value) throws IOException 'Illegal Unicode Codepoint <hex> in stream.' — the bytes were syntactically UTF-8-ish but encode a codepoint that cannot appear in well-formed text.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java:395

        int b4 = bytes[len++] & 0xFF;
        checkB10(b4);
        cpt = utf8ToCodePoint(b1, b2, b3, b4);
      } else if ((b1 & B1111) == B1110) {
        int b2 = bytes[len++] & 0xFF;
        checkB10(b2);
        int b3 = bytes[len++] & 0xFF;
        checkB10(b3);
        cpt = utf8ToCodePoint(b1, b2, b3);
      } else if ((b1 & B111) == B110) {
        int b2 = bytes[len++] & 0xFF;
        checkB10(b2);
        cpt = utf8ToCodePoint(b1, b2);
      } else {
        throw new IOException("Invalid UTF-8 byte "+Integer.toHexString(b1)+
                              " at offset "+(len-1)+" in length of "+utf8Len);
      }
      if (!isValidCodePoint(cpt)) {
        throw new IOException("Illegal Unicode Codepoint "+
                              Integer.toHexString(cpt)+" in stream.");
      }
      sb.appendCodePoint(cpt);
    }
    return sb.toString();
  }
  
  /** Parse a float from a byte array. */
  public static float readFloat(byte[] bytes, int start) {
    return WritableComparator.readFloat(bytes, start);
  }
  
  /** Parse a double from a byte array. */
  public static double readDouble(byte[] bytes, int start) {
    return WritableComparator.readDouble(bytes, start);
  }
  
  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Re-encode the source data as standard UTF-8 (e.g., decode-and-reencode through a strict decoder plus encoder) before feeding record serialization
2. If CESU-8/modified-UTF-8 sources are unavoidable, convert them with a dedicated transcoder upstream
3. Treat occurrences of this error as data-integrity incidents: capture and quarantine the offending records using the hex codepoint in the message
4. Migrate the pipeline off org.apache.hadoop.record

// reject data containing encoded surrogate codepoints (CESU-8 style ED A0-BF ..) before record parsing
static boolean containsEncodedSurrogates(byte[] b) {
  for (int i = 0; i + 2 < b.length; i++) {
    if ((b[i] & 0xFF) == 0xED && (b[i+1] & 0xE0) == 0xA0) return true;
  }
  return false;
}

catch IOException from fromBinaryString; on 'Illegal Unicode Codepoint ... in stream' quarantine the record and alert on data provenance (which source emits CESU-8/modified UTF-8).

When it happens

Trigger: A payload containing CESU-8 or intentionally encoded surrogate codepoints (ED A0 80-style sequences), or bytes that decode past the allowed ranges; each character's decoded value is rejected before appendCodePoint.

Common situations: Data produced by encoders that emit surrogate codepoints (CESU-8, some Java Modified-UTF-8 writers), payload corruption that happens to form 3-byte sequences, or hostile/fuzzed input to legacy record readers.

Related errors

• Illegal Unicode Codepoint {} in string.
• Invalid UTF-8 representation.
• Invalid UTF-8 byte {} at offset {} in length of {}
• Error deserializing string.
• Error deserializing buffer.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0abe9f50a5a792fd. Report an issue: GitHub.