apache/hadoop · error · IOException

Invalid UTF-8 representation.

Error message

Invalid UTF-8 representation.

What it means

checkB10() validates that a byte following a multi-byte UTF-8 lead has the continuation pattern 10xxxxxx ((b & 11) == 10). fromBinaryString() calls it for every continuation byte; a mismatch means the byte sequence is not well-formed UTF-8 and decoding cannot proceed, so it throws IOException 'Invalid UTF-8 representation.'.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java:356

           (b4 & ~B11));
    return cpt;
  }
  
  private static int utf8ToCodePoint(int b1, int b2, int b3) {
    int cpt = 0;
    cpt = (((b1 & ~B1111) << 12) | ((b2 & ~B11) << 6) | (b3 & ~B11));
    return cpt;
  }
  
  private static int utf8ToCodePoint(int b1, int b2) {
    int cpt = 0;
    cpt = (((b1 & ~B111) << 6) | (b2 & ~B11));
    return cpt;
  }
  
  private static void checkB10(int b) throws IOException {
    if ((b & B11) != B10) {
      throw new IOException("Invalid UTF-8 representation.");
    }
  }
  
  static String fromBinaryString(final DataInput din) throws IOException {
    final int utf8Len = readVInt(din);
    final byte[] bytes = new byte[utf8Len];
    din.readFully(bytes);
    int len = 0;
    // For the most commmon case, i.e. ascii, numChars = utf8Len
    StringBuilder sb = new StringBuilder(utf8Len);
    while(len < utf8Len) {
      int cpt = 0;
      final int b1 = bytes[len++] & 0xFF;
      if (b1 <= 0x7F) {
        cpt = b1;
      } else if ((b1 & B11111) == B11110) {
        int b2 = bytes[len++] & 0xFF;
        checkB10(b2);

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure the writer encoded the payload as UTF-8 (Utils.toBinaryString) — same library version on both ends
2. Validate the byte region with java.nio.charset.CharsetDecoder REPORT mode or String/UTF-8 strict decode before parsing records
3. Check the length prefix and framing logic if data may be truncated (the decoder reading past intended field boundaries)
4. Retire org.apache.hadoop.record in favor of Avro/Writable formats with robust UTF-8 handling

Example fix

// before
String s = Utils.fromBinaryString(din);
// after: pre-validate the bytes when parsing untrusted buffers
byte[] b = /* candidate utf8 bytes */;
try {
  CharsetDecoder dec = StandardCharsets.UTF_8.newDecoder()
      .onMalformedInput(CodingErrorAction.REPORT)
      .onUnmappableCharacter(CodingErrorAction.REPORT);
  dec.decode(ByteBuffer.wrap(b));
  String s = Utils.fromBinaryString(din);
} catch (CharacterCodingException e) { /* reject record */ }

static boolean isStrictUtf8(byte[] b) {
  try {
    StandardCharsets.UTF_8.newDecoder()
        .onMalformedInput(CodingErrorAction.REPORT)
        .onUnmappableCharacter(CodingErrorAction.REPORT)
        .decode(ByteBuffer.wrap(b));
    return true;
  } catch (CharacterCodingException e) { return false; }
}

catch IOException from fromBinaryString; treat as corrupt record — quarantine it, log byte offset context, and continue with the next record rather than aborting the whole stream.

When it happens

Trigger: fromBinaryString (legacy record binary deserialization) reads a length-prefixed byte array whose contents are not UTF-8: Latin-1/Windows-1252 text, a sequence truncated mid-character, or bytes written by a different encoder, so the byte after a 2/3/4-byte lead is a standalone ASCII or lead byte instead of a continuation.

Common situations: Producer/consumer charset disagreement (writer encoded with default platform charset on Windows), stream truncation or corruption, buffer offset bugs when hand-assembling record payloads, or data produced by an incompatible Hadoop version.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Invalid UTF-8 byte {} at offset {} in length of {}
• Illegal Unicode Codepoint {} in stream.
• Error deserializing string.
• Error deserializing buffer.
• Illegal Unicode Codepoint {} in string.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a72a7c28aeb9e12c. Report an issue: GitHub.