apache/hadoop · error · IOException

Invalid UTF-8 byte {} at offset {} in length of {}

Error message

Invalid UTF-8 byte {} at offset {} in length of {}

What it means

In Utils.fromBinaryString(), the first byte of each character must be a valid UTF-8 lead byte: 0xxxxxxx, 110xxxxx, 1110xxxx, or 11110xxx. A byte matching none of these (e.g. 0x80-0xBF used as a lead, or 0xF8+) triggers IOException 'Invalid UTF-8 byte <hex> at offset <n> in length of <len>' — the message carries the offending byte, its offset, and the declared UTF-8 length for pinpointing the corruption.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java:391

        int b2 = bytes[len++] & 0xFF;
        checkB10(b2);
        int b3 = bytes[len++] & 0xFF;
        checkB10(b3);
        int b4 = bytes[len++] & 0xFF;
        checkB10(b4);
        cpt = utf8ToCodePoint(b1, b2, b3, b4);
      } else if ((b1 & B1111) == B1110) {
        int b2 = bytes[len++] & 0xFF;
        checkB10(b2);
        int b3 = bytes[len++] & 0xFF;
        checkB10(b3);
        cpt = utf8ToCodePoint(b1, b2, b3);
      } else if ((b1 & B111) == B110) {
        int b2 = bytes[len++] & 0xFF;
        checkB10(b2);
        cpt = utf8ToCodePoint(b1, b2);
      } else {
        throw new IOException("Invalid UTF-8 byte "+Integer.toHexString(b1)+
                              " at offset "+(len-1)+" in length of "+utf8Len);
      }
      if (!isValidCodePoint(cpt)) {
        throw new IOException("Illegal Unicode Codepoint "+
                              Integer.toHexString(cpt)+" in stream.");
      }
      sb.appendCodePoint(cpt);
    }
    return sb.toString();
  }
  
  /** Parse a float from a byte array. */
  public static float readFloat(byte[] bytes, int start) {
    return WritableComparator.readFloat(bytes, start);
  }
  
  /** Parse a double from a byte array. */
  public static double readDouble(byte[] bytes, int start) {

View on GitHub (pinned to 2add963021)

Solutions

1. Use the hex byte and offset in the message to inspect the exact spot in the payload and confirm whether the field should even be a string
2. Verify writer/reader field order and types match (a misparse shifts every subsequent field)
3. Regenerate the data with matched library versions instead of patching bytes
4. Pre-validate the whole buffer with a strict UTF-8 decoder to fail with better diagnostics

// same strict-decode precheck as continuation bytes; additionally sanity-check the length prefix matches remaining bytes
if (utf8Len > remaining) throw new CorruptRecordException("length prefix " + utf8Len + " exceeds buffer");
if (!isStrictUtf8(Arrays.copyOfRange(bytes, off, off + utf8Len))) throw new CorruptRecordException("non-UTF-8 string field");

catch IOException from fromBinaryString and use the byte/offset/length values in the message to locate and isolate the corrupt record; fail the batch only if corruption is systematic.

When it happens

Trigger: Decoding a length-prefixed binary record whose payload is not UTF-8 at the stated offset: pure binary data read as a string field, a misaligned parse consuming garbage after a framing error, or truncated/recombined streams.

Common situations: Schema mismatch where a buffer/int field is read as a string, off-by-one length prefixes after manual record assembly, files mangled in transfer (encoding conversions or CRLF injection), or incompatible writer versions.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Invalid UTF-8 representation.
• Illegal Unicode Codepoint {} in stream.
• Error deserializing string.
• Error deserializing buffer.
• Illegal Unicode Codepoint {} in string.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c92e28fc8cbb6652. Report an issue: GitHub.