apache/hadoop · error · UTFDataFormatException

Invalid UTF8 at {}

Error message

Invalid UTF8 at {}

What it means

Thrown as UTFDataFormatException by UTF8.readChars, the legacy modified-UTF-8 decoder behind UTF8.fromBytes and UTF8.readString. While walking the byte array it hit a lead byte of 0b111110xx or 0b111111xx, which would start a 5- or 6-byte sequence. RFC 3629 removed those forms from UTF-8 in 2003, so Hadoop rejects them instead of decoding. The hex snippet in the message shows the offending bytes so you can identify what actually produced them.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/UTF8.java:328

          throw new UTFDataFormatException("Truncated UTF8 at " +
              StringUtils.byteToHexString(bytes, i - 1, 3));
        }
        // 0b11110xxx: 4-byte sequence
        int codepoint =
            ((b & 0x07) << 18)
          | ((bytes[i++] & 0x3F) <<  12)
          | ((bytes[i++] & 0x3F) <<  6)
          | ((bytes[i++] & 0x3F));
        buffer.append(highSurrogate(codepoint))
              .append(lowSurrogate(codepoint));
      } else {
        // The UTF8 standard describes 5-byte and 6-byte sequences, but
        // these are no longer allowed as of 2003 (see RFC 3629)

        // Only show the next 6 bytes max in the error code - in case the
        // buffer is large, this will prevent an exceedingly large message.
        int endForError = Math.min(i + 5, nBytes);
        throw new UTFDataFormatException("Invalid UTF8 at " +
            StringUtils.byteToHexString(bytes, i - 1, endForError));
      }
    }
  }

  private static char highSurrogate(int codePoint) {
    return (char) ((codePoint >>> 10)
        + (Character.MIN_HIGH_SURROGATE - (Character.MIN_SUPPLEMENTARY_CODE_POINT >>> 10)));
  }

  private static char lowSurrogate(int codePoint) {
    return (char) ((codePoint & 0x3ff) + Character.MIN_LOW_SURROGATE);
  }

  /**
   * @return Write a UTF-8 encoded string.
   *
   * @see DataOutput#writeUTF(String)

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the hex bytes in the message to identify the true encoding of the input (UTF-16 BOM bytes like FE FF, 0x00 padding, or binary garbage all point to a producer-side bug).
2. Fix the producer to emit standard UTF-8 (RFC 3629: max 4-byte sequences), or re-encode the data before it reaches Hadoop serialization.
3. If the input is genuinely another charset, decode it yourself with new String(bytes, Charset) or java.nio.charset.CharsetDecoder instead of UTF8.fromBytes.
4. If the data is a Hadoop record, verify you are reading the matching writer format and correct offsets — the bad lead byte is often just a desynchronized stream position.

Example fix

// before
String s = UTF8.fromBytes(bytes); // throws UTFDataFormatException on 0xF8+ lead bytes

// after
String s = new String(bytes, java.nio.charset.StandardCharsets.UTF_8);
// java.lang.String replaces malformed sequences with U+FFFD instead of throwing;
// pre-validate with CharsetDecoder if you need strict rejection.

static boolean isValidModifiedUtf8(byte[] bytes, int off, int len) {
  int i = off, end = off + len;
  while (i < end) {
    int b = bytes[i] & 0xFF;
    if (b < 0x80) i += 1;
    else if ((b & 0xE0) == 0xC0) i += 2;
    else if ((b & 0xF0) == 0xE0) i += 3;
    else if ((b & 0xF8) == 0xF0) i += 4;
    else return false;          // 0xF8+ lead byte -> UTFDataFormatException
    if (i > end) return false;  // truncated sequence
  }
  return true;
}

try {
  String s = UTF8.fromBytes(bytes);
} catch (UTFDataFormatException e) {
  // e.g. log offending byte offset, fall back to lenient decode
  LOG.warn("invalid modified UTF-8 payload, lenient decode", e);
  return new String(bytes, StandardCharsets.UTF_8);
}

When it happens

Trigger: Calling UTF8.fromBytes(byte[]) or UTF8.readString(DataInput) on bytes containing a lead byte >= 0xF8; feeding binary data, non-UTF-8 text (e.g., UTF-16 with 0x00/0xFF bytes, or random/corrupted bytes) into these APIs; reading a stream at the wrong offset so string length and payload desynchronize.

Common situations: Corrupted files or truncated transfers being re-read; a Java DataOutput.writeUTF producer mixed with bytes from another charset; payloads written by a non-Java system that emits pre-RFC-3629 5/6-byte encodings or CESU-8-style encodings; deserializing Writable fields where the preceding vint/length was misread.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• data was too long to write! Expected less than or equal to
• string was too long to write! Expected less than or equal t
• string too long!
• parent + " is a file"
• Parent directory doesn't exist: " + parent

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e4e389bcf5eee1d1. Report an issue: GitHub.