{"record":{"id":"e4e389bcf5eee1d1","repo":"apache/hadoop","slug":"invalid-utf8-at","errorCode":null,"errorMessage":"Invalid UTF8 at {}","messagePattern":"Invalid UTF8 at (.+?)","errorType":"exception","errorClass":"UTFDataFormatException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/UTF8.java","lineNumber":328,"sourceCode":"          throw new UTFDataFormatException(\"Truncated UTF8 at \" +\n              StringUtils.byteToHexString(bytes, i - 1, 3));\n        }\n        // 0b11110xxx: 4-byte sequence\n        int codepoint =\n            ((b & 0x07) << 18)\n          | ((bytes[i++] & 0x3F) <<  12)\n          | ((bytes[i++] & 0x3F) <<  6)\n          | ((bytes[i++] & 0x3F));\n        buffer.append(highSurrogate(codepoint))\n              .append(lowSurrogate(codepoint));\n      } else {\n        // The UTF8 standard describes 5-byte and 6-byte sequences, but\n        // these are no longer allowed as of 2003 (see RFC 3629)\n\n        // Only show the next 6 bytes max in the error code - in case the\n        // buffer is large, this will prevent an exceedingly large message.\n        int endForError = Math.min(i + 5, nBytes);\n        throw new UTFDataFormatException(\"Invalid UTF8 at \" +\n            StringUtils.byteToHexString(bytes, i - 1, endForError));\n      }\n    }\n  }\n\n  private static char highSurrogate(int codePoint) {\n    return (char) ((codePoint >>> 10)\n        + (Character.MIN_HIGH_SURROGATE - (Character.MIN_SUPPLEMENTARY_CODE_POINT >>> 10)));\n  }\n\n  private static char lowSurrogate(int codePoint) {\n    return (char) ((codePoint & 0x3ff) + Character.MIN_LOW_SURROGATE);\n  }\n\n  /**\n   * @return Write a UTF-8 encoded string.\n   *\n   * @see DataOutput#writeUTF(String)","sourceCodeStart":310,"sourceCodeEnd":346,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/UTF8.java#L310-L346","documentation":"Thrown as UTFDataFormatException by UTF8.readChars, the legacy modified-UTF-8 decoder behind UTF8.fromBytes and UTF8.readString. While walking the byte array it hit a lead byte of 0b111110xx or 0b111111xx, which would start a 5- or 6-byte sequence. RFC 3629 removed those forms from UTF-8 in 2003, so Hadoop rejects them instead of decoding. The hex snippet in the message shows the offending bytes so you can identify what actually produced them.","triggerScenarios":"Calling UTF8.fromBytes(byte[]) or UTF8.readString(DataInput) on bytes containing a lead byte >= 0xF8; feeding binary data, non-UTF-8 text (e.g., UTF-16 with 0x00/0xFF bytes, or random/corrupted bytes) into these APIs; reading a stream at the wrong offset so string length and payload desynchronize.","commonSituations":"Corrupted files or truncated transfers being re-read; a Java DataOutput.writeUTF producer mixed with bytes from another charset; payloads written by a non-Java system that emits pre-RFC-3629 5/6-byte encodings or CESU-8-style encodings; deserializing Writable fields where the preceding vint/length was misread.","solutions":["Inspect the hex bytes in the message to identify the true encoding of the input (UTF-16 BOM bytes like FE FF, 0x00 padding, or binary garbage all point to a producer-side bug).","Fix the producer to emit standard UTF-8 (RFC 3629: max 4-byte sequences), or re-encode the data before it reaches Hadoop serialization.","If the input is genuinely another charset, decode it yourself with new String(bytes, Charset) or java.nio.charset.CharsetDecoder instead of UTF8.fromBytes.","If the data is a Hadoop record, verify you are reading the matching writer format and correct offsets — the bad lead byte is often just a desynchronized stream position."],"exampleFix":"// before\nString s = UTF8.fromBytes(bytes); // throws UTFDataFormatException on 0xF8+ lead bytes\n\n// after\nString s = new String(bytes, java.nio.charset.StandardCharsets.UTF_8);\n// java.lang.String replaces malformed sequences with U+FFFD instead of throwing;\n// pre-validate with CharsetDecoder if you need strict rejection.","handlingStrategy":"try-catch","validationCode":"static boolean isValidModifiedUtf8(byte[] bytes, int off, int len) {\n  int i = off, end = off + len;\n  while (i < end) {\n    int b = bytes[i] & 0xFF;\n    if (b < 0x80) i += 1;\n    else if ((b & 0xE0) == 0xC0) i += 2;\n    else if ((b & 0xF0) == 0xE0) i += 3;\n    else if ((b & 0xF8) == 0xF0) i += 4;\n    else return false;          // 0xF8+ lead byte -> UTFDataFormatException\n    if (i > end) return false;  // truncated sequence\n  }\n  return true;\n}","typeGuard":null,"tryCatchPattern":"try {\n  String s = UTF8.fromBytes(bytes);\n} catch (UTFDataFormatException e) {\n  // e.g. log offending byte offset, fall back to lenient decode\n  LOG.warn(\"invalid modified UTF-8 payload, lenient decode\", e);\n  return new String(bytes, StandardCharsets.UTF_8);\n}","preventionTips":["Validate producer-side encoding: always write strings as UTF-8 via Text or StandardCharsets.UTF_8.","Never feed raw sockets/binary blobs into UTF8.readString without framing checks.","Pre-validate untrusted bytes with a CharsetDecoder configured with CodingErrorAction.REPORT."],"tags":["utf8","encoding","serialization","text","corrupted-data"],"backgroundTag":"invalid-utf8-decoding","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}