{"record":{"id":"a72a7c28aeb9e12c","repo":"apache/hadoop","slug":"invalid-utf-8-representation","errorCode":null,"errorMessage":"Invalid UTF-8 representation.","messagePattern":"Invalid UTF-8 representation\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java","lineNumber":356,"sourceCode":"           (b4 & ~B11));\n    return cpt;\n  }\n  \n  private static int utf8ToCodePoint(int b1, int b2, int b3) {\n    int cpt = 0;\n    cpt = (((b1 & ~B1111) << 12) | ((b2 & ~B11) << 6) | (b3 & ~B11));\n    return cpt;\n  }\n  \n  private static int utf8ToCodePoint(int b1, int b2) {\n    int cpt = 0;\n    cpt = (((b1 & ~B111) << 6) | (b2 & ~B11));\n    return cpt;\n  }\n  \n  private static void checkB10(int b) throws IOException {\n    if ((b & B11) != B10) {\n      throw new IOException(\"Invalid UTF-8 representation.\");\n    }\n  }\n  \n  static String fromBinaryString(final DataInput din) throws IOException {\n    final int utf8Len = readVInt(din);\n    final byte[] bytes = new byte[utf8Len];\n    din.readFully(bytes);\n    int len = 0;\n    // For the most commmon case, i.e. ascii, numChars = utf8Len\n    StringBuilder sb = new StringBuilder(utf8Len);\n    while(len < utf8Len) {\n      int cpt = 0;\n      final int b1 = bytes[len++] & 0xFF;\n      if (b1 <= 0x7F) {\n        cpt = b1;\n      } else if ((b1 & B11111) == B11110) {\n        int b2 = bytes[len++] & 0xFF;\n        checkB10(b2);","sourceCodeStart":338,"sourceCodeEnd":374,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java#L338-L374","documentation":"checkB10() validates that a byte following a multi-byte UTF-8 lead has the continuation pattern 10xxxxxx ((b & 11) == 10). fromBinaryString() calls it for every continuation byte; a mismatch means the byte sequence is not well-formed UTF-8 and decoding cannot proceed, so it throws IOException 'Invalid UTF-8 representation.'.","triggerScenarios":"fromBinaryString (legacy record binary deserialization) reads a length-prefixed byte array whose contents are not UTF-8: Latin-1/Windows-1252 text, a sequence truncated mid-character, or bytes written by a different encoder, so the byte after a 2/3/4-byte lead is a standalone ASCII or lead byte instead of a continuation.","commonSituations":"Producer/consumer charset disagreement (writer encoded with default platform charset on Windows), stream truncation or corruption, buffer offset bugs when hand-assembling record payloads, or data produced by an incompatible Hadoop version.","solutions":["Ensure the writer encoded the payload as UTF-8 (Utils.toBinaryString) — same library version on both ends","Validate the byte region with java.nio.charset.CharsetDecoder REPORT mode or String/UTF-8 strict decode before parsing records","Check the length prefix and framing logic if data may be truncated (the decoder reading past intended field boundaries)","Retire org.apache.hadoop.record in favor of Avro/Writable formats with robust UTF-8 handling"],"exampleFix":"// before\nString s = Utils.fromBinaryString(din);\n// after: pre-validate the bytes when parsing untrusted buffers\nbyte[] b = /* candidate utf8 bytes */;\ntry {\n  CharsetDecoder dec = StandardCharsets.UTF_8.newDecoder()\n      .onMalformedInput(CodingErrorAction.REPORT)\n      .onUnmappableCharacter(CodingErrorAction.REPORT);\n  dec.decode(ByteBuffer.wrap(b));\n  String s = Utils.fromBinaryString(din);\n} catch (CharacterCodingException e) { /* reject record */ }","handlingStrategy":"validation","validationCode":"static boolean isStrictUtf8(byte[] b) {\n  try {\n    StandardCharsets.UTF_8.newDecoder()\n        .onMalformedInput(CodingErrorAction.REPORT)\n        .onUnmappableCharacter(CodingErrorAction.REPORT)\n        .decode(ByteBuffer.wrap(b));\n    return true;\n  } catch (CharacterCodingException e) { return false; }\n}","typeGuard":null,"tryCatchPattern":"catch IOException from fromBinaryString; treat as corrupt record — quarantine it, log byte offset context, and continue with the next record rather than aborting the whole stream.","preventionTips":["Pin every producer to UTF-8 (set -Dfile.encoding, use explicit OutputStreamWriter with UTF_8)","Validate payloads at trust boundaries with a strict decoder before record parsing","Version or checksum serialized blobs exchanged between systems"],"tags":["hadoop-record","utf-8","deserialization","deprecated","data-corruption"],"backgroundTag":"invalid-utf8","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}