{"record":{"id":"c92e28fc8cbb6652","repo":"apache/hadoop","slug":"invalid-utf-8-byte-at-offset-in-length-of","errorCode":null,"errorMessage":"Invalid UTF-8 byte {} at offset {} in length of {}","messagePattern":"Invalid UTF-8 byte (.+?) at offset (.+?) in length of (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java","lineNumber":391,"sourceCode":"        int b2 = bytes[len++] & 0xFF;\n        checkB10(b2);\n        int b3 = bytes[len++] & 0xFF;\n        checkB10(b3);\n        int b4 = bytes[len++] & 0xFF;\n        checkB10(b4);\n        cpt = utf8ToCodePoint(b1, b2, b3, b4);\n      } else if ((b1 & B1111) == B1110) {\n        int b2 = bytes[len++] & 0xFF;\n        checkB10(b2);\n        int b3 = bytes[len++] & 0xFF;\n        checkB10(b3);\n        cpt = utf8ToCodePoint(b1, b2, b3);\n      } else if ((b1 & B111) == B110) {\n        int b2 = bytes[len++] & 0xFF;\n        checkB10(b2);\n        cpt = utf8ToCodePoint(b1, b2);\n      } else {\n        throw new IOException(\"Invalid UTF-8 byte \"+Integer.toHexString(b1)+\n                              \" at offset \"+(len-1)+\" in length of \"+utf8Len);\n      }\n      if (!isValidCodePoint(cpt)) {\n        throw new IOException(\"Illegal Unicode Codepoint \"+\n                              Integer.toHexString(cpt)+\" in stream.\");\n      }\n      sb.appendCodePoint(cpt);\n    }\n    return sb.toString();\n  }\n  \n  /** Parse a float from a byte array. */\n  public static float readFloat(byte[] bytes, int start) {\n    return WritableComparator.readFloat(bytes, start);\n  }\n  \n  /** Parse a double from a byte array. */\n  public static double readDouble(byte[] bytes, int start) {","sourceCodeStart":373,"sourceCodeEnd":409,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java#L373-L409","documentation":"In Utils.fromBinaryString(), the first byte of each character must be a valid UTF-8 lead byte: 0xxxxxxx, 110xxxxx, 1110xxxx, or 11110xxx. A byte matching none of these (e.g. 0x80-0xBF used as a lead, or 0xF8+) triggers IOException 'Invalid UTF-8 byte <hex> at offset <n> in length of <len>' — the message carries the offending byte, its offset, and the declared UTF-8 length for pinpointing the corruption.","triggerScenarios":"Decoding a length-prefixed binary record whose payload is not UTF-8 at the stated offset: pure binary data read as a string field, a misaligned parse consuming garbage after a framing error, or truncated/recombined streams.","commonSituations":"Schema mismatch where a buffer/int field is read as a string, off-by-one length prefixes after manual record assembly, files mangled in transfer (encoding conversions or CRLF injection), or incompatible writer versions.","solutions":["Use the hex byte and offset in the message to inspect the exact spot in the payload and confirm whether the field should even be a string","Verify writer/reader field order and types match (a misparse shifts every subsequent field)","Regenerate the data with matched library versions instead of patching bytes","Pre-validate the whole buffer with a strict UTF-8 decoder to fail with better diagnostics"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// same strict-decode precheck as continuation bytes; additionally sanity-check the length prefix matches remaining bytes\nif (utf8Len > remaining) throw new CorruptRecordException(\"length prefix \" + utf8Len + \" exceeds buffer\");\nif (!isStrictUtf8(Arrays.copyOfRange(bytes, off, off + utf8Len))) throw new CorruptRecordException(\"non-UTF-8 string field\");","typeGuard":null,"tryCatchPattern":"catch IOException from fromBinaryString and use the byte/offset/length values in the message to locate and isolate the corrupt record; fail the batch only if corruption is systematic.","preventionTips":["Ensure writer and reader agree on field order and types so parses never go out of alignment","Reject or transcode non-UTF-8 sources before they reach record serialization","When hand-assembling records, add a framing test with multi-byte characters"],"tags":["hadoop-record","utf-8","deserialization","deprecated","data-corruption"],"backgroundTag":"invalid-utf8","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}