{"record":{"id":"0abe9f50a5a792fd","repo":"apache/hadoop","slug":"illegal-unicode-codepoint-in-stream","errorCode":null,"errorMessage":"Illegal Unicode Codepoint {} in stream.","messagePattern":"Illegal Unicode Codepoint (.+?) in stream\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java","lineNumber":395,"sourceCode":"        int b4 = bytes[len++] & 0xFF;\n        checkB10(b4);\n        cpt = utf8ToCodePoint(b1, b2, b3, b4);\n      } else if ((b1 & B1111) == B1110) {\n        int b2 = bytes[len++] & 0xFF;\n        checkB10(b2);\n        int b3 = bytes[len++] & 0xFF;\n        checkB10(b3);\n        cpt = utf8ToCodePoint(b1, b2, b3);\n      } else if ((b1 & B111) == B110) {\n        int b2 = bytes[len++] & 0xFF;\n        checkB10(b2);\n        cpt = utf8ToCodePoint(b1, b2);\n      } else {\n        throw new IOException(\"Invalid UTF-8 byte \"+Integer.toHexString(b1)+\n                              \" at offset \"+(len-1)+\" in length of \"+utf8Len);\n      }\n      if (!isValidCodePoint(cpt)) {\n        throw new IOException(\"Illegal Unicode Codepoint \"+\n                              Integer.toHexString(cpt)+\" in stream.\");\n      }\n      sb.appendCodePoint(cpt);\n    }\n    return sb.toString();\n  }\n  \n  /** Parse a float from a byte array. */\n  public static float readFloat(byte[] bytes, int start) {\n    return WritableComparator.readFloat(bytes, start);\n  }\n  \n  /** Parse a double from a byte array. */\n  public static double readDouble(byte[] bytes, int start) {\n    return WritableComparator.readDouble(bytes, start);\n  }\n  \n  /**","sourceCodeStart":377,"sourceCodeEnd":413,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java#L377-L413","documentation":"Final validation in Utils.fromBinaryString(): after decoding a multi-byte sequence into codepoint cpt, isValidCodePoint(cpt) is checked. A structurally valid byte pattern that decodes into the surrogate range 0xD800-0xDFFF (or another disallowed value) throws IOException 'Illegal Unicode Codepoint <hex> in stream.' — the bytes were syntactically UTF-8-ish but encode a codepoint that cannot appear in well-formed text.","triggerScenarios":"A payload containing CESU-8 or intentionally encoded surrogate codepoints (ED A0 80-style sequences), or bytes that decode past the allowed ranges; each character's decoded value is rejected before appendCodePoint.","commonSituations":"Data produced by encoders that emit surrogate codepoints (CESU-8, some Java Modified-UTF-8 writers), payload corruption that happens to form 3-byte sequences, or hostile/fuzzed input to legacy record readers.","solutions":["Re-encode the source data as standard UTF-8 (e.g., decode-and-reencode through a strict decoder plus encoder) before feeding record serialization","If CESU-8/modified-UTF-8 sources are unavoidable, convert them with a dedicated transcoder upstream","Treat occurrences of this error as data-integrity incidents: capture and quarantine the offending records using the hex codepoint in the message","Migrate the pipeline off org.apache.hadoop.record"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// reject data containing encoded surrogate codepoints (CESU-8 style ED A0-BF ..) before record parsing\nstatic boolean containsEncodedSurrogates(byte[] b) {\n  for (int i = 0; i + 2 < b.length; i++) {\n    if ((b[i] & 0xFF) == 0xED && (b[i+1] & 0xE0) == 0xA0) return true;\n  }\n  return false;\n}","typeGuard":null,"tryCatchPattern":"catch IOException from fromBinaryString; on 'Illegal Unicode Codepoint ... in stream' quarantine the record and alert on data provenance (which source emits CESU-8/modified UTF-8).","preventionTips":["Transcode CESU-8 / modified-UTF-8 / Java DataOutput UTF sources to standard UTF-8 upstream","Run strict UTF-8 validation plus the surrogate check at ingestion boundaries","Fuzz-test legacy record readers with hostile inputs to confirm quarantining works"],"tags":["hadoop-record","unicode","utf-8","deserialization","deprecated"],"backgroundTag":"invalid-unicode-codepoint","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}