{"record":{"id":"1da5cad8ca650fec","repo":"antlr/antlr4","slug":"invalid-utf-16-high-surrogate-followed-by-code-po","errorCode":null,"errorMessage":"Invalid UTF-16 (high surrogate followed by code point > U+FFFF","messagePattern":"Invalid UTF-16 \\(high surrogate followed by code point > U\\+FFFF","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java","lineNumber":177,"sourceCode":"\t\tfor (int i=0; i<n; i++) {\n\t\t\tif (this.n > 0 && data[this.n - 1] == IntStream.EOF) {\n\t\t\t\treturn i;\n\t\t\t}\n\n\t\t\ttry {\n\t\t\t\tint c = nextChar();\n\t\t\t\tif (c > Character.MAX_VALUE || c == IntStream.EOF) {\n\t\t\t\t\tadd(c);\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tchar ch = (char) c;\n\t\t\t\t\tif (Character.isLowSurrogate(ch)) {\n\t\t\t\t\t\tthrow new RuntimeException(\"Invalid UTF-16 (low surrogate with no preceding high surrogate)\");\n\t\t\t\t\t}\n\t\t\t\t\telse if (Character.isHighSurrogate(ch)) {\n\t\t\t\t\t\tint lowSurrogate = nextChar();\n\t\t\t\t\t\tif (lowSurrogate > Character.MAX_VALUE) {\n\t\t\t\t\t\t\tthrow new RuntimeException(\"Invalid UTF-16 (high surrogate followed by code point > U+FFFF\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse if (lowSurrogate == IntStream.EOF) {\n\t\t\t\t\t\t\tthrow new RuntimeException(\"Invalid UTF-16 (dangling high surrogate at end of file)\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tchar lowSurrogateChar = (char) lowSurrogate;\n\t\t\t\t\t\t\tif (Character.isLowSurrogate(lowSurrogateChar)) {\n\t\t\t\t\t\t\t\tadd(Character.toCodePoint(ch, lowSurrogateChar));\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\telse {\n\t\t\t\t\t\t\t\tthrow new RuntimeException(\"Invalid UTF-16 (dangling high surrogate\");\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tadd(c);\n\t\t\t\t\t}\n\t\t\t\t}","sourceCodeStart":159,"sourceCodeEnd":195,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java#L159-L195","documentation":"UnbufferedCharStream's fill loop throws this when a high surrogate is followed by a value larger than a char can hold. Since the underlying java.io.Reader only ever returns 0..65535, this branch is defensive and practically indicates the same class of problem as other surrogate errors: the byte-to-char decoding upstream does not match the actual input encoding.","triggerScenarios":"Effectively unreachable via a standard Reader; would require a custom Reader returning values above 0xFFFF directly after a high surrogate. Surfaces during fill()/sync() while buffering characters ahead of the lexer.","commonSituations":"Custom Reader implementations that emit raw code points instead of UTF-16 char units; exotic test doubles passed to UnbufferedCharStream.","solutions":["Use a standard Reader (InputStreamReader with an explicit Charset) as the input source","If you implement a custom Reader, return UTF-16 code units (0..65535), never full code points"],"exampleFix":"// before\nclass CodePointReader extends Reader { /* returns 0x1F600 directly */ }\nnew UnbufferedCharStream(new CodePointReader(...));\n\n// after\n// emit surrogate pairs from the custom Reader:\n// high = (char)(0xD800 + ((cp - 0x10000) >> 10)); low = (char)(0xDC00 + ((cp - 0x10000) & 0x3FF));","handlingStrategy":"validation","validationCode":"if (!(reader instanceof InputStreamReader)\n    || ((InputStreamReader) reader).getEncoding() == null) {\n    // custom Reader: ensure it emits UTF-16 code units only\n    assertReadsAtMostCharValues(reader);\n}","typeGuard":null,"tryCatchPattern":"try { new UnbufferedCharStream(customReader); }\ncatch (RuntimeException e) { /* fix the custom Reader to emit surrogate pairs, not raw code points */ }","preventionTips":["Wrap raw InputStreams in InputStreamReader with a charset","Custom Readers must return 0..65535 per read, pairing surrogates themselves"],"tags":["antlr","char-stream","utf-16","surrogate","custom-reader"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}