{"record":{"id":"fb010152ed95831d","repo":"antlr/antlr4","slug":"invalid-utf-16-low-surrogate-with-no-preceding-hi","errorCode":null,"errorMessage":"Invalid UTF-16 (low surrogate with no preceding high surrogate)","messagePattern":"Invalid UTF-16 \\(low surrogate with no preceding high surrogate\\)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java","lineNumber":172,"sourceCode":"\t * Add {@code n} characters to the buffer. Returns the number of characters\n\t * actually added to the buffer. If the return value is less than {@code n},\n\t * then EOF was reached before {@code n} characters could be added.\n\t */\n\tprotected int fill(int n) {\n\t\tfor (int i=0; i<n; i++) {\n\t\t\tif (this.n > 0 && data[this.n - 1] == IntStream.EOF) {\n\t\t\t\treturn i;\n\t\t\t}\n\n\t\t\ttry {\n\t\t\t\tint c = nextChar();\n\t\t\t\tif (c > Character.MAX_VALUE || c == IntStream.EOF) {\n\t\t\t\t\tadd(c);\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tchar ch = (char) c;\n\t\t\t\t\tif (Character.isLowSurrogate(ch)) {\n\t\t\t\t\t\tthrow new RuntimeException(\"Invalid UTF-16 (low surrogate with no preceding high surrogate)\");\n\t\t\t\t\t}\n\t\t\t\t\telse if (Character.isHighSurrogate(ch)) {\n\t\t\t\t\t\tint lowSurrogate = nextChar();\n\t\t\t\t\t\tif (lowSurrogate > Character.MAX_VALUE) {\n\t\t\t\t\t\t\tthrow new RuntimeException(\"Invalid UTF-16 (high surrogate followed by code point > U+FFFF\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse if (lowSurrogate == IntStream.EOF) {\n\t\t\t\t\t\t\tthrow new RuntimeException(\"Invalid UTF-16 (dangling high surrogate at end of file)\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tchar lowSurrogateChar = (char) lowSurrogate;\n\t\t\t\t\t\t\tif (Character.isLowSurrogate(lowSurrogateChar)) {\n\t\t\t\t\t\t\t\tadd(Character.toCodePoint(ch, lowSurrogateChar));\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\telse {\n\t\t\t\t\t\t\t\tthrow new RuntimeException(\"Invalid UTF-16 (dangling high surrogate\");\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}","sourceCodeStart":154,"sourceCodeEnd":190,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java#L154-L190","documentation":"While filling its buffer, UnbufferedCharStream decodes the Reader's UTF-16 output. If it reads a low surrogate (U+DC00..U+DFFF) that was not preceded by a high surrogate, the input is not valid UTF-16 and the stream throws RuntimeException. This guards the buffer's code-point invariant: every stored value is a full code point or the EOF marker.","triggerScenarios":"Feeding UnbufferedCharStream bytes that are not the encoding the Reader expects: e.g., reading UTF-8 or Latin-1 bytes through a UTF-16 Reader, so a byte pair lands on a lone low surrogate; truncated or binary input opened as UTF-16.","commonSituations":"Wrong Charset passed to InputStreamReader (platform default vs UTF-8); files with a BOM mismatch (UTF-8 file parsed as UTF-16); binary or mixed-encoding content routed into the lexer.","solutions":["Open the input with the correct charset: new InputStreamReader(in, StandardCharsets.UTF_8)","Verify the file's real encoding (file -I / hexdump the first bytes, check BOM) and transcode it if needed","Sanitize or reject non-text input before lexing if the source is untrusted"],"exampleFix":"// before\nCharStream cs = new UnbufferedCharStream(\n    new InputStreamReader(new FileInputStream(f))); // platform default charset\n\n// after\nCharStream cs = new UnbufferedCharStream(\n    new InputStreamReader(new FileInputStream(f), StandardCharsets.UTF_8));","handlingStrategy":"try-catch","validationCode":"// pre-check the first bytes for a UTF BOM / plausible UTF-8 before constructing the stream\nbyte[] head = readFirstBytes(in, 4);\nCharset cs = detectCharset(head, StandardCharsets.UTF_8); // your heuristic\nnew UnbufferedCharStream(new InputStreamReader(in, cs));","typeGuard":null,"tryCatchPattern":"try {\n    CharStream cs = new UnbufferedCharStream(new InputStreamReader(in, StandardCharsets.UTF_8));\n    lexer.setInputStream(cs); ...\n} catch (RuntimeException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"UTF-16\"))\n        throw new UserInputException(\"Input is not valid text in the expected encoding\", e);\n    throw e;\n}","preventionTips":["Always pass an explicit Charset to the Reader","Verify file encoding before parsing untrusted input","Reject binary content at the upload boundary"],"tags":["antlr","char-stream","utf-16","encoding","surrogate"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}