antlr/antlr4 · critical · RuntimeException

Invalid UTF-16 (low surrogate with no preceding high surroga

Error message

Invalid UTF-16 (low surrogate with no preceding high surrogate)

What it means

While filling its buffer, UnbufferedCharStream decodes the Reader's UTF-16 output. If it reads a low surrogate (U+DC00..U+DFFF) that was not preceded by a high surrogate, the input is not valid UTF-16 and the stream throws RuntimeException. This guards the buffer's code-point invariant: every stored value is a full code point or the EOF marker.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java:172

	 * Add {@code n} characters to the buffer. Returns the number of characters
	 * actually added to the buffer. If the return value is less than {@code n},
	 * then EOF was reached before {@code n} characters could be added.
	 */
	protected int fill(int n) {
		for (int i=0; i<n; i++) {
			if (this.n > 0 && data[this.n - 1] == IntStream.EOF) {
				return i;
			}

			try {
				int c = nextChar();
				if (c > Character.MAX_VALUE || c == IntStream.EOF) {
					add(c);
				}
				else {
					char ch = (char) c;
					if (Character.isLowSurrogate(ch)) {
						throw new RuntimeException("Invalid UTF-16 (low surrogate with no preceding high surrogate)");
					}
					else if (Character.isHighSurrogate(ch)) {
						int lowSurrogate = nextChar();
						if (lowSurrogate > Character.MAX_VALUE) {
							throw new RuntimeException("Invalid UTF-16 (high surrogate followed by code point > U+FFFF");
						}
						else if (lowSurrogate == IntStream.EOF) {
							throw new RuntimeException("Invalid UTF-16 (dangling high surrogate at end of file)");
						}
						else {
							char lowSurrogateChar = (char) lowSurrogate;
							if (Character.isLowSurrogate(lowSurrogateChar)) {
								add(Character.toCodePoint(ch, lowSurrogateChar));
							}
							else {
								throw new RuntimeException("Invalid UTF-16 (dangling high surrogate");
							}
						}

View on GitHub (pinned to 7d5770395b)

Solutions

Open the input with the correct charset: new InputStreamReader(in, StandardCharsets.UTF_8)
Verify the file's real encoding (file -I / hexdump the first bytes, check BOM) and transcode it if needed
Sanitize or reject non-text input before lexing if the source is untrusted

Example fix

// before
CharStream cs = new UnbufferedCharStream(
    new InputStreamReader(new FileInputStream(f))); // platform default charset

// after
CharStream cs = new UnbufferedCharStream(
    new InputStreamReader(new FileInputStream(f), StandardCharsets.UTF_8));

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check the first bytes for a UTF BOM / plausible UTF-8 before constructing the stream
byte[] head = readFirstBytes(in, 4);
Charset cs = detectCharset(head, StandardCharsets.UTF_8); // your heuristic
new UnbufferedCharStream(new InputStreamReader(in, cs));

Try / catch

try {
    CharStream cs = new UnbufferedCharStream(new InputStreamReader(in, StandardCharsets.UTF_8));
    lexer.setInputStream(cs); ...
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().contains("UTF-16"))
        throw new UserInputException("Input is not valid text in the expected encoding", e);
    throw e;
}

Prevention

Always pass an explicit Charset to the Reader
Verify file encoding before parsing untrusted input
Reject binary content at the upload boundary

When it happens

Trigger: Feeding UnbufferedCharStream bytes that are not the encoding the Reader expects: e.g., reading UTF-8 or Latin-1 bytes through a UTF-16 Reader, so a byte pair lands on a lone low surrogate; truncated or binary input opened as UTF-16.

Common situations: Wrong Charset passed to InputStreamReader (platform default vs UTF-8); files with a BOM mismatch (UTF-8 file parsed as UTF-16); binary or mixed-encoding content routed into the lexer.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/fb010152ed95831d. Report an issue: GitHub.