antlr/antlr4 · error · RuntimeException

Invalid UTF-16 (high surrogate followed by code point > U+FF

Error message

Invalid UTF-16 (high surrogate followed by code point > U+FFFF

What it means

UnbufferedCharStream's fill loop throws this when a high surrogate is followed by a value larger than a char can hold. Since the underlying java.io.Reader only ever returns 0..65535, this branch is defensive and practically indicates the same class of problem as other surrogate errors: the byte-to-char decoding upstream does not match the actual input encoding.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java:177

		for (int i=0; i<n; i++) {
			if (this.n > 0 && data[this.n - 1] == IntStream.EOF) {
				return i;
			}

			try {
				int c = nextChar();
				if (c > Character.MAX_VALUE || c == IntStream.EOF) {
					add(c);
				}
				else {
					char ch = (char) c;
					if (Character.isLowSurrogate(ch)) {
						throw new RuntimeException("Invalid UTF-16 (low surrogate with no preceding high surrogate)");
					}
					else if (Character.isHighSurrogate(ch)) {
						int lowSurrogate = nextChar();
						if (lowSurrogate > Character.MAX_VALUE) {
							throw new RuntimeException("Invalid UTF-16 (high surrogate followed by code point > U+FFFF");
						}
						else if (lowSurrogate == IntStream.EOF) {
							throw new RuntimeException("Invalid UTF-16 (dangling high surrogate at end of file)");
						}
						else {
							char lowSurrogateChar = (char) lowSurrogate;
							if (Character.isLowSurrogate(lowSurrogateChar)) {
								add(Character.toCodePoint(ch, lowSurrogateChar));
							}
							else {
								throw new RuntimeException("Invalid UTF-16 (dangling high surrogate");
							}
						}
					}
					else {
						add(c);
					}
				}

View on GitHub (pinned to 7d5770395b)

Solutions

1. Use a standard Reader (InputStreamReader with an explicit Charset) as the input source
2. If you implement a custom Reader, return UTF-16 code units (0..65535), never full code points

Example fix

// before
class CodePointReader extends Reader { /* returns 0x1F600 directly */ }
new UnbufferedCharStream(new CodePointReader(...));

// after
// emit surrogate pairs from the custom Reader:
// high = (char)(0xD800 + ((cp - 0x10000) >> 10)); low = (char)(0xDC00 + ((cp - 0x10000) & 0x3FF));

if (!(reader instanceof InputStreamReader)
    || ((InputStreamReader) reader).getEncoding() == null) {
    // custom Reader: ensure it emits UTF-16 code units only
    assertReadsAtMostCharValues(reader);
}

try { new UnbufferedCharStream(customReader); }
catch (RuntimeException e) { /* fix the custom Reader to emit surrogate pairs, not raw code points */ }

When it happens

Trigger: Effectively unreachable via a standard Reader; would require a custom Reader returning values above 0xFFFF directly after a high surrogate. Surfaces during fill()/sync() while buffering characters ahead of the lexer.

Common situations: Custom Reader implementations that emit raw code points instead of UTF-16 char units; exotic test doubles passed to UnbufferedCharStream.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Invalid UTF-16 (low surrogate with no preceding high surroga
• Invalid UTF-16 (dangling high surrogate at end of file)
• Invalid UTF-16 (dangling high surrogate
• cannot consume EOF
• cannot consume EOF

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/1da5cad8ca650fec. Report an issue: GitHub.