What causes "Character class maximum nesting level exceeded" in oracle/graal?

Flatten the pattern: merge nested classes into one set ([abc] instead of [a[b[c]]]) or hoist common parts.

oracle/graal · error · UnsupportedRegexException

Character class maximum nesting level exceeded

Error message

Character class maximum nesting level exceeded

What it means

JavaRegexLexer.parseCharClassInternal counts nesting of '[' inside character classes and rejects patterns whose nesting exceeds the TRegexOptions.TRegexParserTreeMaxNestingLevel option, throwing UnsupportedRegexException to bound parser recursion (and avoid stack overflow) on deeply nested classes.

Source

Thrown at regex/src/com.oracle.truffle.regex/src/com/oracle/truffle/regex/flavor/java/JavaRegexLexer.java:613

                }
            }
            if (p == null) {
                throw syntaxError(JavaErrorMessages.unknownUnicodeCharacterProperty(name), ErrorCode.InvalidCharacterClass);
            }
        }
        if (invert) {
            // TODO reference implementation has something with hasSupplementary, do we care about
            // this?;
            p = p.createInverse(Encoding.UTF_16);
        }
        return ClassSetContents.createCharacterClass(p);
    }

    private CodePointSet parseCharClassInternal(boolean consume) throws RegexSyntaxException {
        charClassNesting++;
        try {
            if (charClassNesting > TRegexOptions.TRegexParserTreeMaxNestingLevel) {
                throw new UnsupportedRegexException("Character class maximum nesting level exceeded");
            }
            return parseCharClassInternalBody(consume);
        } finally {
            charClassNesting--;
        }
    }

    private CodePointSet parseCharClassInternalBody(boolean consume) throws RegexSyntaxException {
        boolean invert = false;
        // negation can only occur after a bracket, we cannot have negation after '&&' for example
        if (curChar() == '^' && pattern.charAt(position - 1) == '[') {
            advance();
            invert = true;
        }

        CodePointSet curr = null;
        CodePointSet prev = null;

View on GitHub (pinned to a66e9ccd1d)

Solutions

Flatten the pattern: merge nested classes into one set ([abc] instead of [a[b[c]]]) or hoist common parts.
Raise the engine option regex.TRegexParserTreeMaxNestingLevel if deeper nesting is legitimate for your workload.
Cap pattern complexity at the generator so emitted nesting stays under the configured limit.

Example fix

// before
String p = "[[[[[[[a-z]]]]]]]"; // nesting > limit -> throws

// after
String p = "[a-z]"; // flattened equivalent
// or, when deep nesting is required:
// Context option: regex.TRegexParserTreeMaxNestingLevel=1000

Defensive patterns

Strategy: validation

Validate before calling

static int maxClassNesting(String pattern) {
    int depth = 0, max = 0;
    boolean inClass = false;
    for (int i = 0; i < pattern.length(); i++) {
        char c = pattern.charAt(i);
        if (c == '\\') { i++; continue; }
        if (c == '[') { depth++; max = Math.max(max, depth); }
        else if (c == ']') { depth = Math.max(0, depth - 1); }
    }
    return max;
}
// reject or flatten when maxClassNesting(p) > limit (default TRegexParserTreeMaxNestingLevel)

Try / catch

try {
    return engine.compile(pattern);
} catch (com.oracle.truffle.regex.UnsupportedRegexException e) {
    if (e.getMessage().contains("nesting level")) {
        return engine.compile(flattenCharacterClasses(pattern));
    }
    throw e;
}

Prevention

Flatten nested character classes at pattern-generation time ([a[b[c]]] -> [abc]).
Cap generator output nesting below the TRegexParserTreeMaxNestingLevel option.
Raise regex.TRegexParserTreeMaxNestingLevel deliberately (with a stack-depth budget) rather than as a blind retry.

When it happens

Trigger: Compiling a Java-flavor pattern with nested character classes or class-set operations deeper than the configured limit, e.g. repeated '[a[b[c[...]]]]' or machine-generated patterns with many nested intersections '[[[...]]]'.

Common situations: Programmatic pattern builders (e.g. alternation-to-class optimizations) that nest classes recursively; fuzzed input fed to a regex compiler; generated word-matching patterns from large dictionaries.

Related errors

AI-assisted analysis of oracle/graal@a66e9ccd1d (2026-08-14). Data as JSON: /api/errors/17d3b405efde8299. Report an issue: GitHub.