oracle/graal · error · UnsupportedRegexException

Class set expression maximum nesting level exceeded

Error message

Class set expression maximum nesting level exceeded

What it means

Thrown by RegexLexer.parseClassSetExpression when class-set expression nesting (nested [], [^...], &&, --, ~~ operators or nested class sets in V-mode/OracleDB flavor syntax) exceeds TRegexOptions.TRegexParserTreeMaxNestingLevel (128). The parser counts nesting depth on entry to guard its recursion.

Source

Thrown at regex/src/com.oracle.truffle.regex/src/com/oracle/truffle/regex/tregex/parser/RegexLexer.java:1255

        Difference("--");

        private final String repr;

        ClassSetOperator(String repr) {
            this.repr = repr;
        }

        @Override
        public String toString() {
            return repr;
        }
    }

    protected ClassSetContents parseClassSetExpression() throws RegexSyntaxException {
        classSetNesting++;
        try {
            if (classSetNesting > TRegexOptions.TRegexParserTreeMaxNestingLevel) {
                throw new UnsupportedRegexException("Class set expression maximum nesting level exceeded");
            }
            final boolean invert = consumingLookahead("^");
            ClassSetContentsAccumulator curClassSet = new ClassSetContentsAccumulator();
            ClassSetOperator operator = null;
            boolean firstOperandIsRange = false;
            int startPos = position;
            while (!atEnd()) {
                RegexRootNode.checkThreadInterrupted();
                if (curChar() == ']' && (!featureEnabledCharClassFirstBracketIsLiteral() || position != startPos)) {
                    advance();
                    if (invert && curClassSet.mayContainStrings()) {
                        throw handleComplementOfStringSet();
                    }
                    if (invert) {
                        assert !curClassSet.mayContainStrings() && curClassSet.isCodePointSetOnly();
                        return ClassSetContents.createCharacterClass(complementClassSet(curClassSet.getCodePointSet()));
                    } else {
                        EconomicSet<String> stringsCopy = EconomicSet.create(curClassSet.getStrings().size());

View on GitHub (pinned to a66e9ccd1d)

Solutions

1. Flatten the nested class: [[a[b[c]]]] is equivalent to [abc] in most flavors.
2. Move set operations (&&, --, ~~) out of deeply nested positions into one level.
3. If the class is generated, add a nesting cap in the generator and simplify before compiling.

Example fix

// before
String pattern = "[[[a][b][[c][d]]]]"; // deep nesting

// after
String pattern = "[abcd]"; // flattened

int depth = 0, max = 0; for (char c : pattern.toCharArray()) { if (c == '[') { depth++; max = Math.max(max, depth); } else if (c == ']') depth--; } if (max > 100) throw new IllegalArgumentException("class-set nesting too deep (max 128)");

null

try { compile(pattern); } catch (UnsupportedRegexException e) { if (e.getMessage().contains("nesting")) { /* flatten nested character classes */ } }

When it happens

Trigger: Compiling a character class with more than 128 levels of nested set expressions, e.g. repeated [[[[...]]]] or long chains of intersection/difference operators, in flavors that support class-set syntax (Java V-mode, OracleDB).

Common situations: Generated character classes from data classification; unbalanced input producing accidental deep nesting; users copy-pasting nested classes from PCRE-style documentation into a flavor that recursively nests them.

Related errors

• too many sequences in a single group
• too many terms in a single sequence
• too many capture groups
• DFA transition size explosion
• ASTSuccessor explosion

AI-assisted analysis of oracle/graal@a66e9ccd1d (2026-08-14). Data as JSON: /api/errors/6f9dbfee64a14e23. Report an issue: GitHub.