apache/cassandra · error · LexingError

text could not be lexed

Error message

text could not be lexed

What it means

LexingError raised by pylexotron's LexingRuleSet.lex when the CQL scanner fails to tokenize part of the input text: the scanner returns a non-empty 'unmatched' span, meaning no token pattern matched there. This indicates characters that are invalid in CQL rather than a grammar (parse) error.

Solutions

1. Inspect the text at the position reported in the LexingError (it records the unmatched span) and remove/fix the invalid characters.
2. Ensure string literals and blob literals are properly terminated (matching quotes, correct 0x hex format).
3. Re-save scripts as UTF-8 without BOM and without smart quotes/non-breaking spaces.
4. If generated programmatically, sanitize the input (strip control characters) before passing it to the parser.

Example fix

// before
stmt = "SELECT * FROM t WHERE name = 'O'brien'"
// after
stmt = "SELECT * FROM t WHERE name = 'O''brien'"

from cqlshlib import pylexotron
tokens, unmatched = None, None
# pre-check: strip non-ASCII control chars and verify quotes are balanced
assert text.count("'") % 2 == 0, 'unbalanced single quotes'
text = ''.join(ch for ch in text if ch.isprintable() or ch in '\n\t')

def is_lexable(lexer, text):
    try:
        lexer.lex(text)
        return True
    except pylexotron.LexingError:
        return False

from cqlshlib.pylexotron import LexingError
try:
    statements = cql_split_statements(text)
except LexingError as e:
    print('Cannot lex input near:', getattr(e, 'unmatched_text', text))

When it happens

Trigger: Calling cql_parse / cql_split_statements / lex_and_parse with text containing unmatched or illegal characters, e.g. stray control characters, unterminated string or blob literals, or stray operators like `;` inside an identifier position or unbalanced quotes so the tail cannot be matched.

Common situations: Pasting statements with smart quotes or non-breaking spaces from editors/word processors; unterminated single-quoted strings or unicode string literals; corrupted statement text from encoding issues (wrong file encoding fed to cqlsh); embedded NUL or control bytes in a query script.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• cannot parse map value from
• Cannot parse map value from
• Cannot parse map value from
• Cannot parse map value from
• Could not decode JSON string as a map

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/92cb065ecd5ae3a5. Report an issue: GitHub.

Appendix: source

Thrown at pylib/cqlshlib/pylexotron.py:493

    def register_completer(self, func, rulename, symname):
        self.ruleset[(rulename, symname)] = func

    def make_lexer(self):
        def make_handler(name):
            if name == 'JUNK':
                return None
            return lambda s, t: (name, t, s.match.span())

        regexes = [(p.pattern(), make_handler(name)) for (name, p) in self.terminals]
        return SaferScanner(regexes, re.IGNORECASE | re.DOTALL | re.UNICODE).scan

    def lex(self, text):
        if self.scanner is None:
            self.scanner = self.make_lexer()
        tokens, unmatched = self.scanner(text)
        if unmatched:
            raise LexingError.from_text(text, unmatched, 'text could not be lexed')
        return tokens

    def parse(self, startsymbol, tokens, init_bindings=None):
        if init_bindings is None:
            init_bindings = {}
        ctxt = ParseContext(self.ruleset, init_bindings, (), tuple(tokens), startsymbol)
        pattern = self.ruleset[startsymbol]
        return pattern.match(ctxt, None)

    def whole_match(self, startsymbol, tokens, srcstr=None):
        bindings = {}
        if srcstr is not None:
            bindings['*SRC*'] = srcstr
        for val in self.parse(startsymbol, tokens, init_bindings=bindings):
            if not val.remainder:
                return val

    def lex_and_parse(self, text, startsymbol='Start'):

View on GitHub (pinned to 88fd0f6a0e)