{"record":{"id":"92cb065ecd5ae3a5","repo":"apache/cassandra","slug":"text-could-not-be-lexed","errorCode":null,"errorMessage":"text could not be lexed","messagePattern":"text could not be lexed","errorType":"exception","errorClass":"LexingError","httpStatus":null,"severity":"error","filePath":"pylib/cqlshlib/pylexotron.py","lineNumber":493,"sourceCode":"\n    def register_completer(self, func, rulename, symname):\n        self.ruleset[(rulename, symname)] = func\n\n    def make_lexer(self):\n        def make_handler(name):\n            if name == 'JUNK':\n                return None\n            return lambda s, t: (name, t, s.match.span())\n\n        regexes = [(p.pattern(), make_handler(name)) for (name, p) in self.terminals]\n        return SaferScanner(regexes, re.IGNORECASE | re.DOTALL | re.UNICODE).scan\n\n    def lex(self, text):\n        if self.scanner is None:\n            self.scanner = self.make_lexer()\n        tokens, unmatched = self.scanner(text)\n        if unmatched:\n            raise LexingError.from_text(text, unmatched, 'text could not be lexed')\n        return tokens\n\n    def parse(self, startsymbol, tokens, init_bindings=None):\n        if init_bindings is None:\n            init_bindings = {}\n        ctxt = ParseContext(self.ruleset, init_bindings, (), tuple(tokens), startsymbol)\n        pattern = self.ruleset[startsymbol]\n        return pattern.match(ctxt, None)\n\n    def whole_match(self, startsymbol, tokens, srcstr=None):\n        bindings = {}\n        if srcstr is not None:\n            bindings['*SRC*'] = srcstr\n        for val in self.parse(startsymbol, tokens, init_bindings=bindings):\n            if not val.remainder:\n                return val\n\n    def lex_and_parse(self, text, startsymbol='Start'):","sourceCodeStart":475,"sourceCodeEnd":511,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/pylib/cqlshlib/pylexotron.py#L475-L511","documentation":"LexingError raised by pylexotron's LexingRuleSet.lex when the CQL scanner fails to tokenize part of the input text: the scanner returns a non-empty 'unmatched' span, meaning no token pattern matched there. This indicates characters that are invalid in CQL rather than a grammar (parse) error.","triggerScenarios":"Calling cql_parse / cql_split_statements / lex_and_parse with text containing unmatched or illegal characters, e.g. stray control characters, unterminated string or blob literals, or stray operators like `;` inside an identifier position or unbalanced quotes so the tail cannot be matched.","commonSituations":"Pasting statements with smart quotes or non-breaking spaces from editors/word processors; unterminated single-quoted strings or unicode string literals; corrupted statement text from encoding issues (wrong file encoding fed to cqlsh); embedded NUL or control bytes in a query script.","solutions":["Inspect the text at the position reported in the LexingError (it records the unmatched span) and remove/fix the invalid characters.","Ensure string literals and blob literals are properly terminated (matching quotes, correct 0x hex format).","Re-save scripts as UTF-8 without BOM and without smart quotes/non-breaking spaces.","If generated programmatically, sanitize the input (strip control characters) before passing it to the parser."],"exampleFix":"// before\nstmt = \"SELECT * FROM t WHERE name = 'O'brien'\"\n// after\nstmt = \"SELECT * FROM t WHERE name = 'O''brien'\"","handlingStrategy":"try-catch","validationCode":"from cqlshlib import pylexotron\ntokens, unmatched = None, None\n# pre-check: strip non-ASCII control chars and verify quotes are balanced\nassert text.count(\"'\") % 2 == 0, 'unbalanced single quotes'\ntext = ''.join(ch for ch in text if ch.isprintable() or ch in '\\n\\t')","typeGuard":"def is_lexable(lexer, text):\n    try:\n        lexer.lex(text)\n        return True\n    except pylexotron.LexingError:\n        return False","tryCatchPattern":"from cqlshlib.pylexotron import LexingError\ntry:\n    statements = cql_split_statements(text)\nexcept LexingError as e:\n    print('Cannot lex input near:', getattr(e, 'unmatched_text', text))","preventionTips":["Save CQL scripts as UTF-8 without BOM; avoid smart quotes and non-breaking spaces.","Terminate all string and blob literals before parsing.","Strip control characters from programmatically generated statements."],"tags":["lexer","cql","parsing","syntax-error"],"backgroundTag":"invalid-argument-format","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}