If this method is called, something is seriously wrong (e.g. a JFlex bug producing a faulty\n * scanner etc.).\n *\n *
Usual syntax/scanner level error handling should be done in error fallback rules.\n *\n * @param errorCode the code of the error message to display.\n */\n private static void zzScanError(int errorCode) {\n String message;\n try {\n message = ZZ_ERROR_MSG[errorCode];\n } catch (ArrayIndexOutOfBoundsException e) {\n message = ZZ_ERROR_MSG[ZZ_UNKNOWN_ERROR];\n }\n\n throw new Error(message);\n }\n\n\n /**\n * Pushes the specified amount of characters back into the input stream.\n *\n *
They will be read again by then next call of the scanning method.\n *\n * @param number the number of characters to be read again. This number must not be greater than\n * {@link #yylength()}.\n */\n public void yypushback(int number) {\n if ( number > yylength() )\n zzScanError(ZZ_PUSHBACK_2BIG);\n\n zzMarkedPos -= number;\n }\n","sourceCodeStart":12757,"sourceCodeEnd":12793,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/french/process/FrenchLexer.java#L12757-L12793","documentation":"FrenchLexer is a JFlex-generated scanner; when the input contains a character or sequence no lexer rule can match, zzScanError builds a message from ZZ_ERROR_MSG and throws a plain java.lang.Error (LexerError style) rather than an Exception. 'Error: could not match input' means the character buffer position had no matching token rule.","triggerScenarios":"Tokenizing text containing characters outside the lexer's Unicode rule set (rare control characters, unusual symbols) via getNextToken()/getNext().","commonSituations":"Processing raw web text or OCR output with stray bytes; feeding binary or wrongly-decoded data (e.g. Latin-1 bytes read as UTF-8) to the tokenizer.","solutions":["Pre-clean/normalize input text (e.g. with a normalizer or by stripping control characters) before lexing","Wrap getNextToken() calls in try/catch (java.lang.Error) and skip/recover the offending character","Inspect the text at the failure offset for non-standard characters and add a sanitization filter","Ensure correct character encoding when reading input"],"exampleFix":"// before\nwhile (lexer.hasNext()) { tokens.add(lexer.next()); }\n// after\ntry {\n while (lexer.hasNext()) { tokens.add(lexer.next()); }\n} catch (java.lang.Error e) {\n // skip unmatchable input position and continue\n}","handlingStrategy":"try-catch","validationCode":"String clean = input.replaceAll(\"\\\\p{Cntrl}\", \"\").trim();\nif (clean.isEmpty()) return java.util.Collections.emptyList();","typeGuard":null,"tryCatchPattern":"try { return lexer.next(); } catch (java.lang.Error e) { log.warn(\"Unlexable input skipped: \" + e.getMessage()); return recoverOrSkip(); }","preventionTips":["Normalize and sanitize text before tokenization","Verify file encoding is UTF-8 end-to-end","Wrap iteration in error recovery that advances past bad positions"],"tags":["tokenizer","lexing","input-validation"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}