If this method is called, something is seriously wrong (e.g. a JFlex bug producing a faulty\n * scanner etc.).\n *\n *
Usual syntax/scanner level error handling should be done in error fallback rules.\n *\n * @param errorCode the code of the error message to display.\n */\n private static void zzScanError(int errorCode) {\n String message;\n try {\n message = ZZ_ERROR_MSG[errorCode];\n } catch (ArrayIndexOutOfBoundsException e) {\n message = ZZ_ERROR_MSG[ZZ_UNKNOWN_ERROR];\n }\n\n throw new Error(message);\n }\n\n\n /**\n * Pushes the specified amount of characters back into the input stream.\n *\n *
They will be read again by then next call of the scanning method.\n *\n * @param number the number of characters to be read again. This number must not be greater than\n * {@link #yylength()}.\n */\n public void yypushback(int number) {\n if ( number > yylength() )\n zzScanError(ZZ_PUSHBACK_2BIG);\n\n zzMarkedPos -= number;\n }\n","sourceCodeStart":1256,"sourceCodeEnd":1292,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/arabic/process/ArabicLexer.java#L1256-L1292","documentation":"This is JFlex-generated scanner code: when the lexer input cannot be matched by any token rule, zzScanError throws a java.lang.Error with the message for the error code (falling back to the unknown-error message). It indicates the Arabic lexer encountered characters its rules do not cover, such as control bytes or unexpected Latin characters.","triggerScenarios":"Tokenizing input containing bytes/characters outside the lexer's grammar (e.g. raw bytes read with wrong charset, stray control characters, unmatched punctuation) while calling next() on the ArabicLexer.","commonSituations":"Reading files with an incorrect encoding (not UTF-8), dirty text with control characters or emoji, or passing binary/HTML content to the Arabic tokenizer.","solutions":["Verify the input is read as UTF-8 (pass encoding='UTF-8' to the lexer factory)","Pre-clean the text of control/non-printable characters before tokenizing","Locate the offending character near the failure point (binary-search the input)","Strip HTML/markup or normalize text before running the Arabic lexer"],"exampleFix":"// before\nReader r = new InputStreamReader(new FileInputStream(f)); // platform default charset\n// after\nReader r = new InputStreamReader(new FileInputStream(f), StandardCharsets.UTF_8);","handlingStrategy":"try-catch","validationCode":"String clean = text.replaceAll(\"\\\\p{Cntrl}\", \"\");\nbyte[] bytes = clean.getBytes(StandardCharsets.UTF_8);\n// verify round-trip\nif (!new String(bytes, StandardCharsets.UTF_8).equals(clean)) {\n throw new IllegalArgumentException(\"Text contains characters unmatched by Arabic lexer\");\n}","typeGuard":null,"tryCatchPattern":"try {\n String token = lexer.next();\n} catch (Error e) {\n if (\"Error: could not match input\".equals(e.getMessage())) {\n // log position, sanitize input, skip this document\n return Collections.emptyList();\n }\n throw e;\n}","preventionTips":["Always read input as UTF-8","Strip control characters and non-Arabic markup before tokenizing","Binary-search inputs to find unmatched characters when failures occur"],"tags":["arabic-nlp","tokenization","jflex","encoding"],"backgroundTag":"invalid-characters-in-input","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}