stanfordnlp/CoreNLP · error · Error

Error: could not match input

Error message

Error: could not match input

What it means

This is JFlex-generated scanner code: when the lexer input cannot be matched by any token rule, zzScanError throws a java.lang.Error with the message for the error code (falling back to the unknown-error message). It indicates the Arabic lexer encountered characters its rules do not cover, such as control bytes or unexpected Latin characters.

Solutions

  1. Verify the input is read as UTF-8 (pass encoding='UTF-8' to the lexer factory)
  2. Pre-clean the text of control/non-printable characters before tokenizing
  3. Locate the offending character near the failure point (binary-search the input)
  4. Strip HTML/markup or normalize text before running the Arabic lexer

Example fix

// before
Reader r = new InputStreamReader(new FileInputStream(f)); // platform default charset
// after
Reader r = new InputStreamReader(new FileInputStream(f), StandardCharsets.UTF_8);
Defensive patterns

Strategy: try-catch

Validate before calling

String clean = text.replaceAll("\\p{Cntrl}", "");
byte[] bytes = clean.getBytes(StandardCharsets.UTF_8);
// verify round-trip
if (!new String(bytes, StandardCharsets.UTF_8).equals(clean)) {
  throw new IllegalArgumentException("Text contains characters unmatched by Arabic lexer");
}

Try / catch

try {
  String token = lexer.next();
} catch (Error e) {
  if ("Error: could not match input".equals(e.getMessage())) {
    // log position, sanitize input, skip this document
    return Collections.emptyList();
  }
  throw e;
}

Prevention

When it happens

Trigger: Tokenizing input containing bytes/characters outside the lexer's grammar (e.g. raw bytes read with wrong charset, stray control characters, unmatched punctuation) while calling next() on the ArabicLexer.

Common situations: Reading files with an incorrect encoding (not UTF-8), dirty text with control characters or emoji, or passing binary/HTML content to the Arabic tokenizer.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/b7f07734d0b9a194. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/arabic/process/ArabicLexer.java:1274

   * match-all fallback rule) this method will only be called with things that
   * "Can't Possibly Happen".
   *
   * <p>If this method is called, something is seriously wrong (e.g. a JFlex bug producing a faulty
   * scanner etc.).
   *
   * <p>Usual syntax/scanner level error handling should be done in error fallback rules.
   *
   * @param errorCode the code of the error message to display.
   */
  private static void zzScanError(int errorCode) {
    String message;
    try {
      message = ZZ_ERROR_MSG[errorCode];
    } catch (ArrayIndexOutOfBoundsException e) {
      message = ZZ_ERROR_MSG[ZZ_UNKNOWN_ERROR];
    }

    throw new Error(message);
  }


  /**
   * Pushes the specified amount of characters back into the input stream.
   *
   * <p>They will be read again by then next call of the scanning method.
   *
   * @param number the number of characters to be read again. This number must not be greater than
   *     {@link #yylength()}.
   */
  public void yypushback(int number)  {
    if ( number > yylength() )
      zzScanError(ZZ_PUSHBACK_2BIG);

    zzMarkedPos -= number;
  }

View on GitHub (pinned to 1b7edd19c4)