stanfordnlp/CoreNLP · error · Error

Error: could not match input

Error message

Error: could not match input

What it means

FrenchLexer is a JFlex-generated scanner; when the input contains a character or sequence no lexer rule can match, zzScanError builds a message from ZZ_ERROR_MSG and throws a plain java.lang.Error (LexerError style) rather than an Exception. 'Error: could not match input' means the character buffer position had no matching token rule.

Solutions

1. Pre-clean/normalize input text (e.g. with a normalizer or by stripping control characters) before lexing
2. Wrap getNextToken() calls in try/catch (java.lang.Error) and skip/recover the offending character
3. Inspect the text at the failure offset for non-standard characters and add a sanitization filter
4. Ensure correct character encoding when reading input

Example fix

// before
while (lexer.hasNext()) { tokens.add(lexer.next()); }
// after
try {
  while (lexer.hasNext()) { tokens.add(lexer.next()); }
} catch (java.lang.Error e) {
  // skip unmatchable input position and continue
}

String clean = input.replaceAll("\\p{Cntrl}", "").trim();
if (clean.isEmpty()) return java.util.Collections.emptyList();

try { return lexer.next(); } catch (java.lang.Error e) { log.warn("Unlexable input skipped: " + e.getMessage()); return recoverOrSkip(); }

When it happens

Trigger: Tokenizing text containing characters outside the lexer's Unicode rule set (rare control characters, unusual symbols) via getNextToken()/getNext().

Common situations: Processing raw web text or OCR output with stray bytes; feeding binary or wrongly-decoded data (e.g. Latin-1 bytes read as UTF-8) to the tokenizer.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• adjustFinalToken: Unexpected final char: |
• Bad character encoding
• Bad line:
• Could not load tokenizer factory
• eolString cannot be null

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/5a81632b0adb4c07. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/french/process/FrenchLexer.java:12775

   * match-all fallback rule) this method will only be called with things that
   * "Can't Possibly Happen".
   *
   * <p>If this method is called, something is seriously wrong (e.g. a JFlex bug producing a faulty
   * scanner etc.).
   *
   * <p>Usual syntax/scanner level error handling should be done in error fallback rules.
   *
   * @param errorCode the code of the error message to display.
   */
  private static void zzScanError(int errorCode) {
    String message;
    try {
      message = ZZ_ERROR_MSG[errorCode];
    } catch (ArrayIndexOutOfBoundsException e) {
      message = ZZ_ERROR_MSG[ZZ_UNKNOWN_ERROR];
    }

    throw new Error(message);
  }


  /**
   * Pushes the specified amount of characters back into the input stream.
   *
   * <p>They will be read again by then next call of the scanning method.
   *
   * @param number the number of characters to be read again. This number must not be greater than
   *     {@link #yylength()}.
   */
  public void yypushback(int number)  {
    if ( number > yylength() )
      zzScanError(ZZ_PUSHBACK_2BIG);

    zzMarkedPos -= number;
  }

View on GitHub (pinned to 1b7edd19c4)