stanfordnlp/CoreNLP · error · IllegalArgumentException

ArabicLexer: the invertible option requires a…

Error message

ArabicLexer: the invertible option requires a CoreLabelTokenFactory

What it means

ArabicLexer's invertible mode keeps the original text and whitespace before each token, which requires storing data in CoreLabel objects. If the supplied TokenFactory is not a CoreLabelTokenFactory, the constructor throws this IllegalArgumentException because the cast to CoreLabel in getNext() would otherwise fail.

Solutions

1. Set the tokenizer factory to a CoreLabelTokenFactory (e.g. tf = new CoreLabelTokenFactory())
2. Remove the invertible option if you don't need original text/before-text recovery
3. Check ArabicLexerFactory/props so the 'invertible' option and the factory option agree

Example fix

// before
ArabicLexer lexer = new ArabicLexer(new BufferedReader(r), options, false, "UTF-8", props, new WhitespaceTokenFactory());
// after
ArabicLexer lexer = new ArabicLexer(new BufferedReader(r), options, false, "UTF-8", props, new CoreLabelTokenFactory());

Properties p = new Properties();
boolean invertible = PropertiesUtils.getBool(p, "invertible", false);
if (invertible && !(tf instanceof CoreLabelTokenFactory)) {
  throw new IllegalArgumentException("invertible=true requires a CoreLabelTokenFactory");
}

boolean canUseInvertible(TokenFactory tf) {
  return tf instanceof CoreLabelTokenFactory;
}

When it happens

Trigger: Constructing an ArabicLexer (e.g. via ArabicLexerFactory or getLexer) with invertible=true while passing a TokenizerFactory whose token factory is a plain WhitespaceTokenFactory or other non-CoreLabel factory.

Common situations: Configuring ArabicTokenizer/ArabicCrashTokenizer with invertible option but a custom or default factory that doesn't produce CoreLabels; copying tokenizer setup code from a non-invertible example.

Related errors

• : Token factory is null.
• Error: could not match input
• Error loading flags.readerAndWriter
• Error loading flags.plainTextDocumentReaderAndWriter
• Unsupported inference type: " + flags.crfType

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/bab6957f63d3c70f. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/arabic/process/ArabicLexer.java:832

   invertible = PropertiesUtils.getBool(props, "invertible", false);
   normArDigits = PropertiesUtils.getBool(props, "normArDigits", false);
   normArPunc = PropertiesUtils.getBool(props, "normArPunc", false);
   normAlif = PropertiesUtils.getBool(props, "normAlif", false);
   normYa = PropertiesUtils.getBool(props, "normYa", false);
   removeDiacritics = PropertiesUtils.getBool(props, "removeDiacritics", false);
   removeTatweel = PropertiesUtils.getBool(props, "removeTatweel", false);
   removeQuranChars = PropertiesUtils.getBool(props, "removeQuranChars", false);
   removeProMarker = PropertiesUtils.getBool(props, "removeProMarker", false);
   removeSegMarker = PropertiesUtils.getBool(props, "removeSegMarker", false);
   removeMorphMarker = PropertiesUtils.getBool(props, "removeMorphMarker", false);
   removeLengthening = PropertiesUtils.getBool(props, "removeLengthening", false);
   atbEscaping = PropertiesUtils.getBool(props, "atbEscaping", false);

   setupNormalizationMap();

   if (invertible) {
     if (!(tf instanceof CoreLabelTokenFactory)) {
       throw new IllegalArgumentException("ArabicLexer: the invertible option requires a CoreLabelTokenFactory");
     }
     prevWord = (CoreLabel) tf.makeToken("", 0, 0);
     prevWordAfter = new StringBuilder();
   }
 }

 private void setupNormalizationMap() {
   normMap = Generics.newHashMap(200);

   // Junk characters that we always remove
   normMap.put("\u0600","#");
   normMap.put("\u0601","");
   normMap.put("\u0602","");
   normMap.put("\u0603","");
   normMap.put("\u0606","\u221B");
   normMap.put("\u0607","\u221C");
   normMap.put("\u0608","");
   normMap.put("\u0609","%");

View on GitHub (pinned to 1b7edd19c4)