stanfordnlp/CoreNLP · error · IllegalArgumentException

ArabicLexer: the invertible option requires a…

Error message

ArabicLexer: the invertible option requires a CoreLabelTokenFactory

What it means

ArabicLexer's constructor validates that when the invertible option is enabled (so original text can be recovered during tokenization), the provided TokenFactory can produce CoreLabel instances. Any other factory cannot store the required OriginalTextAnnotation, so IllegalArgumentException is thrown during lexer setup.

Solutions

1. Pass a CoreLabelTokenFactory when constructing/configuring the ArabicLexer
2. Set invertible=false if original-text recovery is not needed
3. Check tokenizer factory configuration in pipeline options (token.factory / tokenizerFactory options)

Example fix

// before
ArabicTokenizer<CoreMap> tok = ArabicTokenizer.getTokenizer(r, "invertible=true", new WhitespaceTokenFactory());
// after
ArabicTokenizer<CoreMap> tok = ArabicTokenizer.getTokenizer(r, "invertible=true", new CoreLabelTokenFactory());

if (invertible && !(factory instanceof CoreLabelTokenFactory)) {
  factory = new CoreLabelTokenFactory();
}

boolean supportsInvertible(TokenFactory tf) {
  return tf instanceof CoreLabelTokenFactory;
}

try {
  lexer = new ArabicLexer(reader, factory, invertible);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("requires a CoreLabelTokenFactory")) {
    lexer = new ArabicLexer(reader, new CoreLabelTokenFactory(), invertible);
  } else throw e;
}

When it happens

Trigger: Creating an ArabicLexer with invertible=true and a TokenFactory that is not a CoreLabelTokenFactory (e.g. WhitespaceTokenFactory, PTBTokenizer's default factory, or a custom factory).

Common situations: Configuring ArabicTokenizer with invertible=true but a non-CoreLabel token factory; copying tokenizer options between languages; pipeline code that overrides the token factory globally.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• : Token factory is null.
• this.getClass().getName() + ": Case is presently…
• Arabic does not support feature type: " + feat.toString()
• : Inconsistent u2b/b2u arrays.
• Word ( ) mapped to null

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/f2477fdf27c1273c. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/arabic/process/ArabicLexer.flex:97

   invertible = PropertiesUtils.getBool(props, "invertible", false);
   normArDigits = PropertiesUtils.getBool(props, "normArDigits", false);
   normArPunc = PropertiesUtils.getBool(props, "normArPunc", false);
   normAlif = PropertiesUtils.getBool(props, "normAlif", false);
   normYa = PropertiesUtils.getBool(props, "normYa", false);
   removeDiacritics = PropertiesUtils.getBool(props, "removeDiacritics", false);
   removeTatweel = PropertiesUtils.getBool(props, "removeTatweel", false);
   removeQuranChars = PropertiesUtils.getBool(props, "removeQuranChars", false);
   removeProMarker = PropertiesUtils.getBool(props, "removeProMarker", false);
   removeSegMarker = PropertiesUtils.getBool(props, "removeSegMarker", false);
   removeMorphMarker = PropertiesUtils.getBool(props, "removeMorphMarker", false);
   removeLengthening = PropertiesUtils.getBool(props, "removeLengthening", false);
   atbEscaping = PropertiesUtils.getBool(props, "atbEscaping", false);

   setupNormalizationMap();

   if (invertible) {
     if (!(tf instanceof CoreLabelTokenFactory)) {
       throw new IllegalArgumentException("ArabicLexer: the invertible option requires a CoreLabelTokenFactory");
     }
     prevWord = (CoreLabel) tf.makeToken("", 0, 0);
     prevWordAfter = new StringBuilder();
   }
 }

 private void setupNormalizationMap() {
   normMap = Generics.newHashMap(200);

   // Junk characters that we always remove
   normMap.put("\u0600","#");
   normMap.put("\u0601","");
   normMap.put("\u0602","");
   normMap.put("\u0603","");
   normMap.put("\u0606","\u221B");
   normMap.put("\u0607","\u221C");
   normMap.put("\u0608","");
   normMap.put("\u0609","%");

View on GitHub (pinned to 1b7edd19c4)