stanfordnlp/CoreNLP · error · IllegalArgumentException

FrenchLexer: the invertible option requires a…

Error message

FrenchLexer: the invertible option requires a CoreLabelTokenFactory

What it means

FrenchLexer supports the 'invertible' option only when the supplied TokenFactory is a CoreLabelTokenFactory, because invertibility requires keeping original text offsets in CoreLabel fields. If invertible=true and the factory is any other TokenFactory (e.g. a plain WordTokenFactory or HasWord factory), the constructor throws this IllegalArgumentException. Thrown once during lexer construction.

Solutions

Pass a CoreLabelTokenFactory as the TokenFactory when invertible=true.
If you only need plain tokens, remove the invertible option from the options map.
If you need a custom factory, make it extend or wrap CoreLabelTokenFactory so the instanceof check passes.
Guard construction: check (tf instanceof CoreLabelTokenFactory) before setting invertible.

Example fix

// before
lex = new FrenchLexer(reader, new WordTokenFactory(), optionsWithInvertible);
// after
lex = new FrenchLexer(reader, new CoreLabelTokenFactory(), optionsWithInvertible);

Defensive patterns

Strategy: validation

Validate before calling

boolean invertible = Boolean.parseBoolean(options.getOrDefault("invertible", "false"));
if (invertible && !(tf instanceof CoreLabelTokenFactory))
  throw new IllegalArgumentException("invertible requires CoreLabelTokenFactory");

Type guard

boolean isCoreLabelFactory(TokenFactory tf) { return tf instanceof CoreLabelTokenFactory; }

Try / catch

try {
  lex = new FrenchLexer(reader, tf, options);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("invertible option requires")) {
    lex = new FrenchLexer(reader, new CoreLabelTokenFactory(), options);
  } else throw e;
}

Prevention

Always use CoreLabelTokenFactory when you need invertible tokenization.
Centralize lexer construction in one factory method that enforces the invertible/factory pairing.
Document in your config schema that invertible implies a CoreLabel token factory.

When it happens

Trigger: new FrenchLexer(reader, someNonCoreLabelFactory, options) where options contains invertible=true; e.g. passing WordTokenFactory while enabling invertible for downstream inverting of original character offsets.

Common situations: Enabling invertible in pipeline properties while the tokenizer is created with a default/non-CoreLabel factory; copying PTBTokenizer setups where invertible was supported with a different factory; custom TokenFactory implementations not extending CoreLabelTokenFactory.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/2bc5d91af53eaaac. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/french/process/FrenchLexer.flex:191

            case "firstKeep":
              untokenizable = UntokenizableOptions.FIRST_KEEP;
              break;
            case "allKeep":
              untokenizable = UntokenizableOptions.ALL_KEEP;
              break;
            default:
              throw new IllegalArgumentException("FrenchLexer: Invalid option value in constructor: " + key + ": " + value);
          }
        } else if ("strictTreebank3".equals(key)) {
          strictTreebank3 = val;
        } else {
          System.err.printf("%s: Invalid options key in constructor: %s%n", this.getClass().getName(), key);
        }
      }
      // this.seenUntokenizableCharacter = false; // unnecessary, it's default initialized
      if (invertible) {
        if ( ! (tf instanceof CoreLabelTokenFactory)) {
          throw new IllegalArgumentException("FrenchLexer: the invertible option requires a CoreLabelTokenFactory");
        }
        prevWord = (CoreLabel) tf.makeToken("", 0, 0);
        prevWordAfter = new StringBuilder();
      }
    }


    /** Turn on to find out how things were tokenized. */
    private static final boolean DEBUG = false;

    /** A logger for this class */
    private static final Redwood.RedwoodChannels logger = Redwood.channels(FrenchLexer.class);


    private LexedTokenFactory<?> tokenFactory;
    private CoreLabel prevWord;
    private StringBuilder prevWordAfter;
    private boolean seenUntokenizableCharacter;

View on GitHub (pinned to 1b7edd19c4)