stanfordnlp/CoreNLP · error · IllegalArgumentException

PTBLexer: the invertible option requires a…

Error message

PTBLexer: the invertible option requires a CoreLabelTokenFactory

What it means

The invertible option in PTBLexer requires recording original character offsets before/after each token, which is only implemented for CoreLabelTokenFactory. If another TokenFactory (e.g. WordTokenFactory or a custom one) is supplied together with invertible=true, the constructor throws IllegalArgumentException.

Solutions

Switch the TokenFactory to CoreLabelTokenFactory (or CoreLabelTokenFactory with a label factory)
Disable the invertible option if original-token surroundings are not required
Use the default PTBTokenizer constructor that creates CoreLabels

Example fix

// before
new PTBTokenizer<>(reader, new WordTokenFactory(), "invertible=true");
// after
new PTBTokenizer<>(reader, new CoreLabelTokenFactory(), "invertible=true");

Defensive patterns

Strategy: validation

Validate before calling

boolean invertible = Boolean.parseBoolean(options.getOrDefault("invertible","false"));
if (invertible && !(factory instanceof CoreLabelTokenFactory)) throw new IllegalArgumentException("invertible requires CoreLabelTokenFactory");

Type guard

boolean supportsInvertible(TokenFactory tf) { return tf instanceof CoreLabelTokenFactory; }

Try / catch

try {
  tokenizer = new PTBTokenizer<>(reader, factory, options);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("invertible option requires")) { factory = new CoreLabelTokenFactory(); tokenizer = new PTBTokenizer<>(reader, factory, options); }
  else throw e;
}

Prevention

Always use CoreLabelTokenFactory when you need invertible/offset-preserving tokenization
Document the pairing of invertible with the factory choice in your tokenizer setup code
Prefer default PTBTokenizer constructors that produce CoreLabels

When it happens

Trigger: Constructing PTBTokenizer/PTBLexer with invertible option enabled and a TokenFactory that is not a CoreLabelTokenFactory, e.g. new PTBTokenizer<>(reader, new WordTokenFactory(), "invertible=true").

Common situations: Needing before/after text (original string preservation) while reusing a legacy WordTokenFactory; copy-pasted tokenizer construction from code that does not use invertible.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/aa9f1ab63e54c87d. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/PTBLexer.flex:256

                untokenizable = UntokenizableOptions.ALL_KEEP;
                break;
              default:
                throw new IllegalArgumentException("PTBLexer: Invalid option value in constructor: " + key + ": " + value);
            }
          } else if ("strictTreebank3".equals(key)) {
            strictFraction = val;
            strictAcronym = val;
          } else if ("strictFraction".equals(key)) {
            strictFraction = val;
          } else if ("strictAcronym".equals(key)) {
            strictAcronym = val;
          } else {
            throw new IllegalArgumentException("PTBLexer: Invalid options key in constructor: " + key);
          }
        }
        if (invertible) {
          if ( ! (tf instanceof CoreLabelTokenFactory)) {
            throw new IllegalArgumentException("PTBLexer: the invertible option requires a CoreLabelTokenFactory");
          }
          prevWord = (CoreLabel) tf.makeToken("", 0, 0);
          prevWordAfter = new StringBuilder();
        }
        if (tokenizePerLine) {
          yybegin(YyTokenizePerLine);
        } else {
          yybegin(YyNotTokenizePerLine);
        }
      }


      /** Turn on to find out how things were tokenized. */
      private static final boolean DEBUG = false;

      /** A logger for this class */
      private static final Redwood.RedwoodChannels logger = Redwood.channels(PTBLexer.class);

View on GitHub (pinned to 1b7edd19c4)