stanfordnlp/CoreNLP · error · IllegalArgumentException

PTBLexer: the invertible option requires a…

Error message

PTBLexer: the invertible option requires a CoreLabelTokenFactory

What it means

The invertible option makes PTBLexer record before/after text (prevWord/prevWordAfter) so original character offsets and whitespace can be reconstructed. That bookkeeping requires tokens created by a CoreLabelTokenFactory; the constructor checks the supplied TokenizerFactory token factory and throws this IllegalArgumentException when it is anything else.

Solutions

Pass new CoreLabelTokenFactory() (optionally with invertible-specific options) when using invertible=true.
Remove invertible=true from the options if you do not need original whitespace/offset reconstruction.
If you need a different token type, run a second tokenization pass without invertible instead of combining the two.

Example fix

// before
new PTBTokenizer<>(reader, new WordTokenFactory(), "invertible=true");
// after
new PTBTokenizer<>(reader, new CoreLabelTokenFactory(), "invertible=true");

Defensive patterns

Strategy: type-guard

Validate before calling

if (options.contains("invertible=true") && !(factory instanceof CoreLabelTokenFactory)) {
  throw new IllegalArgumentException("invertible=true requires CoreLabelTokenFactory");
}

Type guard

boolean invertibleCompatible(Object factory) { return factory instanceof CoreLabelTokenFactory; }

Try / catch

try { return new PTBTokenizer<>(reader, factory, opts); } catch (IllegalArgumentException e) { if (e.getMessage().contains("requires a CoreLabelTokenFactory")) { return new PTBTokenizer<>(reader, new CoreLabelTokenFactory(), opts); } throw e; }

Prevention

Pair invertible=true exclusively with CoreLabelTokenFactory in your factory-selection code.
Document that offset/whitespace reconstruction requires CoreLabel tokens.
Centralize tokenizer construction in one helper that enforces this pairing.

When it happens

Trigger: new PTBTokenizer<>(reader, someNonCoreLabelFactory, "invertible=true") — e.g. passing a WordTokenFactory or a custom factory that is not an instance of CoreLabelTokenFactory together with invertible=true.

Common situations: Switching token factory for downstream code that needs String/Word tokens while leaving invertible enabled from earlier config; enabling invertible to preserve offsets without realizing it constrains the factory type.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/2c315c063af077af. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/PTBLexer.java:61181

                untokenizable = UntokenizableOptions.ALL_KEEP;
                break;
              default:
                throw new IllegalArgumentException("PTBLexer: Invalid option value in constructor: " + key + ": " + value);
            }
          } else if ("strictTreebank3".equals(key)) {
            strictFraction = val;
            strictAcronym = val;
          } else if ("strictFraction".equals(key)) {
            strictFraction = val;
          } else if ("strictAcronym".equals(key)) {
            strictAcronym = val;
          } else {
            throw new IllegalArgumentException("PTBLexer: Invalid options key in constructor: " + key);
          }
        }
        if (invertible) {
          if ( ! (tf instanceof CoreLabelTokenFactory)) {
            throw new IllegalArgumentException("PTBLexer: the invertible option requires a CoreLabelTokenFactory");
          }
          prevWord = (CoreLabel) tf.makeToken("", 0, 0);
          prevWordAfter = new StringBuilder();
        }
        if (tokenizePerLine) {
          yybegin(YyTokenizePerLine);
        } else {
          yybegin(YyNotTokenizePerLine);
        }
      }


      /** Turn on to find out how things were tokenized. */
      private static final boolean DEBUG = false;

      /** A logger for this class */
      private static final Redwood.RedwoodChannels logger = Redwood.channels(PTBLexer.class);

View on GitHub (pinned to 1b7edd19c4)