What causes "PTBLexer: Invalid options key in constructor: ${key}" in stanfordnlp/CoreNLP?

stanfordnlp/CoreNLP · error · IllegalArgumentException

PTBLexer: Invalid options key in constructor

Error message

PTBLexer: Invalid options key in constructor: ${key}

What it means

The PTBLexer constructor iterates over option key/value pairs and recognizes a fixed set of keys (americanize, normalizeCurrency, quotes, ellipses, dashes, untokenizable, strictTreebank3, strictFraction, strictAcronym, etc.). A key matching none of the branches hits the final else and throws this IllegalArgumentException naming the unknown key.

Solutions

Replace the key with one supported by PTBLexer (e.g. americanize, normalizeCurrency, normalizeOtherBrackets, quotes, ellipses, dashes, escapeForwardSlashAsterisk, untokenizable, splitHyphenated, splitForwardSlash, splitAssimilations, strictTreebank3, strictFraction, strictAcronym, tokenizeNLs, invertible).
Fix spelling of the option key in your tokenize.options string or properties file.
Move keys that belong to other pipeline components out of the tokenizer options.

Example fix

// before
new PTBTokenizer<>(reader, factory, "verbose=true,untokenizable=allKeep");
// after
new PTBTokenizer<>(reader, factory, "untokenizable=allKeep");

Defensive patterns

Strategy: validation

Validate before calling

Set<String> known = Set.of("americanize","normalizeCurrency","normalizeParentheses","normalizeOtherBrackets","quotes","ellipses","dashes","escapeForwardSlashAsterisk","untokenizable","splitHyphenated","splitForwardSlash","splitAssimilations","strictTreebank3","strictFraction","strictAcronym","tokenizeNLs","invertible");
for (String k : optionsMap.keySet()) if (!known.contains(k)) throw new IllegalArgumentException("Unknown tokenizer option: " + k);

Try / catch

try { new PTBTokenizer<>(reader, factory, opts); } catch (IllegalArgumentException e) { if (e.getMessage().contains("Invalid options key")) { throw new ConfigurationException("Check tokenize.options keys", e); } throw e; }

Prevention

Maintain a whitelist of tokenizer keys and validate config against it.
Strip non-tokenizer properties before building tokenize.options.
Construction-time smoke test in CI with the exact options string used in production.

When it happens

Trigger: Passing an options string/map to PTBTokenizer or PTBLexer with a misspelled or unsupported key, e.g. tokenize.options="normaliseCurrency=true,verbose=true".

Common situations: Options copied from other tokenizers (OpenNLP, spaCy-style flags); typos like "untokenizible"; passing tokenizer-unrelated StanfordCoreNLP properties into tokenize.options.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/f4025ae53e5995a8. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/PTBLexer.java:61176

                break;
              case "firstKeep":
                untokenizable = UntokenizableOptions.FIRST_KEEP;
                break;
              case "allKeep":
                untokenizable = UntokenizableOptions.ALL_KEEP;
                break;
              default:
                throw new IllegalArgumentException("PTBLexer: Invalid option value in constructor: " + key + ": " + value);
            }
          } else if ("strictTreebank3".equals(key)) {
            strictFraction = val;
            strictAcronym = val;
          } else if ("strictFraction".equals(key)) {
            strictFraction = val;
          } else if ("strictAcronym".equals(key)) {
            strictAcronym = val;
          } else {
            throw new IllegalArgumentException("PTBLexer: Invalid options key in constructor: " + key);
          }
        }
        if (invertible) {
          if ( ! (tf instanceof CoreLabelTokenFactory)) {
            throw new IllegalArgumentException("PTBLexer: the invertible option requires a CoreLabelTokenFactory");
          }
          prevWord = (CoreLabel) tf.makeToken("", 0, 0);
          prevWordAfter = new StringBuilder();
        }
        if (tokenizePerLine) {
          yybegin(YyTokenizePerLine);
        } else {
          yybegin(YyNotTokenizePerLine);
        }
      }


      /** Turn on to find out how things were tokenized. */

View on GitHub (pinned to 1b7edd19c4)