What causes "PTBLexer: Invalid options key in constructor: " in stanfordnlp/CoreNLP?

Check the key against the if/else chain in PTBLexer.flex and correct the spelling

stanfordnlp/CoreNLP · error · IllegalArgumentException

PTBLexer: Invalid options key in constructor:

Error message

PTBLexer: Invalid options key in constructor:

What it means

PTBLexer's options-map constructor validates each key against a known set (normalize, ellipses, dashes, escapeForwardSlashAsterisk, untokenizable, strictTreebank3, strictFraction, strictAcronym, invertible, tokenizePerLine, etc.). An unknown key falls into the final else and throws IllegalArgumentException.

Solutions

Check the key against the if/else chain in PTBLexer.flex and correct the spelling
Remove the unsupported key from the options map
Consult the PTBTokenizer javadoc for the current list of valid option keys

Example fix

// before
options.put("americanise", "true");
// after
options.put("normalizeParentheses", "true");

Defensive patterns

Strategy: validation

Validate before calling

Set<String> known = Set.of("normalize","normalizeParentheses","normalizeOtherBrackets","ellipses","dashes","escapeForwardSlashAsterisk","untokenizable","strictTreebank3","strictFraction","strictAcronym","invertible","tokenizePerLine","tokenizeNLs");
for (String k : options.keySet()) if (!known.contains(k)) throw new IllegalArgumentException("unknown PTBTokenizer option: " + k);

Try / catch

try {
  lexer = new PTBLexer(reader, options);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("Invalid options key")) { /* log the bad key and drop it, or fail config load */ }
  throw e;
}

Prevention

Copy option keys from the PTBTokenizer javadoc, not memory
Check keys against the current CoreNLP version when upgrading
Centralize option-key names in a shared constants class

When it happens

Trigger: Calling PTBLexer(Reader, Map<String,String>) or PTBTokenizer with a Map containing a misspelled or unsupported key, e.g. 'tokenizeNLs' vs 'tokenizePerLine', or a stale key from an older CoreNLP version.

Common situations: Typos in option keys; options copied from other tokenizers (e.g. WhitespaceTokenizer options); keys removed/renamed across CoreNLP releases.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/1168351d56bbff4b. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/PTBLexer.flex:251

                break;
              case "firstKeep":
                untokenizable = UntokenizableOptions.FIRST_KEEP;
                break;
              case "allKeep":
                untokenizable = UntokenizableOptions.ALL_KEEP;
                break;
              default:
                throw new IllegalArgumentException("PTBLexer: Invalid option value in constructor: " + key + ": " + value);
            }
          } else if ("strictTreebank3".equals(key)) {
            strictFraction = val;
            strictAcronym = val;
          } else if ("strictFraction".equals(key)) {
            strictFraction = val;
          } else if ("strictAcronym".equals(key)) {
            strictAcronym = val;
          } else {
            throw new IllegalArgumentException("PTBLexer: Invalid options key in constructor: " + key);
          }
        }
        if (invertible) {
          if ( ! (tf instanceof CoreLabelTokenFactory)) {
            throw new IllegalArgumentException("PTBLexer: the invertible option requires a CoreLabelTokenFactory");
          }
          prevWord = (CoreLabel) tf.makeToken("", 0, 0);
          prevWordAfter = new StringBuilder();
        }
        if (tokenizePerLine) {
          yybegin(YyTokenizePerLine);
        } else {
          yybegin(YyNotTokenizePerLine);
        }
      }


      /** Turn on to find out how things were tokenized. */

View on GitHub (pinned to 1b7edd19c4)