stanfordnlp/CoreNLP · error · IllegalArgumentException

PTBLexer: Invalid option value in constructor:

Error message

PTBLexer: Invalid option value in constructor:

What it means

PTBLexer's constructor option parser throws this when an option key/value pair receives a value outside the allowed set — notably for 'untokenizable', whose value must match one of the UntokenizableOptions cases (e.g. noneDelete, delete, firstKeep, allKeep). Any unrecognized value reaches the default branch and throws.

Solutions

  1. Use an accepted UntokenizableOptions string: noneDelete, firstKeep, allKeep (or the other documented cases)
  2. Inspect the switch in PTBLexer.flex to confirm the exact accepted values for the offending key
  3. Fix the option spelling/casing in the tokenizer options string

Example fix

// before
new PTBTokenizer<>(reader, new CoreLabelTokenFactory(), "untokenizable=keepAll");
// after
new PTBTokenizer<>(reader, new CoreLabelTokenFactory(), "untokenizable=allKeep");
Defensive patterns

Strategy: validation

Validate before calling

List<String> ok = Arrays.asList("noneDelete","firstDelete","firstKeep","allKeep");
if (key.equals("untokenizable") && !ok.contains(value)) throw new IllegalArgumentException("invalid untokenizable: " + value);

Try / catch

try {
  lexer = new PTBLexer(reader, options);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("Invalid option value")) { /* correct the value or fall back to "untokenizable=noneDelete" */ }
  throw e;
}

Prevention

When it happens

Trigger: Passing an options string like 'untokenizable=keepAll' or any 'key=value' where value fails the accepted switch cases for keys validated via a switch (untokenizable, etc.).

Common situations: Typos such as 'untokenizable=allkeep' vs 'allKeep' are tolerated by case but misspellings like 'keepall' are not; copying options from another tokenizer; assuming arbitrary strings are accepted.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/10953d7cee2c659d. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/PTBLexer.flex:241

                untokenizable = UntokenizableOptions.NONE_DELETE;
                break;
              case "firstDelete":
                untokenizable = UntokenizableOptions.FIRST_DELETE;
                break;
              case "allDelete":
                untokenizable = UntokenizableOptions.ALL_DELETE;
                break;
              case "noneKeep":
                untokenizable = UntokenizableOptions.NONE_KEEP;
                break;
              case "firstKeep":
                untokenizable = UntokenizableOptions.FIRST_KEEP;
                break;
              case "allKeep":
                untokenizable = UntokenizableOptions.ALL_KEEP;
                break;
              default:
                throw new IllegalArgumentException("PTBLexer: Invalid option value in constructor: " + key + ": " + value);
            }
          } else if ("strictTreebank3".equals(key)) {
            strictFraction = val;
            strictAcronym = val;
          } else if ("strictFraction".equals(key)) {
            strictFraction = val;
          } else if ("strictAcronym".equals(key)) {
            strictAcronym = val;
          } else {
            throw new IllegalArgumentException("PTBLexer: Invalid options key in constructor: " + key);
          }
        }
        if (invertible) {
          if ( ! (tf instanceof CoreLabelTokenFactory)) {
            throw new IllegalArgumentException("PTBLexer: the invertible option requires a CoreLabelTokenFactory");
          }
          prevWord = (CoreLabel) tf.makeToken("", 0, 0);
          prevWordAfter = new StringBuilder();

View on GitHub (pinned to 1b7edd19c4)