stanfordnlp/CoreNLP · error · IllegalArgumentException

PTBLexer: Invalid option value in constructor

Error message

PTBLexer: Invalid option value in constructor: ${key}: ${value}

What it means

PTBLexer's constructor parses option keys; for keys whose values go through the untokenizable switch (noneDelete, firstDelete, firstKeep, noneKeep, allDelete, allKeep), an unrecognized value reaches the switch default and throws this IllegalArgumentException naming both key and value.

Solutions

1. Use one of the exact values: noneDelete, firstDelete, firstKeep, noneKeep, allDelete, allKeep.
2. Check UntokenizableOptions enum in your CoreNLP version for the authoritative list.
3. Fix empty/truncated values in your options string or properties file.

Example fix

// before
new PTBTokenizer<>(reader, new CoreLabelTokenFactory(), "untokenizable=keep");
// after
new PTBTokenizer<>(reader, new CoreLabelTokenFactory(), "untokenizable=allKeep");

Set<String> allowed = Set.of("noneDelete","firstDelete","firstKeep","noneKeep","allDelete","allKeep");
if (key.equals("untokenizable") && !allowed.contains(value)) throw new IllegalArgumentException("untokenizable must be one of " + allowed);

try { new PTBTokenizer<>(reader, factory, options); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith("PTBLexer: Invalid option value")) { throw new ConfigurationException(e); } throw e; }

When it happens

Trigger: Constructing PTBLexer/PTBTokenizer with untokenizable=<unknown>, e.g. untokenizable=keep or untokenizable=ignore; any other key that routes to this switch with an unexpected value.

Common situations: Guessing the accepted untokenizable values instead of reading UntokenizableOptions; truncating a properties line so the value is empty; upgrading CoreNLP and using values from another tokenizer library.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Not a valid dashes style
• Not a valid ellipses style:
• Not a valid ellipses style:
• Not a valid ellipses style
• Not a valid quotes style:

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/4588ad6d6ef9454e. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/PTBLexer.java:61166

                untokenizable = UntokenizableOptions.NONE_DELETE;
                break;
              case "firstDelete":
                untokenizable = UntokenizableOptions.FIRST_DELETE;
                break;
              case "allDelete":
                untokenizable = UntokenizableOptions.ALL_DELETE;
                break;
              case "noneKeep":
                untokenizable = UntokenizableOptions.NONE_KEEP;
                break;
              case "firstKeep":
                untokenizable = UntokenizableOptions.FIRST_KEEP;
                break;
              case "allKeep":
                untokenizable = UntokenizableOptions.ALL_KEEP;
                break;
              default:
                throw new IllegalArgumentException("PTBLexer: Invalid option value in constructor: " + key + ": " + value);
            }
          } else if ("strictTreebank3".equals(key)) {
            strictFraction = val;
            strictAcronym = val;
          } else if ("strictFraction".equals(key)) {
            strictFraction = val;
          } else if ("strictAcronym".equals(key)) {
            strictAcronym = val;
          } else {
            throw new IllegalArgumentException("PTBLexer: Invalid options key in constructor: " + key);
          }
        }
        if (invertible) {
          if ( ! (tf instanceof CoreLabelTokenFactory)) {
            throw new IllegalArgumentException("PTBLexer: the invertible option requires a CoreLabelTokenFactory");
          }
          prevWord = (CoreLabel) tf.makeToken("", 0, 0);
          prevWordAfter = new StringBuilder();

View on GitHub (pinned to 1b7edd19c4)