{"record":{"id":"aa9f1ab63e54c87d","repo":"stanfordnlp/CoreNLP","slug":"ptblexer-the-invertible-option-requires-a-corelab-aa9f1a","errorCode":null,"errorMessage":"PTBLexer: the invertible option requires a CoreLabelTokenFactory","messagePattern":"PTBLexer: the invertible option requires a CoreLabelTokenFactory","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/PTBLexer.flex","lineNumber":256,"sourceCode":"                untokenizable = UntokenizableOptions.ALL_KEEP;\n                break;\n              default:\n                throw new IllegalArgumentException(\"PTBLexer: Invalid option value in constructor: \" + key + \": \" + value);\n            }\n          } else if (\"strictTreebank3\".equals(key)) {\n            strictFraction = val;\n            strictAcronym = val;\n          } else if (\"strictFraction\".equals(key)) {\n            strictFraction = val;\n          } else if (\"strictAcronym\".equals(key)) {\n            strictAcronym = val;\n          } else {\n            throw new IllegalArgumentException(\"PTBLexer: Invalid options key in constructor: \" + key);\n          }\n        }\n        if (invertible) {\n          if ( ! (tf instanceof CoreLabelTokenFactory)) {\n            throw new IllegalArgumentException(\"PTBLexer: the invertible option requires a CoreLabelTokenFactory\");\n          }\n          prevWord = (CoreLabel) tf.makeToken(\"\", 0, 0);\n          prevWordAfter = new StringBuilder();\n        }\n        if (tokenizePerLine) {\n          yybegin(YyTokenizePerLine);\n        } else {\n          yybegin(YyNotTokenizePerLine);\n        }\n      }\n\n\n      /** Turn on to find out how things were tokenized. */\n      private static final boolean DEBUG = false;\n\n      /** A logger for this class */\n      private static final Redwood.RedwoodChannels logger = Redwood.channels(PTBLexer.class);\n","sourceCodeStart":238,"sourceCodeEnd":274,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/PTBLexer.flex#L238-L274","documentation":"The invertible option in PTBLexer requires recording original character offsets before/after each token, which is only implemented for CoreLabelTokenFactory. If another TokenFactory (e.g. WordTokenFactory or a custom one) is supplied together with invertible=true, the constructor throws IllegalArgumentException.","triggerScenarios":"Constructing PTBTokenizer/PTBLexer with invertible option enabled and a TokenFactory that is not a CoreLabelTokenFactory, e.g. new PTBTokenizer<>(reader, new WordTokenFactory(), \"invertible=true\").","commonSituations":"Needing before/after text (original string preservation) while reusing a legacy WordTokenFactory; copy-pasted tokenizer construction from code that does not use invertible.","solutions":["Switch the TokenFactory to CoreLabelTokenFactory (or CoreLabelTokenFactory with a label factory)","Disable the invertible option if original-token surroundings are not required","Use the default PTBTokenizer constructor that creates CoreLabels"],"exampleFix":"// before\nnew PTBTokenizer<>(reader, new WordTokenFactory(), \"invertible=true\");\n// after\nnew PTBTokenizer<>(reader, new CoreLabelTokenFactory(), \"invertible=true\");","handlingStrategy":"validation","validationCode":"boolean invertible = Boolean.parseBoolean(options.getOrDefault(\"invertible\",\"false\"));\nif (invertible && !(factory instanceof CoreLabelTokenFactory)) throw new IllegalArgumentException(\"invertible requires CoreLabelTokenFactory\");","typeGuard":"boolean supportsInvertible(TokenFactory tf) { return tf instanceof CoreLabelTokenFactory; }","tryCatchPattern":"try {\n  tokenizer = new PTBTokenizer<>(reader, factory, options);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"invertible option requires\")) { factory = new CoreLabelTokenFactory(); tokenizer = new PTBTokenizer<>(reader, factory, options); }\n  else throw e;\n}","preventionTips":["Always use CoreLabelTokenFactory when you need invertible/offset-preserving tokenization","Document the pairing of invertible with the factory choice in your tokenizer setup code","Prefer default PTBTokenizer constructors that produce CoreLabels"],"tags":["tokenization","options","incompatible-options"],"backgroundTag":"conflicting-config-options","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}