{"record":{"id":"2c315c063af077af","repo":"stanfordnlp/CoreNLP","slug":"ptblexer-the-invertible-option-requires-a-corelab","errorCode":null,"errorMessage":"PTBLexer: the invertible option requires a CoreLabelTokenFactory","messagePattern":"PTBLexer: the invertible option requires a CoreLabelTokenFactory","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/PTBLexer.java","lineNumber":61181,"sourceCode":"                untokenizable = UntokenizableOptions.ALL_KEEP;\n                break;\n              default:\n                throw new IllegalArgumentException(\"PTBLexer: Invalid option value in constructor: \" + key + \": \" + value);\n            }\n          } else if (\"strictTreebank3\".equals(key)) {\n            strictFraction = val;\n            strictAcronym = val;\n          } else if (\"strictFraction\".equals(key)) {\n            strictFraction = val;\n          } else if (\"strictAcronym\".equals(key)) {\n            strictAcronym = val;\n          } else {\n            throw new IllegalArgumentException(\"PTBLexer: Invalid options key in constructor: \" + key);\n          }\n        }\n        if (invertible) {\n          if ( ! (tf instanceof CoreLabelTokenFactory)) {\n            throw new IllegalArgumentException(\"PTBLexer: the invertible option requires a CoreLabelTokenFactory\");\n          }\n          prevWord = (CoreLabel) tf.makeToken(\"\", 0, 0);\n          prevWordAfter = new StringBuilder();\n        }\n        if (tokenizePerLine) {\n          yybegin(YyTokenizePerLine);\n        } else {\n          yybegin(YyNotTokenizePerLine);\n        }\n      }\n\n\n      /** Turn on to find out how things were tokenized. */\n      private static final boolean DEBUG = false;\n\n      /** A logger for this class */\n      private static final Redwood.RedwoodChannels logger = Redwood.channels(PTBLexer.class);\n","sourceCodeStart":61163,"sourceCodeEnd":61199,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/PTBLexer.java#L61163-L61199","documentation":"The invertible option makes PTBLexer record before/after text (prevWord/prevWordAfter) so original character offsets and whitespace can be reconstructed. That bookkeeping requires tokens created by a CoreLabelTokenFactory; the constructor checks the supplied TokenizerFactory token factory and throws this IllegalArgumentException when it is anything else.","triggerScenarios":"new PTBTokenizer<>(reader, someNonCoreLabelFactory, \"invertible=true\") — e.g. passing a WordTokenFactory or a custom factory that is not an instance of CoreLabelTokenFactory together with invertible=true.","commonSituations":"Switching token factory for downstream code that needs String/Word tokens while leaving invertible enabled from earlier config; enabling invertible to preserve offsets without realizing it constrains the factory type.","solutions":["Pass new CoreLabelTokenFactory() (optionally with invertible-specific options) when using invertible=true.","Remove invertible=true from the options if you do not need original whitespace/offset reconstruction.","If you need a different token type, run a second tokenization pass without invertible instead of combining the two."],"exampleFix":"// before\nnew PTBTokenizer<>(reader, new WordTokenFactory(), \"invertible=true\");\n// after\nnew PTBTokenizer<>(reader, new CoreLabelTokenFactory(), \"invertible=true\");","handlingStrategy":"type-guard","validationCode":"if (options.contains(\"invertible=true\") && !(factory instanceof CoreLabelTokenFactory)) {\n  throw new IllegalArgumentException(\"invertible=true requires CoreLabelTokenFactory\");\n}","typeGuard":"boolean invertibleCompatible(Object factory) { return factory instanceof CoreLabelTokenFactory; }","tryCatchPattern":"try { return new PTBTokenizer<>(reader, factory, opts); } catch (IllegalArgumentException e) { if (e.getMessage().contains(\"requires a CoreLabelTokenFactory\")) { return new PTBTokenizer<>(reader, new CoreLabelTokenFactory(), opts); } throw e; }","preventionTips":["Pair invertible=true exclusively with CoreLabelTokenFactory in your factory-selection code.","Document that offset/whitespace reconstruction requires CoreLabel tokens.","Centralize tokenizer construction in one helper that enforces this pairing."],"tags":["tokenizer","configuration","type-mismatch"],"backgroundTag":"incompatible-source-type","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}