{"record":{"id":"8540d31fcaeccf56","repo":"stanfordnlp/CoreNLP","slug":"frenchlexer-the-invertible-option-requires-a-core","errorCode":null,"errorMessage":"FrenchLexer: the invertible option requires a CoreLabelTokenFactory","messagePattern":"FrenchLexer: the invertible option requires a CoreLabelTokenFactory","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/french/process/FrenchLexer.java","lineNumber":12452,"sourceCode":"            case \"firstKeep\":\n              untokenizable = UntokenizableOptions.FIRST_KEEP;\n              break;\n            case \"allKeep\":\n              untokenizable = UntokenizableOptions.ALL_KEEP;\n              break;\n            default:\n              throw new IllegalArgumentException(\"FrenchLexer: Invalid option value in constructor: \" + key + \": \" + value);\n          }\n        } else if (\"strictTreebank3\".equals(key)) {\n          strictTreebank3 = val;\n        } else {\n          System.err.printf(\"%s: Invalid options key in constructor: %s%n\", this.getClass().getName(), key);\n        }\n      }\n      // this.seenUntokenizableCharacter = false; // unnecessary, it's default initialized\n      if (invertible) {\n        if ( ! (tf instanceof CoreLabelTokenFactory)) {\n          throw new IllegalArgumentException(\"FrenchLexer: the invertible option requires a CoreLabelTokenFactory\");\n        }\n        prevWord = (CoreLabel) tf.makeToken(\"\", 0, 0);\n        prevWordAfter = new StringBuilder();\n      }\n    }\n\n\n    /** Turn on to find out how things were tokenized. */\n    private static final boolean DEBUG = false;\n\n    /** A logger for this class */\n    private static final Redwood.RedwoodChannels logger = Redwood.channels(FrenchLexer.class);\n\n\n    private LexedTokenFactory<?> tokenFactory;\n    private CoreLabel prevWord;\n    private StringBuilder prevWordAfter;\n    private boolean seenUntokenizableCharacter;","sourceCodeStart":12434,"sourceCodeEnd":12470,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/french/process/FrenchLexer.java#L12434-L12470","documentation":"When the 'invertible' option is enabled, FrenchLexer must record before/after text around each token, which requires a CoreLabelTokenFactory. If the supplied TokenFactory is not a CoreLabelTokenFactory, the constructor throws this IllegalArgumentException immediately.","triggerScenarios":"Constructing FrenchLexer with options containing 'invertible=true' while passing a LexedTokenFactory that is not a CoreLabelTokenFactory (e.g. WordTokenFactory).","commonSituations":"Using FrenchLexer inside a pipeline configured with a custom token factory; switching token factories for efficiency while keeping invertible enabled for token-before/after information.","solutions":["Pass a CoreLabelTokenFactory when using invertible=true","Remove the invertible option if token before/after strings are not needed","Ensure the factory instance (not just any TokenFactory) is CoreLabelTokenFactory"],"exampleFix":"// before\nnew FrenchLexer(options, reader, new WordTokenFactory(), true);\n// after\nnew FrenchLexer(options, reader, new CoreLabelTokenFactory(), true);","handlingStrategy":"type-guard","validationCode":"if (options.contains(\"invertible=true\") && !(tf instanceof CoreLabelTokenFactory)) throw new IllegalStateException(\"invertible requires CoreLabelTokenFactory\");","typeGuard":"boolean supportsInvertible(LexedTokenFactory tf) { return tf instanceof CoreLabelTokenFactory; }","tryCatchPattern":"try { lexer = new FrenchLexer(options, reader, tf); } catch (IllegalArgumentException e) { log.error(\"Factory/option conflict: \" + e.getMessage()); lexer = new FrenchLexer(options, reader, new CoreLabelTokenFactory()); }","preventionTips":["Always pair invertible=true with CoreLabelTokenFactory","Audit custom token factories for compatibility with invertible options","Drop invertible when before/after text is not needed"],"tags":["tokenizer","configuration","corelabels"],"backgroundTag":"conflicting-config-options","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}