{"record":{"id":"2bc5d91af53eaaac","repo":"stanfordnlp/CoreNLP","slug":"frenchlexer-the-invertible-option-requires-a-core-2bc5d9","errorCode":null,"errorMessage":"FrenchLexer: the invertible option requires a CoreLabelTokenFactory","messagePattern":"FrenchLexer: the invertible option requires a CoreLabelTokenFactory","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/french/process/FrenchLexer.flex","lineNumber":191,"sourceCode":"            case \"firstKeep\":\n              untokenizable = UntokenizableOptions.FIRST_KEEP;\n              break;\n            case \"allKeep\":\n              untokenizable = UntokenizableOptions.ALL_KEEP;\n              break;\n            default:\n              throw new IllegalArgumentException(\"FrenchLexer: Invalid option value in constructor: \" + key + \": \" + value);\n          }\n        } else if (\"strictTreebank3\".equals(key)) {\n          strictTreebank3 = val;\n        } else {\n          System.err.printf(\"%s: Invalid options key in constructor: %s%n\", this.getClass().getName(), key);\n        }\n      }\n      // this.seenUntokenizableCharacter = false; // unnecessary, it's default initialized\n      if (invertible) {\n        if ( ! (tf instanceof CoreLabelTokenFactory)) {\n          throw new IllegalArgumentException(\"FrenchLexer: the invertible option requires a CoreLabelTokenFactory\");\n        }\n        prevWord = (CoreLabel) tf.makeToken(\"\", 0, 0);\n        prevWordAfter = new StringBuilder();\n      }\n    }\n\n\n    /** Turn on to find out how things were tokenized. */\n    private static final boolean DEBUG = false;\n\n    /** A logger for this class */\n    private static final Redwood.RedwoodChannels logger = Redwood.channels(FrenchLexer.class);\n\n\n    private LexedTokenFactory<?> tokenFactory;\n    private CoreLabel prevWord;\n    private StringBuilder prevWordAfter;\n    private boolean seenUntokenizableCharacter;","sourceCodeStart":173,"sourceCodeEnd":209,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/french/process/FrenchLexer.flex#L173-L209","documentation":"FrenchLexer supports the 'invertible' option only when the supplied TokenFactory is a CoreLabelTokenFactory, because invertibility requires keeping original text offsets in CoreLabel fields. If invertible=true and the factory is any other TokenFactory (e.g. a plain WordTokenFactory or HasWord factory), the constructor throws this IllegalArgumentException. Thrown once during lexer construction.","triggerScenarios":"new FrenchLexer(reader, someNonCoreLabelFactory, options) where options contains invertible=true; e.g. passing WordTokenFactory while enabling invertible for downstream inverting of original character offsets.","commonSituations":"Enabling invertible in pipeline properties while the tokenizer is created with a default/non-CoreLabel factory; copying PTBTokenizer setups where invertible was supported with a different factory; custom TokenFactory implementations not extending CoreLabelTokenFactory.","solutions":["Pass a CoreLabelTokenFactory as the TokenFactory when invertible=true.","If you only need plain tokens, remove the invertible option from the options map.","If you need a custom factory, make it extend or wrap CoreLabelTokenFactory so the instanceof check passes.","Guard construction: check (tf instanceof CoreLabelTokenFactory) before setting invertible."],"exampleFix":"// before\nlex = new FrenchLexer(reader, new WordTokenFactory(), optionsWithInvertible);\n// after\nlex = new FrenchLexer(reader, new CoreLabelTokenFactory(), optionsWithInvertible);","handlingStrategy":"validation","validationCode":"boolean invertible = Boolean.parseBoolean(options.getOrDefault(\"invertible\", \"false\"));\nif (invertible && !(tf instanceof CoreLabelTokenFactory))\n  throw new IllegalArgumentException(\"invertible requires CoreLabelTokenFactory\");","typeGuard":"boolean isCoreLabelFactory(TokenFactory tf) { return tf instanceof CoreLabelTokenFactory; }","tryCatchPattern":"try {\n  lex = new FrenchLexer(reader, tf, options);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"invertible option requires\")) {\n    lex = new FrenchLexer(reader, new CoreLabelTokenFactory(), options);\n  } else throw e;\n}","preventionTips":["Always use CoreLabelTokenFactory when you need invertible tokenization.","Centralize lexer construction in one factory method that enforces the invertible/factory pairing.","Document in your config schema that invertible implies a CoreLabel token factory."],"tags":["java","tokenization","configuration","incompatible-options"],"backgroundTag":"conflicting-config-options","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}