{"record":{"id":"1bf1fbf23e42e7d6","repo":"stanfordnlp/CoreNLP","slug":"spanishlexer-the-invertible-option-requires-a-cor-1bf1fb","errorCode":null,"errorMessage":"SpanishLexer: the invertible option requires a CoreLabelTokenFactory","messagePattern":"SpanishLexer: the invertible option requires a CoreLabelTokenFactory","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/spanish/process/SpanishLexer.flex","lineNumber":186,"sourceCode":"              case \"firstKeep\":\n                untokenizable = UntokenizableOptions.FIRST_KEEP;\n                break;\n              case \"allKeep\":\n                untokenizable = UntokenizableOptions.ALL_KEEP;\n                break;\n              default:\n                throw new IllegalArgumentException(\"SpanishLexer: Invalid option value in constructor: \" + key + \": \" + value);\n            }\n          } else if (\"strictTreebank3\".equals(key)) {\n            strictTreebank3 = val;\n          } else {\n            throw new IllegalArgumentException(String.format(\"%s: Invalid options key in constructor: %s%n\", this.getClass().getName(), key));\n          }\n        }\n        // this.seenUntokenizableCharacter = false; // unnecessary, it's default initialized\n        if (invertible) {\n          if ( ! (tf instanceof CoreLabelTokenFactory)) {\n            throw new IllegalArgumentException(\"SpanishLexer: the invertible option requires a CoreLabelTokenFactory\");\n          }\n          prevWord = (CoreLabel) tf.makeToken(\"\", 0, 0);\n          prevWordAfter = new StringBuilder();\n        }\n      }\n\n\n      /** Turn on to find out how things were tokenized. */\n      private static final boolean DEBUG = false;\n\n      /** A logger for this class */\n      private static final Redwood.RedwoodChannels logger = Redwood.channels(SpanishLexer.class);\n\n      private LexedTokenFactory<?> tokenFactory;\n      private CoreLabel prevWord;\n      private StringBuilder prevWordAfter;\n      private boolean seenUntokenizableCharacter;\n      private enum UntokenizableOptions { NONE_DELETE, FIRST_DELETE, ALL_DELETE, NONE_KEEP, FIRST_KEEP, ALL_KEEP }","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/spanish/process/SpanishLexer.flex#L168-L204","documentation":"SpanishLexer supports the 'invertible' option only with a CoreLabelTokenFactory, since invertibility stores original character offsets in CoreLabel fields. If invertible=true and the provided TokenFactory is not a CoreLabelTokenFactory, the constructor throws this IllegalArgumentException immediately.","triggerScenarios":"new SpanishLexer(reader, nonCoreLabelTokenFactory, options) where options sets invertible=true; e.g. passing WordTokenFactory while requesting invertible tokenization for offset recovery.","commonSituations":"Enabling invertible in CoreNLP pipeline properties while using a default or custom factory; copying lexer setups from code that used a different TokenFactory; custom factories that do not subclass CoreLabelTokenFactory.","solutions":["Pass a CoreLabelTokenFactory whenever invertible=true.","Drop the invertible option if you do not need original-text offsets.","Make any custom factory extend CoreLabelTokenFactory so the instanceof check succeeds.","Check (tf instanceof CoreLabelTokenFactory) before enabling invertible in code that builds options dynamically."],"exampleFix":"// before\nlex = new SpanishLexer(reader, new WordTokenFactory(), invertibleOptions);\n// after\nlex = new SpanishLexer(reader, new CoreLabelTokenFactory(), invertibleOptions);","handlingStrategy":"validation","validationCode":"boolean invertible = Boolean.parseBoolean(options.getOrDefault(\"invertible\", \"false\"));\nif (invertible && !(tf instanceof CoreLabelTokenFactory))\n  throw new IllegalArgumentException(\"invertible requires CoreLabelTokenFactory\");","typeGuard":"boolean isCoreLabelFactory(TokenFactory tf) { return tf instanceof CoreLabelTokenFactory; }","tryCatchPattern":"try {\n  lex = new SpanishLexer(reader, tf, options);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"invertible option requires\")) {\n    lex = new SpanishLexer(reader, new CoreLabelTokenFactory(), options);\n  } else throw e;\n}","preventionTips":["Pair invertible=true only with CoreLabelTokenFactory.","Centralize lexer creation so the factory/invertible pairing is enforced in one place.","Review custom TokenFactory implementations for CoreLabel compatibility."],"tags":["java","tokenization","configuration","incompatible-options"],"backgroundTag":"conflicting-config-options","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}