{"record":{"id":"b4ed25c56a2db77d","repo":"stanfordnlp/CoreNLP","slug":"spanishlexer-the-invertible-option-requires-a-cor","errorCode":null,"errorMessage":"SpanishLexer: the invertible option requires a CoreLabelTokenFactory","messagePattern":"SpanishLexer: the invertible option requires a CoreLabelTokenFactory","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/spanish/process/SpanishLexer.java","lineNumber":14194,"sourceCode":"              case \"firstKeep\":\n                untokenizable = UntokenizableOptions.FIRST_KEEP;\n                break;\n              case \"allKeep\":\n                untokenizable = UntokenizableOptions.ALL_KEEP;\n                break;\n              default:\n                throw new IllegalArgumentException(\"SpanishLexer: Invalid option value in constructor: \" + key + \": \" + value);\n            }\n          } else if (\"strictTreebank3\".equals(key)) {\n            strictTreebank3 = val;\n          } else {\n            throw new IllegalArgumentException(String.format(\"%s: Invalid options key in constructor: %s%n\", this.getClass().getName(), key));\n          }\n        }\n        // this.seenUntokenizableCharacter = false; // unnecessary, it's default initialized\n        if (invertible) {\n          if ( ! (tf instanceof CoreLabelTokenFactory)) {\n            throw new IllegalArgumentException(\"SpanishLexer: the invertible option requires a CoreLabelTokenFactory\");\n          }\n          prevWord = (CoreLabel) tf.makeToken(\"\", 0, 0);\n          prevWordAfter = new StringBuilder();\n        }\n      }\n\n\n      /** Turn on to find out how things were tokenized. */\n      private static final boolean DEBUG = false;\n\n      /** A logger for this class */\n      private static final Redwood.RedwoodChannels logger = Redwood.channels(SpanishLexer.class);\n\n      private LexedTokenFactory<?> tokenFactory;\n      private CoreLabel prevWord;\n      private StringBuilder prevWordAfter;\n      private boolean seenUntokenizableCharacter;\n      private enum UntokenizableOptions { NONE_DELETE, FIRST_DELETE, ALL_DELETE, NONE_KEEP, FIRST_KEEP, ALL_KEEP }","sourceCodeStart":14176,"sourceCodeEnd":14212,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/spanish/process/SpanishLexer.java#L14176-L14212","documentation":"The invertible option of SpanishLexer requires the token factory to be a CoreLabelTokenFactory, because invertibility (recording original character offsets / before-text) needs CoreLabel tokens. If invertible is requested with any other LexedTokenFactory (e.g. WhitespaceTokenFactory or a WordTokenFactory), the constructor throws this IllegalArgumentException.","triggerScenarios":"new SpanishLexer(\"invertible=true\", false, someNonCoreLabelTokenFactory) - i.e. requesting invertible=true while passing a token factory that is not an instance of CoreLabelTokenFactory.","commonSituations":"Configuring CoreNLP with tokenize.invertible=true but a tokenize.language/token factory combination that does not use CoreLabelTokenFactory; custom code reusing a WordTokenFactory built for plain tokenization.","solutions":["Pass a CoreLabelTokenFactory as the token factory when invertible=true.","Disable the invertible option if CoreLabel output (and offsets) are not needed.","When using CoreNLP properties, ensure tokenize.invertible is false or that the language pipeline uses CoreLabelTokenFactory."],"exampleFix":"// before\nnew SpanishLexer(\"invertible=true\", false, new WhitespaceTokenFactory());\n// after\nnew SpanishLexer(\"invertible=true\", false, new CoreLabelTokenFactory());","handlingStrategy":"validation","validationCode":"if (invertible && !(tokenFactory instanceof CoreLabelTokenFactory)) {\n  throw new IllegalArgumentException(\"invertible=true requires CoreLabelTokenFactory\");\n}","typeGuard":"boolean canBeInvertible = (tf instanceof CoreLabelTokenFactory);","tryCatchPattern":"try {\n  lexer = new SpanishLexer(\"invertible=true\", false, tf);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"invertible\")) {\n    lexer = new SpanishLexer(\"invertible=false\", false, tf); // fallback without offsets\n  } else throw e;\n}","preventionTips":["Default to CoreLabelTokenFactory whenever you need character offsets or invertibility.","Keep the invertible flag and the token factory type in the same configuration object so they cannot drift apart.","Document that invertible implies CoreLabel output in your project's tokenizer wrapper."],"tags":["tokenizer","configuration","token-factory"],"backgroundTag":"conflicting-config-options","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}