{"record":{"id":"f2477fdf27c1273c","repo":"stanfordnlp/CoreNLP","slug":"arabiclexer-the-invertible-option-requires-a-core-f2477f","errorCode":null,"errorMessage":"ArabicLexer: the invertible option requires a CoreLabelTokenFactory","messagePattern":"ArabicLexer: the invertible option requires a CoreLabelTokenFactory","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/arabic/process/ArabicLexer.flex","lineNumber":97,"sourceCode":"   invertible = PropertiesUtils.getBool(props, \"invertible\", false);\n   normArDigits = PropertiesUtils.getBool(props, \"normArDigits\", false);\n   normArPunc = PropertiesUtils.getBool(props, \"normArPunc\", false);\n   normAlif = PropertiesUtils.getBool(props, \"normAlif\", false);\n   normYa = PropertiesUtils.getBool(props, \"normYa\", false);\n   removeDiacritics = PropertiesUtils.getBool(props, \"removeDiacritics\", false);\n   removeTatweel = PropertiesUtils.getBool(props, \"removeTatweel\", false);\n   removeQuranChars = PropertiesUtils.getBool(props, \"removeQuranChars\", false);\n   removeProMarker = PropertiesUtils.getBool(props, \"removeProMarker\", false);\n   removeSegMarker = PropertiesUtils.getBool(props, \"removeSegMarker\", false);\n   removeMorphMarker = PropertiesUtils.getBool(props, \"removeMorphMarker\", false);\n   removeLengthening = PropertiesUtils.getBool(props, \"removeLengthening\", false);\n   atbEscaping = PropertiesUtils.getBool(props, \"atbEscaping\", false);\n\n   setupNormalizationMap();\n\n   if (invertible) {\n     if (!(tf instanceof CoreLabelTokenFactory)) {\n       throw new IllegalArgumentException(\"ArabicLexer: the invertible option requires a CoreLabelTokenFactory\");\n     }\n     prevWord = (CoreLabel) tf.makeToken(\"\", 0, 0);\n     prevWordAfter = new StringBuilder();\n   }\n }\n\n private void setupNormalizationMap() {\n   normMap = Generics.newHashMap(200);\n\n   // Junk characters that we always remove\n   normMap.put(\"\\u0600\",\"#\");\n   normMap.put(\"\\u0601\",\"\");\n   normMap.put(\"\\u0602\",\"\");\n   normMap.put(\"\\u0603\",\"\");\n   normMap.put(\"\\u0606\",\"\\u221B\");\n   normMap.put(\"\\u0607\",\"\\u221C\");\n   normMap.put(\"\\u0608\",\"\");\n   normMap.put(\"\\u0609\",\"%\");","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/arabic/process/ArabicLexer.flex#L79-L115","documentation":"ArabicLexer's constructor validates that when the invertible option is enabled (so original text can be recovered during tokenization), the provided TokenFactory can produce CoreLabel instances. Any other factory cannot store the required OriginalTextAnnotation, so IllegalArgumentException is thrown during lexer setup.","triggerScenarios":"Creating an ArabicLexer with invertible=true and a TokenFactory that is not a CoreLabelTokenFactory (e.g. WhitespaceTokenFactory, PTBTokenizer's default factory, or a custom factory).","commonSituations":"Configuring ArabicTokenizer with invertible=true but a non-CoreLabel token factory; copying tokenizer options between languages; pipeline code that overrides the token factory globally.","solutions":["Pass a CoreLabelTokenFactory when constructing/configuring the ArabicLexer","Set invertible=false if original-text recovery is not needed","Check tokenizer factory configuration in pipeline options (token.factory / tokenizerFactory options)"],"exampleFix":"// before\nArabicTokenizer<CoreMap> tok = ArabicTokenizer.getTokenizer(r, \"invertible=true\", new WhitespaceTokenFactory());\n// after\nArabicTokenizer<CoreMap> tok = ArabicTokenizer.getTokenizer(r, \"invertible=true\", new CoreLabelTokenFactory());","handlingStrategy":"type-guard","validationCode":"if (invertible && !(factory instanceof CoreLabelTokenFactory)) {\n  factory = new CoreLabelTokenFactory();\n}","typeGuard":"boolean supportsInvertible(TokenFactory tf) {\n  return tf instanceof CoreLabelTokenFactory;\n}","tryCatchPattern":"try {\n  lexer = new ArabicLexer(reader, factory, invertible);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"requires a CoreLabelTokenFactory\")) {\n    lexer = new ArabicLexer(reader, new CoreLabelTokenFactory(), invertible);\n  } else throw e;\n}","preventionTips":["Always pair invertible=true with CoreLabelTokenFactory","Centralize tokenizer factory construction in one utility","Review pipeline options that override token factories globally"],"tags":["java","tokenization","arabic","token-factory"],"backgroundTag":"invalid-argument-value","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}