{"record":{"id":"bab6957f63d3c70f","repo":"stanfordnlp/CoreNLP","slug":"arabiclexer-the-invertible-option-requires-a-core","errorCode":null,"errorMessage":"ArabicLexer: the invertible option requires a CoreLabelTokenFactory","messagePattern":"ArabicLexer: the invertible option requires a CoreLabelTokenFactory","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/arabic/process/ArabicLexer.java","lineNumber":832,"sourceCode":"   invertible = PropertiesUtils.getBool(props, \"invertible\", false);\n   normArDigits = PropertiesUtils.getBool(props, \"normArDigits\", false);\n   normArPunc = PropertiesUtils.getBool(props, \"normArPunc\", false);\n   normAlif = PropertiesUtils.getBool(props, \"normAlif\", false);\n   normYa = PropertiesUtils.getBool(props, \"normYa\", false);\n   removeDiacritics = PropertiesUtils.getBool(props, \"removeDiacritics\", false);\n   removeTatweel = PropertiesUtils.getBool(props, \"removeTatweel\", false);\n   removeQuranChars = PropertiesUtils.getBool(props, \"removeQuranChars\", false);\n   removeProMarker = PropertiesUtils.getBool(props, \"removeProMarker\", false);\n   removeSegMarker = PropertiesUtils.getBool(props, \"removeSegMarker\", false);\n   removeMorphMarker = PropertiesUtils.getBool(props, \"removeMorphMarker\", false);\n   removeLengthening = PropertiesUtils.getBool(props, \"removeLengthening\", false);\n   atbEscaping = PropertiesUtils.getBool(props, \"atbEscaping\", false);\n\n   setupNormalizationMap();\n\n   if (invertible) {\n     if (!(tf instanceof CoreLabelTokenFactory)) {\n       throw new IllegalArgumentException(\"ArabicLexer: the invertible option requires a CoreLabelTokenFactory\");\n     }\n     prevWord = (CoreLabel) tf.makeToken(\"\", 0, 0);\n     prevWordAfter = new StringBuilder();\n   }\n }\n\n private void setupNormalizationMap() {\n   normMap = Generics.newHashMap(200);\n\n   // Junk characters that we always remove\n   normMap.put(\"\\u0600\",\"#\");\n   normMap.put(\"\\u0601\",\"\");\n   normMap.put(\"\\u0602\",\"\");\n   normMap.put(\"\\u0603\",\"\");\n   normMap.put(\"\\u0606\",\"\\u221B\");\n   normMap.put(\"\\u0607\",\"\\u221C\");\n   normMap.put(\"\\u0608\",\"\");\n   normMap.put(\"\\u0609\",\"%\");","sourceCodeStart":814,"sourceCodeEnd":850,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/arabic/process/ArabicLexer.java#L814-L850","documentation":"ArabicLexer's invertible mode keeps the original text and whitespace before each token, which requires storing data in CoreLabel objects. If the supplied TokenFactory is not a CoreLabelTokenFactory, the constructor throws this IllegalArgumentException because the cast to CoreLabel in getNext() would otherwise fail.","triggerScenarios":"Constructing an ArabicLexer (e.g. via ArabicLexerFactory or getLexer) with invertible=true while passing a TokenizerFactory whose token factory is a plain WhitespaceTokenFactory or other non-CoreLabel factory.","commonSituations":"Configuring ArabicTokenizer/ArabicCrashTokenizer with invertible option but a custom or default factory that doesn't produce CoreLabels; copying tokenizer setup code from a non-invertible example.","solutions":["Set the tokenizer factory to a CoreLabelTokenFactory (e.g. tf = new CoreLabelTokenFactory())","Remove the invertible option if you don't need original text/before-text recovery","Check ArabicLexerFactory/props so the 'invertible' option and the factory option agree"],"exampleFix":"// before\nArabicLexer lexer = new ArabicLexer(new BufferedReader(r), options, false, \"UTF-8\", props, new WhitespaceTokenFactory());\n// after\nArabicLexer lexer = new ArabicLexer(new BufferedReader(r), options, false, \"UTF-8\", props, new CoreLabelTokenFactory());","handlingStrategy":"type-guard","validationCode":"Properties p = new Properties();\nboolean invertible = PropertiesUtils.getBool(p, \"invertible\", false);\nif (invertible && !(tf instanceof CoreLabelTokenFactory)) {\n  throw new IllegalArgumentException(\"invertible=true requires a CoreLabelTokenFactory\");\n}","typeGuard":"boolean canUseInvertible(TokenFactory tf) {\n  return tf instanceof CoreLabelTokenFactory;\n}","tryCatchPattern":null,"preventionTips":["Always pair invertible=true with CoreLabelTokenFactory","Default to CoreLabelTokenFactory for Arabic tokenizers","Check factory/invertible agreement in config-loading code"],"tags":["arabic-nlp","tokenization","config"],"backgroundTag":"invalid-constructor-argument","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}