{"record":{"id":"12ca25055500e734","repo":"stanfordnlp/CoreNLP","slug":"option-removesegmarker-cannot-be-used-with-arabi","errorCode":null,"errorMessage":"Option 'removeSegMarker' cannot be used with ArabicSegmenter","messagePattern":"Option 'removeSegMarker' cannot be used with ArabicSegmenter","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/arabic/process/ArabicSegmenter.java","lineNumber":196,"sourceCode":"  }\n\n  /**\n   * Creates an ArabicTokenizer. The default tokenizer\n   * is ArabicTokenizer.atbFactory(), which produces the\n   * same orthographic normalization as Green and Manning (2010).\n   *\n   * @return A TokenizerFactory that produces each Arabic token as a CoreLabel\n   */\n  private TokenizerFactory<CoreLabel> getTokenizerFactory() {\n    TokenizerFactory<CoreLabel> tokFactory = null;\n    if ( ! isTokenized) {\n      if (tokenizerOptions == null) {\n        tokFactory = ArabicTokenizer.atbFactory();\n        String atbVocOptions = \"removeProMarker,removeMorphMarker,removeLengthening\";\n        tokFactory.setOptions(atbVocOptions);\n      } else {\n        if (tokenizerOptions.contains(\"removeSegMarker\")) {\n          throw new RuntimeException(\"Option 'removeSegMarker' cannot be used with ArabicSegmenter\");\n        }\n        tokFactory = ArabicTokenizer.factory();\n        tokFactory.setOptions(tokenizerOptions);\n      }\n      log.info(\"Loaded ArabicTokenizer with options: \" + tokenizerOptions);\n    }\n    return tokFactory;\n  }\n\n  @Override\n  public void initializeTraining(double numTrees) {\n    throw new UnsupportedOperationException(\"Training is not supported!\");\n  }\n\n  @Override\n  public void train(Collection<Tree> trees) {\n    throw new UnsupportedOperationException(\"Training is not supported!\");\n  }","sourceCodeStart":178,"sourceCodeEnd":214,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/arabic/process/ArabicSegmenter.java#L178-L214","documentation":"ArabicSegmenter.getTokenizerFactory builds its own tokenizer options and cannot accept 'removeSegMarker' among user-supplied ArabicTokenizer options, because the segmenter relies on the segmentation marker token. Supplying it throws this RuntimeException.","triggerScenarios":"Calling ArabicSegmenter with a tokenizerOptions string (property -tokenizerOptions or serialized config) that contains the substring 'removeSegMarker'.","commonSituations":"Copying tokenizer options from a parser configuration (where removing the segmentation marker is valid) into the segmenter's properties.","solutions":["Remove 'removeSegMarker' from your tokenizerOptions list","Keep other options (e.g. removeProMarker,removeMorphMarker,removeLengthening) which are allowed","Use the default atbFactory() options by leaving tokenizerOptions unset"],"exampleFix":"// before\nseg.props.setProperty(\"tokenizerOptions\", \"removeSegMarker,removeLengthening\");\n// after\nseg.props.setProperty(\"tokenizerOptions\", \"removeLengthening\");","handlingStrategy":"validation","validationCode":"String opts = props.getProperty(\"tokenizerOptions\", \"\");\nif (opts.contains(\"removeSegMarker\")) {\n  throw new IllegalArgumentException(\"removeSegMarker is not allowed with ArabicSegmenter\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never copy parser tokenizer options into the segmenter config","Filter disallowed options (removeSegMarker) from shared configs","Let ArabicSegmenter use its default tokenizer options"],"tags":["arabic-nlp","segmentation","tokenizer"],"backgroundTag":"conflicting-config-options","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}