stanfordnlp/CoreNLP · error · RuntimeException

Option 'removeSegMarker' cannot be used with ArabicSegmenter

Error message

Option 'removeSegMarker' cannot be used with ArabicSegmenter

What it means

ArabicSegmenter.getTokenizerFactory builds its own tokenizer options and cannot accept 'removeSegMarker' among user-supplied ArabicTokenizer options, because the segmenter relies on the segmentation marker token. Supplying it throws this RuntimeException.

Solutions

1. Remove 'removeSegMarker' from your tokenizerOptions list
2. Keep other options (e.g. removeProMarker,removeMorphMarker,removeLengthening) which are allowed
3. Use the default atbFactory() options by leaving tokenizerOptions unset

Example fix

// before
seg.props.setProperty("tokenizerOptions", "removeSegMarker,removeLengthening");
// after
seg.props.setProperty("tokenizerOptions", "removeLengthening");

String opts = props.getProperty("tokenizerOptions", "");
if (opts.contains("removeSegMarker")) {
  throw new IllegalArgumentException("removeSegMarker is not allowed with ArabicSegmenter");
}

When it happens

Trigger: Calling ArabicSegmenter with a tokenizerOptions string (property -tokenizerOptions or serialized config) that contains the substring 'removeSegMarker'.

Common situations: Copying tokenizer options from a parser configuration (where removing the segmentation marker is valid) into the segmenter's properties.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Cannot use custom feature factory with localFeaturesOnly…
• Training is not supported!
• : No 1best segmentation available
• Messy token:
• Invalid mapping line:

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/12ca25055500e734. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/arabic/process/ArabicSegmenter.java:196

  }

  /**
   * Creates an ArabicTokenizer. The default tokenizer
   * is ArabicTokenizer.atbFactory(), which produces the
   * same orthographic normalization as Green and Manning (2010).
   *
   * @return A TokenizerFactory that produces each Arabic token as a CoreLabel
   */
  private TokenizerFactory<CoreLabel> getTokenizerFactory() {
    TokenizerFactory<CoreLabel> tokFactory = null;
    if ( ! isTokenized) {
      if (tokenizerOptions == null) {
        tokFactory = ArabicTokenizer.atbFactory();
        String atbVocOptions = "removeProMarker,removeMorphMarker,removeLengthening";
        tokFactory.setOptions(atbVocOptions);
      } else {
        if (tokenizerOptions.contains("removeSegMarker")) {
          throw new RuntimeException("Option 'removeSegMarker' cannot be used with ArabicSegmenter");
        }
        tokFactory = ArabicTokenizer.factory();
        tokFactory.setOptions(tokenizerOptions);
      }
      log.info("Loaded ArabicTokenizer with options: " + tokenizerOptions);
    }
    return tokFactory;
  }

  @Override
  public void initializeTraining(double numTrees) {
    throw new UnsupportedOperationException("Training is not supported!");
  }

  @Override
  public void train(Collection<Tree> trees) {
    throw new UnsupportedOperationException("Training is not supported!");
  }

View on GitHub (pinned to 1b7edd19c4)