stanfordnlp/CoreNLP · error · RuntimeException

No segmenter implemented for

Error message

No segmenter implemented for: ${LanguageInfo.getLanguageFromString(props.getProperty("tokenize.language"))}

What it means

When the pipeline requests segmentation-based tokenization (whitespace=false and a segmenter-relevant setting), TokenizerAnnotator only supports Arabic and Chinese segmenters. If tokenize.language resolves to any other HumanLanguage in that branch, it throws RuntimeException because no segmenter is implemented.

Solutions

1. Use Chinese or Arabic only in this code path; for other languages rely on the default whitespace/PTB tokenizer.
2. For Japanese/Thai consider an external segmenter annotator or set tokenize.whitespace=true and pre-segment the text.
3. Check LanguageInfo.getLanguageFromString on your tokenize.language value to see what it resolves to.
4. Update CoreNLP — newer versions may add more segmenter languages.

Example fix

// before
props.setProperty("tokenize.language", "japanese");
// after
props.setProperty("tokenize.language", "chinese"); // or remove and pre-segment

String lang = props.getProperty("tokenize.language");
Set<String> segmentable = Set.of("arabic", "chinese");
if (!props.bool("tokenize.whitespace") && lang != null && !segmentable.contains(lang.toLowerCase()))
  log.warn("Language " + lang + " has no CoreNLP segmenter; falling back to default tokenization");

boolean hasSegmenter(String l) { return l != null && (l.equalsIgnoreCase("arabic") || l.equalsIgnoreCase("chinese")); }

try { return new TokenizerAnnotator(props); } catch (RuntimeException e) { if (e.getMessage().startsWith("No segmenter")) { props.setProperty("tokenize.whitespace","true"); return new TokenizerAnnotator(props); } throw e; }

When it happens

Trigger: Setting tokenize.whitespace=false with tokenize.language set to a language that routes into the segmenter branch but is neither ARABIC nor CHINESE, e.g. Japanese or Thai, via TokenizerAnnotator(props) constructor.

Common situations: Assuming CoreNLP segments Japanese/Thai out of the box; language auto-detection resolving to an unsupported segmenting language; configs copied from Chinese/Arabic examples with the language changed.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Shouldn't happen:
• Error reading saved links
• RuntimeIOException wrapping IOException
• Error creating data exporter
• Error setting up training

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/09d5b6655873d3d8. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java:220

  }

  public TokenizerAnnotator(boolean verbose, Properties props, String options) {
    if (props == null) {
      props = new Properties();
    }
    // check if segmenting must be done (Chinese or Arabic and not tokenizing on whitespace)
    boolean whitespace = Boolean.parseBoolean(props.getProperty("tokenize.whitespace", "false"));
    if (props.getProperty("tokenize.language") != null &&
        LanguageInfo.isSegmenterLanguage(props.getProperty("tokenize.language")) &&
        !whitespace) {
      cdcAnnotator = null;
      if (LanguageInfo.getLanguageFromString(props.getProperty("tokenize.language")) == LanguageInfo.HumanLanguage.ARABIC) {
        segmenterAnnotator = new ArabicSegmenterAnnotator("segment", props);
      } else if (LanguageInfo.getLanguageFromString(props.getProperty("tokenize.language")) == LanguageInfo.HumanLanguage.CHINESE) {
        segmenterAnnotator = new ChineseSegmenterAnnotator("segment", props);
      } else {
        segmenterAnnotator = null;
        throw new RuntimeException("No segmenter implemented for: "+
                                   LanguageInfo.getLanguageFromString(props.getProperty("tokenize.language")));
      }
    } else if (props.getProperty(STANFORD_CDC_TOKENIZE + ".model", null) != null) {
      cdcAnnotator = new StatTokSentAnnotator(props);
      segmenterAnnotator = null;
    } else {
      segmenterAnnotator = null;
      cdcAnnotator = null;
    }

    // load any custom token post processing
    String postProcessorClass = props.getProperty("tokenize.postProcessor", "");
    List<CoreLabelProcessor> processors = new ArrayList<>();
    try {
      if (!postProcessorClass.equals("")) {
        processors.add(ReflectionLoading.loadByReflection(postProcessorClass));
      }
    } catch (RuntimeException e) {

View on GitHub (pinned to 1b7edd19c4)