{"record":{"id":"09d5b6655873d3d8","repo":"stanfordnlp/CoreNLP","slug":"no-segmenter-implemented-for-languageinfo-getla","errorCode":null,"errorMessage":"No segmenter implemented for: ${LanguageInfo.getLanguageFromString(props.getProperty(\"tokenize.language\"))}","messagePattern":"No segmenter implemented for: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java","lineNumber":220,"sourceCode":"  }\n\n  public TokenizerAnnotator(boolean verbose, Properties props, String options) {\n    if (props == null) {\n      props = new Properties();\n    }\n    // check if segmenting must be done (Chinese or Arabic and not tokenizing on whitespace)\n    boolean whitespace = Boolean.parseBoolean(props.getProperty(\"tokenize.whitespace\", \"false\"));\n    if (props.getProperty(\"tokenize.language\") != null &&\n        LanguageInfo.isSegmenterLanguage(props.getProperty(\"tokenize.language\")) &&\n        !whitespace) {\n      cdcAnnotator = null;\n      if (LanguageInfo.getLanguageFromString(props.getProperty(\"tokenize.language\")) == LanguageInfo.HumanLanguage.ARABIC) {\n        segmenterAnnotator = new ArabicSegmenterAnnotator(\"segment\", props);\n      } else if (LanguageInfo.getLanguageFromString(props.getProperty(\"tokenize.language\")) == LanguageInfo.HumanLanguage.CHINESE) {\n        segmenterAnnotator = new ChineseSegmenterAnnotator(\"segment\", props);\n      } else {\n        segmenterAnnotator = null;\n        throw new RuntimeException(\"No segmenter implemented for: \"+\n                                   LanguageInfo.getLanguageFromString(props.getProperty(\"tokenize.language\")));\n      }\n    } else if (props.getProperty(STANFORD_CDC_TOKENIZE + \".model\", null) != null) {\n      cdcAnnotator = new StatTokSentAnnotator(props);\n      segmenterAnnotator = null;\n    } else {\n      segmenterAnnotator = null;\n      cdcAnnotator = null;\n    }\n\n    // load any custom token post processing\n    String postProcessorClass = props.getProperty(\"tokenize.postProcessor\", \"\");\n    List<CoreLabelProcessor> processors = new ArrayList<>();\n    try {\n      if (!postProcessorClass.equals(\"\")) {\n        processors.add(ReflectionLoading.loadByReflection(postProcessorClass));\n      }\n    } catch (RuntimeException e) {","sourceCodeStart":202,"sourceCodeEnd":238,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java#L202-L238","documentation":"When the pipeline requests segmentation-based tokenization (whitespace=false and a segmenter-relevant setting), TokenizerAnnotator only supports Arabic and Chinese segmenters. If tokenize.language resolves to any other HumanLanguage in that branch, it throws RuntimeException because no segmenter is implemented.","triggerScenarios":"Setting tokenize.whitespace=false with tokenize.language set to a language that routes into the segmenter branch but is neither ARABIC nor CHINESE, e.g. Japanese or Thai, via TokenizerAnnotator(props) constructor.","commonSituations":"Assuming CoreNLP segments Japanese/Thai out of the box; language auto-detection resolving to an unsupported segmenting language; configs copied from Chinese/Arabic examples with the language changed.","solutions":["Use Chinese or Arabic only in this code path; for other languages rely on the default whitespace/PTB tokenizer.","For Japanese/Thai consider an external segmenter annotator or set tokenize.whitespace=true and pre-segment the text.","Check LanguageInfo.getLanguageFromString on your tokenize.language value to see what it resolves to.","Update CoreNLP — newer versions may add more segmenter languages."],"exampleFix":"// before\nprops.setProperty(\"tokenize.language\", \"japanese\");\n// after\nprops.setProperty(\"tokenize.language\", \"chinese\"); // or remove and pre-segment","handlingStrategy":"validation","validationCode":"String lang = props.getProperty(\"tokenize.language\");\nSet<String> segmentable = Set.of(\"arabic\", \"chinese\");\nif (!props.bool(\"tokenize.whitespace\") && lang != null && !segmentable.contains(lang.toLowerCase()))\n  log.warn(\"Language \" + lang + \" has no CoreNLP segmenter; falling back to default tokenization\");","typeGuard":"boolean hasSegmenter(String l) { return l != null && (l.equalsIgnoreCase(\"arabic\") || l.equalsIgnoreCase(\"chinese\")); }","tryCatchPattern":"try { return new TokenizerAnnotator(props); } catch (RuntimeException e) { if (e.getMessage().startsWith(\"No segmenter\")) { props.setProperty(\"tokenize.whitespace\",\"true\"); return new TokenizerAnnotator(props); } throw e; }","preventionTips":["Only route Arabic/Chinese to the segmenter path","Pre-segment other languages (e.g. Japanese) before passing text to CoreNLP","Document supported segmenter languages for your users"],"tags":["java","segmenter","unsupported-language","corenlp"],"backgroundTag":"unsupported-operation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}