stanfordnlp/CoreNLP · warning

WARNING: blank line in lexicon

Error message

WARNING: blank line in lexicon

What it means

MaxMatchSegmenter (Chinese word segmentation) builds its lexicon from lines of a training/lexicon file via addStringToLexicon. A blank line produces an empty string, which is logged as "WARNING: blank line in lexicon" and skipped. The word is simply not added; segmentation quality may degrade slightly.

Solutions

1. Remove empty lines from the lexicon file (e.g., grep -v '^$' lexicon.txt > lexicon.clean.txt)
2. Pre-filter lines in code before passing to train/addLexicon
3. Ignore the warning if blank lines are harmless in your pipeline — the entry is safely skipped
4. Ensure any preprocessing (sentence splitting) doesn't emit empty strings into the lexicon builder

Example fix

// before
segmenter.addLexicon("lexicon.txt"); // file has blank lines
// after
// strip blank lines first:
grep -v '^[[:space:]]*$' lexicon.txt > lexicon.clean.txt
segmenter.addLexicon("lexicon.clean.txt");

// Java: strip blank lines before building the lexicon
List<String> words = Files.readAllLines(Paths.get(lexiconPath)).stream()
    .map(String::trim)
    .filter(s -> !s.isEmpty())
    .collect(Collectors.toList());

When it happens

Trigger: Calling train(...) or addLexicon(...) with a file that contains empty lines; splitting input on newlines where consecutive newlines or a trailing newline yield "" entries.

Common situations: Hand-edited lexicon files with stray blank lines; files exported from editors adding trailing newlines; concatenated corpora with double newlines.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• WARNING: word with space in lexicon
• Bad line:
• Bad serialized file:
• ChineseCharacterBasedLexicon has no rule iterator!
• Error: could not match input

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/5d2354cab00ef958. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/wordseg/MaxMatchSegmenter.java:113

    ArrayList<Word> postProcessedSent = postProcessSentence(sent);
    printlnErr("processed output: "+ SentenceUtils.listToString(postProcessedSent));
    ChineseStringUtils.CTPPostProcessor postProcessor = new ChineseStringUtils.CTPPostProcessor();
    String postSentString = postProcessor.postProcessingAnswer(postProcessedSent.toString(), false);
    printlnErr("Sighan2005 output: "+postSentString);
    String[] postSentArray = postSentString.split("\\s+");
    ArrayList<Word> postSent = new ArrayList<>();
    for(String w : postSentArray) {
      postSent.add(new Word(w));
    }
    return new ArrayList<>(postSent);
  }

  /**
   * Add a word to the lexicon, unless it contains some non-Chinese character.
   */
  private void addStringToLexicon(String str) {
    if(str.equals("")) {
      logger.warn("WARNING: blank line in lexicon");
    } else if(str.contains(" ")) {
      logger.warn("WARNING: word with space in lexicon");
    } else {
      if(excludeChar(str)) {
        printlnErr("skipping word: "+str);
        return;
      }
      // printlnErr("adding word: "+str);
      words.add(str);
    }
  }

  /**
   * Read lexicon from a one-column text file.
   */
  private void addLexicon(String filename) {
    try {
      BufferedReader lexiconReader = new BufferedReader(new InputStreamReader(new FileInputStream(filename), "UTF-8"));

View on GitHub (pinned to 1b7edd19c4)