stanfordnlp/CoreNLP · warning

WARNING: word with space in lexicon

Error message

WARNING: word with space in lexicon

What it means

In MaxMatchSegmenter.addStringToLexicon, a candidate word containing a space character is rejected with "WARNING: word with space in lexicon". Chinese words should not contain spaces; such lines indicate malformed lexicon entries and are skipped.

Solutions

1. Strip anything after the first whitespace or split columns and keep only the word token before adding lines
2. Pre-clean the file: sed 's/ .*//' lexicon.txt > lexicon.clean.txt
3. Check the logged word to see whether it's genuinely malformed or a valid phrase your segmenter build should accept
4. If multi-character phrases with spaces are legitimate for your use, subclass and relax the check

Example fix

// before
segmenter.addLexicon("dict.tsv"); // lines like "词语 100"
// after
String line = rawLine.split("\\s+")[0]; // keep first column only
segmenter.train(new StringReader(cleanedText));

// Java: keep only the word column and reject entries with internal spaces
String word = rawLine.trim().split("\\s+")[0];
if (word.contains(" ")) throw new IllegalStateException("Bad lexicon entry: " + rawLine);

When it happens

Trigger: train(...) or addLexicon(...) reading a lexicon file whose lines contain spaces — e.g., lines with "word frequency" columns, tabs rendered as spaces, or English/Chinese mixed content.

Common situations: Lexicon files exported from databases or TSV-like sources that weren't reduced to one word per line; files mixing word+count columns; copy-pasted text with internal spaces.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• WARNING: blank line in lexicon
• Bad line:
• Bad serialized file:
• ChineseCharacterBasedLexicon has no rule iterator!
• Error: could not match input

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/1fdb74accd3a9db3. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/wordseg/MaxMatchSegmenter.java:115

    ChineseStringUtils.CTPPostProcessor postProcessor = new ChineseStringUtils.CTPPostProcessor();
    String postSentString = postProcessor.postProcessingAnswer(postProcessedSent.toString(), false);
    printlnErr("Sighan2005 output: "+postSentString);
    String[] postSentArray = postSentString.split("\\s+");
    ArrayList<Word> postSent = new ArrayList<>();
    for(String w : postSentArray) {
      postSent.add(new Word(w));
    }
    return new ArrayList<>(postSent);
  }

  /**
   * Add a word to the lexicon, unless it contains some non-Chinese character.
   */
  private void addStringToLexicon(String str) {
    if(str.equals("")) {
      logger.warn("WARNING: blank line in lexicon");
    } else if(str.contains(" ")) {
      logger.warn("WARNING: word with space in lexicon");
    } else {
      if(excludeChar(str)) {
        printlnErr("skipping word: "+str);
        return;
      }
      // printlnErr("adding word: "+str);
      words.add(str);
    }
  }

  /**
   * Read lexicon from a one-column text file.
   */
  private void addLexicon(String filename) {
    try {
      BufferedReader lexiconReader = new BufferedReader(new InputStreamReader(new FileInputStream(filename), "UTF-8"));
      String lexiconLine;
      while ((lexiconLine = lexiconReader.readLine()) != null) {

View on GitHub (pinned to 1b7edd19c4)