{"record":{"id":"1fdb74accd3a9db3","repo":"stanfordnlp/CoreNLP","slug":"warning-word-with-space-in-lexicon","errorCode":null,"errorMessage":"WARNING: word with space in lexicon","messagePattern":"WARNING: word with space in lexicon","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/wordseg/MaxMatchSegmenter.java","lineNumber":115,"sourceCode":"    ChineseStringUtils.CTPPostProcessor postProcessor = new ChineseStringUtils.CTPPostProcessor();\n    String postSentString = postProcessor.postProcessingAnswer(postProcessedSent.toString(), false);\n    printlnErr(\"Sighan2005 output: \"+postSentString);\n    String[] postSentArray = postSentString.split(\"\\\\s+\");\n    ArrayList<Word> postSent = new ArrayList<>();\n    for(String w : postSentArray) {\n      postSent.add(new Word(w));\n    }\n    return new ArrayList<>(postSent);\n  }\n\n  /**\n   * Add a word to the lexicon, unless it contains some non-Chinese character.\n   */\n  private void addStringToLexicon(String str) {\n    if(str.equals(\"\")) {\n      logger.warn(\"WARNING: blank line in lexicon\");\n    } else if(str.contains(\" \")) {\n      logger.warn(\"WARNING: word with space in lexicon\");\n    } else {\n      if(excludeChar(str)) {\n        printlnErr(\"skipping word: \"+str);\n        return;\n      }\n      // printlnErr(\"adding word: \"+str);\n      words.add(str);\n    }\n  }\n\n  /**\n   * Read lexicon from a one-column text file.\n   */\n  private void addLexicon(String filename) {\n    try {\n      BufferedReader lexiconReader = new BufferedReader(new InputStreamReader(new FileInputStream(filename), \"UTF-8\"));\n      String lexiconLine;\n      while ((lexiconLine = lexiconReader.readLine()) != null) {","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/wordseg/MaxMatchSegmenter.java#L97-L133","documentation":"In MaxMatchSegmenter.addStringToLexicon, a candidate word containing a space character is rejected with \"WARNING: word with space in lexicon\". Chinese words should not contain spaces; such lines indicate malformed lexicon entries and are skipped.","triggerScenarios":"train(...) or addLexicon(...) reading a lexicon file whose lines contain spaces — e.g., lines with \"word  frequency\" columns, tabs rendered as spaces, or English/Chinese mixed content.","commonSituations":"Lexicon files exported from databases or TSV-like sources that weren't reduced to one word per line; files mixing word+count columns; copy-pasted text with internal spaces.","solutions":["Strip anything after the first whitespace or split columns and keep only the word token before adding lines","Pre-clean the file: sed 's/ .*//' lexicon.txt > lexicon.clean.txt","Check the logged word to see whether it's genuinely malformed or a valid phrase your segmenter build should accept","If multi-character phrases with spaces are legitimate for your use, subclass and relax the check"],"exampleFix":"// before\nsegmenter.addLexicon(\"dict.tsv\"); // lines like \"词语 100\"\n// after\nString line = rawLine.split(\"\\\\s+\")[0]; // keep first column only\nsegmenter.train(new StringReader(cleanedText));","handlingStrategy":"validation","validationCode":"// Java: keep only the word column and reject entries with internal spaces\nString word = rawLine.trim().split(\"\\\\s+\")[0];\nif (word.contains(\" \")) throw new IllegalStateException(\"Bad lexicon entry: \" + rawLine);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Convert TSV/database exports to one-word-per-line before training","Inspect any logged offending word to confirm it's genuinely malformed","Separate frequency/count columns from the word string early in preprocessing","Remember the segmenter silently skips such entries — validate input line format"],"tags":["chinese-segmentation","lexicon","input-validation"],"backgroundTag":"invalid-argument-format","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}