{"record":{"id":"5d2354cab00ef958","repo":"stanfordnlp/CoreNLP","slug":"warning-blank-line-in-lexicon","errorCode":null,"errorMessage":"WARNING: blank line in lexicon","messagePattern":"WARNING: blank line in lexicon","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/wordseg/MaxMatchSegmenter.java","lineNumber":113,"sourceCode":"    ArrayList<Word> postProcessedSent = postProcessSentence(sent);\n    printlnErr(\"processed output: \"+ SentenceUtils.listToString(postProcessedSent));\n    ChineseStringUtils.CTPPostProcessor postProcessor = new ChineseStringUtils.CTPPostProcessor();\n    String postSentString = postProcessor.postProcessingAnswer(postProcessedSent.toString(), false);\n    printlnErr(\"Sighan2005 output: \"+postSentString);\n    String[] postSentArray = postSentString.split(\"\\\\s+\");\n    ArrayList<Word> postSent = new ArrayList<>();\n    for(String w : postSentArray) {\n      postSent.add(new Word(w));\n    }\n    return new ArrayList<>(postSent);\n  }\n\n  /**\n   * Add a word to the lexicon, unless it contains some non-Chinese character.\n   */\n  private void addStringToLexicon(String str) {\n    if(str.equals(\"\")) {\n      logger.warn(\"WARNING: blank line in lexicon\");\n    } else if(str.contains(\" \")) {\n      logger.warn(\"WARNING: word with space in lexicon\");\n    } else {\n      if(excludeChar(str)) {\n        printlnErr(\"skipping word: \"+str);\n        return;\n      }\n      // printlnErr(\"adding word: \"+str);\n      words.add(str);\n    }\n  }\n\n  /**\n   * Read lexicon from a one-column text file.\n   */\n  private void addLexicon(String filename) {\n    try {\n      BufferedReader lexiconReader = new BufferedReader(new InputStreamReader(new FileInputStream(filename), \"UTF-8\"));","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/wordseg/MaxMatchSegmenter.java#L95-L131","documentation":"MaxMatchSegmenter (Chinese word segmentation) builds its lexicon from lines of a training/lexicon file via addStringToLexicon. A blank line produces an empty string, which is logged as \"WARNING: blank line in lexicon\" and skipped. The word is simply not added; segmentation quality may degrade slightly.","triggerScenarios":"Calling train(...) or addLexicon(...) with a file that contains empty lines; splitting input on newlines where consecutive newlines or a trailing newline yield \"\" entries.","commonSituations":"Hand-edited lexicon files with stray blank lines; files exported from editors adding trailing newlines; concatenated corpora with double newlines.","solutions":["Remove empty lines from the lexicon file (e.g., grep -v '^$' lexicon.txt > lexicon.clean.txt)","Pre-filter lines in code before passing to train/addLexicon","Ignore the warning if blank lines are harmless in your pipeline — the entry is safely skipped","Ensure any preprocessing (sentence splitting) doesn't emit empty strings into the lexicon builder"],"exampleFix":"// before\nsegmenter.addLexicon(\"lexicon.txt\"); // file has blank lines\n// after\n// strip blank lines first:\ngrep -v '^[[:space:]]*$' lexicon.txt > lexicon.clean.txt\nsegmenter.addLexicon(\"lexicon.clean.txt\");","handlingStrategy":"validation","validationCode":"// Java: strip blank lines before building the lexicon\nList<String> words = Files.readAllLines(Paths.get(lexiconPath)).stream()\n    .map(String::trim)\n    .filter(s -> !s.isEmpty())\n    .collect(Collectors.toList());","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Normalize lexicon files: trim lines, drop empties (grep -v '^$')","Check for trailing newlines or CRLF artifacts after editing files on different OSes","Treat the warning as a data-quality signal, not noise","Generate lexicons programmatically to avoid hand-editing blanks"],"tags":["chinese-segmentation","lexicon","input-validation"],"backgroundTag":"empty-required-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}