{"record":{"id":"1253a8c5d6e5318b","repo":"stanfordnlp/CoreNLP","slug":"ate-the-whole-text-without-matching-expected-is","errorCode":null,"errorMessage":"Ate the whole text without matching.  Expected is '\" + w + \"', ate '\" + sb.toString() + \"'","messagePattern":"Ate the whole text without matching\\.  Expected is '\" \\+ w \\+ \"', ate '\" \\+ sb\\.toString\\(\\) \\+ \"'","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/ChineseSegmenterAnnotator.java","lineNumber":295,"sourceCode":"    annotation.set(SegmenterCoreAnnotations.CharactersAnnotation.class, charTokens);\n  }\n\n  /** Move the pos pointer to point into sentChars after passing w.\n   *  This is a bit subtle, because there can be multi-char codepoints in sentChars elements.\n   *\n   *  @return The position of the next thing in sentChars to look at\n   */\n  private static int advancePos(List<CoreLabel> sentChars, int pos, String w) {\n    // splitCharacters only keeps \\n, no \\r, so just ignore all \\r\n    if (w.equals(\"\\r\")) {\n      w = \"\\n\";\n    } else {\n      w = w.replaceAll(\"\\r\", \"\");\n    }\n    StringBuilder sb = new StringBuilder();\n    while ( ! w.equals(sb.toString())) {\n      if (pos >= sentChars.size()) {\n        throw new RuntimeException(\"Ate the whole text without matching.  Expected is '\" + w +\n                                   \"', ate '\" + sb.toString() + \"'\");\n      }\n      sb.append(sentChars.get(pos).get(CoreAnnotations.ChineseCharAnnotation.class));\n      pos++;\n    }\n    return pos;\n  }\n\n  private void runSegmentation(CoreMap annotation) {\n    //0 2\n    // A BC D E\n    // 1 10 1 1\n    // 0 12 3 4\n    // 0, 0+1 ,\n\n    String text = annotation.get(CoreAnnotations.TextAnnotation.class); // the original text String\n    List<CoreLabel> sentChars = annotation.get(SegmenterCoreAnnotations.CharactersAnnotation.class); // the way it was divided by splitCharacters\n    if (VERBOSE) {","sourceCodeStart":277,"sourceCodeEnd":313,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/ChineseSegmenterAnnotator.java#L277-L313","documentation":"In runSegmentation, advancePos() re-scans the character-annotated input to align a segmented word 'w' with the original character list. If it consumes all characters without the accumulated string equaling the expected word, it throws this RuntimeException — meaning the segmenter's output no longer aligns with the source text.","triggerScenarios":"The CRF segmenter returns a word whose characters don't match the original sentence characters at the current position (e.g. after newline normalization, whitespace handling, or model output that skips/duplicates characters), so the while loop exhausts sentChars.","commonSituations":"Text containing characters the model maps differently (full-width vs half-width, weird whitespace, control characters); mismatched preprocessing between tokenization and segmentation; unusual encodings.","solutions":["Normalize input text (whitespace, newlines, character width) before segmentation so segmenter output matches the source characters","Check that the segmentation model matches the character annotation scheme being used","Capture the sentence text from the exception (expected vs ate) and inspect offending characters","Update CoreNLP — alignment bugs in the segmenter have been patched across versions"],"exampleFix":"// before\nString text = rawText; // contains \\r\\n and odd whitespace\n// after\nString text = rawText.replaceAll(\"\\\\s+\", \" \").trim();","handlingStrategy":"validation","validationCode":"String normalized = input.replaceAll(\"\\\\r\", \"\").replaceAll(\"\\\\s+\", \" \").trim();\nif (normalized.isEmpty()) throw new IllegalArgumentException(\"Empty text for segmentation\");","typeGuard":null,"tryCatchPattern":"try {\n  segmenter.annotate(annotation);\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"Ate the whole text\")) {\n    // log expected vs ate, normalize input and retry\n  } else throw e;\n}","preventionTips":["Normalize whitespace/newlines/character width before segmentation","Use a model consistent with your character annotation scheme","Keep CoreNLP updated; alignment bugs have been fixed in later releases"],"tags":["stanford-corenlp","chinese-segmenter","alignment-failure","nlp"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}