stanfordnlp/CoreNLP · error · RuntimeException

Ate the whole text without matching. Expected is '" + w +…

Error message

Ate the whole text without matching.  Expected is '" + w + "', ate '" + sb.toString() + "'

What it means

In runSegmentation, advancePos() re-scans the character-annotated input to align a segmented word 'w' with the original character list. If it consumes all characters without the accumulated string equaling the expected word, it throws this RuntimeException — meaning the segmenter's output no longer aligns with the source text.

Solutions

  1. Normalize input text (whitespace, newlines, character width) before segmentation so segmenter output matches the source characters
  2. Check that the segmentation model matches the character annotation scheme being used
  3. Capture the sentence text from the exception (expected vs ate) and inspect offending characters
  4. Update CoreNLP — alignment bugs in the segmenter have been patched across versions

Example fix

// before
String text = rawText; // contains \r\n and odd whitespace
// after
String text = rawText.replaceAll("\\s+", " ").trim();
Defensive patterns

Strategy: validation

Validate before calling

String normalized = input.replaceAll("\\r", "").replaceAll("\\s+", " ").trim();
if (normalized.isEmpty()) throw new IllegalArgumentException("Empty text for segmentation");

Try / catch

try {
  segmenter.annotate(annotation);
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Ate the whole text")) {
    // log expected vs ate, normalize input and retry
  } else throw e;
}

Prevention

When it happens

Trigger: The CRF segmenter returns a word whose characters don't match the original sentence characters at the current position (e.g. after newline normalization, whitespace handling, or model output that skips/duplicates characters), so the while loop exhausts sentChars.

Common situations: Text containing characters the model maps differently (full-width vs half-width, weird whitespace, control characters); mismatched preprocessing between tokenization and segmentation; unusual encodings.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/1253a8c5d6e5318b. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/ChineseSegmenterAnnotator.java:295

    annotation.set(SegmenterCoreAnnotations.CharactersAnnotation.class, charTokens);
  }

  /** Move the pos pointer to point into sentChars after passing w.
   *  This is a bit subtle, because there can be multi-char codepoints in sentChars elements.
   *
   *  @return The position of the next thing in sentChars to look at
   */
  private static int advancePos(List<CoreLabel> sentChars, int pos, String w) {
    // splitCharacters only keeps \n, no \r, so just ignore all \r
    if (w.equals("\r")) {
      w = "\n";
    } else {
      w = w.replaceAll("\r", "");
    }
    StringBuilder sb = new StringBuilder();
    while ( ! w.equals(sb.toString())) {
      if (pos >= sentChars.size()) {
        throw new RuntimeException("Ate the whole text without matching.  Expected is '" + w +
                                   "', ate '" + sb.toString() + "'");
      }
      sb.append(sentChars.get(pos).get(CoreAnnotations.ChineseCharAnnotation.class));
      pos++;
    }
    return pos;
  }

  private void runSegmentation(CoreMap annotation) {
    //0 2
    // A BC D E
    // 1 10 1 1
    // 0 12 3 4
    // 0, 0+1 ,

    String text = annotation.get(CoreAnnotations.TextAnnotation.class); // the original text String
    List<CoreLabel> sentChars = annotation.get(SegmenterCoreAnnotations.CharactersAnnotation.class); // the way it was divided by splitCharacters
    if (VERBOSE) {

View on GitHub (pinned to 1b7edd19c4)