stanfordnlp/CoreNLP · error · RuntimeException

Messy token:

Error message

Messy token: 

What it means

LabeledATBDataset's processPreterminal rewrites Arabic preterminal tokens to segmentation-prefixed forms (SEGC/SEGL/SEGR) based on which sides of the word have segment markers. If a token has neither left nor right boundary markers, the pipeline cannot classify it and throws this RuntimeException with the offending raw word.

Solutions

1. Inspect the reported rawWord for missing segment boundary markers
2. Pre-segment the input text with the same Arabic segmenter assumed by the ATB pipeline
3. Check the trees were produced from the same ATB release/preprocessing as the pipeline expects
4. Skip or sanitize tokens without markers before calling processAppliedTree/processPreterminal

Example fix

// before
tree = dataset.processAppliedTree(tree);
// after
for (Tree leaf : tree.getLeaves()) {
  String w = leaf.value();
  if (w == null || !(w.contains("+") || w.endsWith("#") || w.startsWith("#"))) {
    continue; // or log & fix segmentation
  }
}
tree = dataset.processAppliedTree(tree);

for (Tree leaf : tree.getLeaves()) {
  String w = leaf.value();
  if (w == null || w.isEmpty()) throw new IllegalArgumentException("Empty leaf");
  boolean hasSegMark = w.contains("+") || w.contains("#");
  if (!hasSegMark) throw new IllegalArgumentException("Leaf lacks segmentation markers: " + w);
}

When it happens

Trigger: Processing an ATB-parsed Arabic tree whose preterminal leaves lack the expected segment continuation/boundary markers (# or equivalent), e.g. feeding unsegmented or differently-preprocessed text into the ATB dataset transformation.

Common situations: Using trees produced by a different tokenizer/segmenter, ATB releases preprocessed with non-standard scripts, or mixing raw and segmented trees in one corpus load.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Word ( ) mapped to null
• Invalid mapping line:
• ArabicLexer: the invertible option requires a…
• : Token factory is null.
• Error: could not match input

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/d3d55e8d5e50dd7f. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/arabic/pipeline/LabeledATBDataset.java:91

      
      String rawWord = node.firstChild().value().trim();
      
      Matcher left = leftClitic.matcher(rawWord);
      boolean hasLeft = left.find();
      Matcher right = rightClitic.matcher(rawWord);
      boolean hasRight = right.find();
      
      if(rawTag.equals("PUNC") || !(hasRight || hasLeft)) {
        node.firstChild().setValue("XSEG");
      
      } else if(hasRight && hasLeft){
        node.firstChild().setValue("SEGC");
      } else if(hasRight) {
        node.firstChild().setValue("SEGL");
      } else if(hasLeft) {
        node.firstChild().setValue("SEGR");
      } else {
        throw new RuntimeException("Messy token: " + rawWord);
      }
    }
  }
}

View on GitHub (pinned to 1b7edd19c4)