What causes "tokens.size(): %d != pos.size(): %d%n" in stanfordnlp/CoreNLP?

Ensure both lists come from the same tokenization and have equal size before calling fromStringReps

stanfordnlp/CoreNLP · error · RuntimeException

tokens.size(): != pos.size(): %n

Error message

tokens.size(): %d != pos.size(): %d%n

What it means

fromStringReps requires parallel lists: one entry per token in tokens and one POS tag per token in posTags. If the sizes differ, it throws this RuntimeException reporting both sizes, because it would otherwise zip mismatched word/tag pairs into the tree. It is a strict input-consistency check before building the TreeGraphNode word/POS lists.

Solutions

Ensure both lists come from the same tokenization and have equal size before calling fromStringReps
Add an assertion/log listing token-tag pairs to spot where the lists diverge
Align punctuation and multi-word tokens (e.g. split "don't" consistently on both sides)
Use a single Sentence/tokenizer pipeline to produce tokens and posTags so they stay in sync

Example fix

// before
fromStringReps(Arrays.asList("I","saw"), Arrays.asList("PRP","VBD",".") /* extra '.' */, deps);
// after
fromStringReps(Arrays.asList("I","saw","."), Arrays.asList("PRP","VBD","."), deps);

Defensive patterns

Strategy: validation

Validate before calling

if (tokens.size() != posTags.size()) {
    throw new IllegalArgumentException("tokens/posTags size mismatch: " + tokens.size() + " vs " + posTags.size());
}

Try / catch

try {
    GrammaticalStructure gs = GrammaticalStructure.fromStringReps(tokens, posTags, deps);
} catch (RuntimeException e) {
    if (e.getMessage().startsWith("tokens.size():")) {
        // re-tokenize so tokens and tags align, then retry
    } else throw e;
}

Prevention

Derive tokens and POS tags from the same tokenizer/tagger run
Check list sizes before every fromStringReps call
Watch for punctuation and contraction splitting differences between tools
Pair each token with its tag in one data structure instead of parallel lists

When it happens

Trigger: Calling GrammaticalStructure.fromStringReps(tokens, posTags, deps) where tokens.size() != posTags.size() — e.g. POS tags that include or omit punctuation/indices, tokens split differently than tags, or an off-by-one when building the lists.

Common situations: Reading tokens and tags from different preprocessing steps with different tokenizations; including the ROOT pseudo-item in one list but not the other; merging outputs of a tokenizer and tagger that disagree on contractions/hyphens.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/a774d91a774972f7. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/trees/GrammaticalStructure.java:338

     throw new RuntimeException(String.format("Dependencies should be for the format 'type(arg-idx, arg-idx)'. Could not parse '%s'", dep));
  }

  /**
   * Create a grammatical structure from its string representation.
   *
   * Like buildCoNLLXGrammaticalStructure,
   * this method fakes up the parts of the tree structure that are not
   * used by the grammatical relation transformation operations.
   *
   * <i>Note:</i> Added by daniel cer
   *
   * @param tokens
   * @param posTags
   * @param deps
   */
  public static GrammaticalStructure fromStringReps(List<String> tokens, List<String> posTags, List<String> deps) {
    if (tokens.size() != posTags.size()) {
      throw new RuntimeException(String.format(
              "tokens.size(): %d != pos.size(): %d%n", tokens.size(), posTags
                      .size()));
    }

    List<TreeGraphNode> tgWordNodes = new ArrayList<>(tokens.size());
    List<TreeGraphNode> tgPOSNodes = new ArrayList<>(tokens.size());

    CoreLabel rootLabel = new CoreLabel();
    rootLabel.setValue("ROOT");
    List<IndexedWord> nodeWords = new ArrayList<>(tgPOSNodes.size() + 1);
    nodeWords.add(new IndexedWord(rootLabel));

    UniversalSemanticHeadFinder headFinder = new UniversalSemanticHeadFinder();

    Iterator<String> posIter = posTags.iterator();
    for (String wordString : tokens) {
      String posString = posIter.next();
      CoreLabel wordLabel = new CoreLabel();

View on GitHub (pinned to 1b7edd19c4)