Found a tree which was not properly binarized. So-called…

Error message

Found a tree which was not properly binarized.  So-called binarized tree is as follows:
${tree.pennString()}

What it means

ShiftReduceParser training requires every input tree to be binarized (each internal node has at most two children). During binarizeTreebank, if a tree fails tree.isBinarized(), CoreNLP logs this warning with the tree's Penn string and skips the tree entirely — that example is silently dropped from training.

Solutions

Run the trees through the parser's expected binarization pipeline (TreeBinarizer with the same head finder / options the parser uses) before training.
Log/inspect the offending trees (the warning prints the full Penn string) to find which transformation broke binarization, and reorder or fix that transform.
Check the training properties: ensure no pre-processing (e.g., custom Annotator or tree transform) un-binarizes trees between binarization and training.
If skipping is intentional, silence is fine — but verify how many trees were skipped; a large count signals a systematic preprocessing bug.

Example fix

// before: passing raw trees to train
List<Tree> trees = readTrees("train.mrg");
parser.train(trees, ...);

// after: binarize with the parser's binarizer first
Options op = parser.getOp();
TreeBinarizer binarizer = TreeBinarizer.buildTreeBinarizer(op.tlpParams.headFinder(), op.tlpParams.treebankLanguagePack(),
    op.trainOptions.unaryAtTop, false, op.trainOptions.trainTreebank.getTreebankLangLangParams(),
    op.trainOptions.horizFinalMarkov, op.trainOptions.vertFinalMarkov);
List<Tree> binarized = trees.stream().map(t -> binarizer.transformTree(t)).collect(Collectors.toList());
parser.train(binarized, ...);

Defensive patterns

Strategy: validation

Validate before calling

// Validate trees before handing them to the shift-reduce parser
for (Tree tree : trainingTrees) {
  if (!tree.isBinarized()) {
    throw new IllegalArgumentException("Tree not binarized: " + tree.pennString());
  }
}

Prevention

Binarize training trees with TreeBinarizer using the same head finder/options as the parser
Never apply tree transformations after binarization that can create 3+ child nodes
Count warning occurrences during training — a non-zero systematic rate indicates a preprocessing bug
Keep the treebank preprocessing pipeline version-consistent with the CoreNLP version

When it happens

Trigger: Calling binarizeTreebank (via binarized()) on a treebank whose trees were pre-binarized incorrectly, or were transformed (e.g., punctuation stripping, tree collapasing) after binarization in a way that re-created 3+ child nodes.

Common situations: Training a shift-reduce parser on custom treebank files that were binarized with a different tool or wrong options; applying custom tree transformations before passing trees to the parser; a version mismatch between the binarizer and parser code.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/20d4b915302d9ba1. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/parser/shiftreduce/ShiftReduceParser.java:383

  }

  public static List<Tree> binarizeTreebank(Iterable<Tree> treebank, Options op) {
    TreeBinarizer binarizer = TreeBinarizer.simpleTreeBinarizer(op.tlpParams.headFinder(), op.tlpParams.treebankLanguagePack());
    BasicCategoryTreeTransformer basicTransformer = new BasicCategoryTreeTransformer(op.langpack());
    CompositeTreeTransformer transformer = new CompositeTreeTransformer();
    transformer.addTransformer(binarizer);
    transformer.addTransformer(basicTransformer);

    List<Tree> transformedTrees = new ArrayList<>();
    for (Tree tree : treebank) {
      transformedTrees.add(transformer.transformTree(tree));
    }

    HeadFinder binaryHeadFinder = new BinaryHeadFinder(op.tlpParams.headFinder());
    List<Tree> binarizedTrees = new ArrayList<>();
    for (Tree tree : transformedTrees) {
      if (!tree.isBinarized()) {
        log.warn("Found a tree which was not properly binarized.  So-called binarized tree is as follows:\n" +
                 tree.pennString());
        continue;
      }
      Trees.convertToCoreLabels(tree);
      tree.percolateHeadAnnotations(binaryHeadFinder);
      // Index from 1.  Tools downstream expect index from 1, so for
      // uses internal to the srparser we have to renormalize the
      // indices, with the result that here we have to index from 1
      tree.indexLeaves(1, true);
      binarizedTrees.add(tree);
    }
    return binarizedTrees;
  }

  public static Set<String> findKnownStates(List<Tree> binarizedTrees) {
    Set<String> knownStates = Generics.newHashSet();
    for (Tree tree : binarizedTrees) {
      findKnownStates(tree, knownStates);

View on GitHub (pinned to 1b7edd19c4)