{"record":{"id":"20d4b915302d9ba1","repo":"stanfordnlp/CoreNLP","slug":"found-a-tree-which-was-not-properly-binarized-so","errorCode":null,"errorMessage":"Found a tree which was not properly binarized.  So-called binarized tree is as follows:\n${tree.pennString()}","messagePattern":"Found a tree which was not properly binarized\\.  So-called binarized tree is as follows:\n(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/parser/shiftreduce/ShiftReduceParser.java","lineNumber":383,"sourceCode":"  }\n\n  public static List<Tree> binarizeTreebank(Iterable<Tree> treebank, Options op) {\n    TreeBinarizer binarizer = TreeBinarizer.simpleTreeBinarizer(op.tlpParams.headFinder(), op.tlpParams.treebankLanguagePack());\n    BasicCategoryTreeTransformer basicTransformer = new BasicCategoryTreeTransformer(op.langpack());\n    CompositeTreeTransformer transformer = new CompositeTreeTransformer();\n    transformer.addTransformer(binarizer);\n    transformer.addTransformer(basicTransformer);\n\n    List<Tree> transformedTrees = new ArrayList<>();\n    for (Tree tree : treebank) {\n      transformedTrees.add(transformer.transformTree(tree));\n    }\n\n    HeadFinder binaryHeadFinder = new BinaryHeadFinder(op.tlpParams.headFinder());\n    List<Tree> binarizedTrees = new ArrayList<>();\n    for (Tree tree : transformedTrees) {\n      if (!tree.isBinarized()) {\n        log.warn(\"Found a tree which was not properly binarized.  So-called binarized tree is as follows:\\n\" +\n                 tree.pennString());\n        continue;\n      }\n      Trees.convertToCoreLabels(tree);\n      tree.percolateHeadAnnotations(binaryHeadFinder);\n      // Index from 1.  Tools downstream expect index from 1, so for\n      // uses internal to the srparser we have to renormalize the\n      // indices, with the result that here we have to index from 1\n      tree.indexLeaves(1, true);\n      binarizedTrees.add(tree);\n    }\n    return binarizedTrees;\n  }\n\n  public static Set<String> findKnownStates(List<Tree> binarizedTrees) {\n    Set<String> knownStates = Generics.newHashSet();\n    for (Tree tree : binarizedTrees) {\n      findKnownStates(tree, knownStates);","sourceCodeStart":365,"sourceCodeEnd":401,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/parser/shiftreduce/ShiftReduceParser.java#L365-L401","documentation":"ShiftReduceParser training requires every input tree to be binarized (each internal node has at most two children). During binarizeTreebank, if a tree fails tree.isBinarized(), CoreNLP logs this warning with the tree's Penn string and skips the tree entirely — that example is silently dropped from training.","triggerScenarios":"Calling binarizeTreebank (via binarized()) on a treebank whose trees were pre-binarized incorrectly, or were transformed (e.g., punctuation stripping, tree collapasing) after binarization in a way that re-created 3+ child nodes.","commonSituations":"Training a shift-reduce parser on custom treebank files that were binarized with a different tool or wrong options; applying custom tree transformations before passing trees to the parser; a version mismatch between the binarizer and parser code.","solutions":["Run the trees through the parser's expected binarization pipeline (TreeBinarizer with the same head finder / options the parser uses) before training.","Log/inspect the offending trees (the warning prints the full Penn string) to find which transformation broke binarization, and reorder or fix that transform.","Check the training properties: ensure no pre-processing (e.g., custom Annotator or tree transform) un-binarizes trees between binarization and training.","If skipping is intentional, silence is fine — but verify how many trees were skipped; a large count signals a systematic preprocessing bug."],"exampleFix":"// before: passing raw trees to train\nList<Tree> trees = readTrees(\"train.mrg\");\nparser.train(trees, ...);\n\n// after: binarize with the parser's binarizer first\nOptions op = parser.getOp();\nTreeBinarizer binarizer = TreeBinarizer.buildTreeBinarizer(op.tlpParams.headFinder(), op.tlpParams.treebankLanguagePack(),\n    op.trainOptions.unaryAtTop, false, op.trainOptions.trainTreebank.getTreebankLangLangParams(),\n    op.trainOptions.horizFinalMarkov, op.trainOptions.vertFinalMarkov);\nList<Tree> binarized = trees.stream().map(t -> binarizer.transformTree(t)).collect(Collectors.toList());\nparser.train(binarized, ...);","handlingStrategy":"validation","validationCode":"// Validate trees before handing them to the shift-reduce parser\nfor (Tree tree : trainingTrees) {\n  if (!tree.isBinarized()) {\n    throw new IllegalArgumentException(\"Tree not binarized: \" + tree.pennString());\n  }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Binarize training trees with TreeBinarizer using the same head finder/options as the parser","Never apply tree transformations after binarization that can create 3+ child nodes","Count warning occurrences during training — a non-zero systematic rate indicates a preprocessing bug","Keep the treebank preprocessing pipeline version-consistent with the CoreNLP version"],"tags":["nlp","parsing","training-data","treebank"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}