{"record":{"id":"d3d55e8d5e50dd7f","repo":"stanfordnlp/CoreNLP","slug":"messy-token","errorCode":null,"errorMessage":"Messy token: ","messagePattern":"Messy token: ","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/arabic/pipeline/LabeledATBDataset.java","lineNumber":91,"sourceCode":"      \n      String rawWord = node.firstChild().value().trim();\n      \n      Matcher left = leftClitic.matcher(rawWord);\n      boolean hasLeft = left.find();\n      Matcher right = rightClitic.matcher(rawWord);\n      boolean hasRight = right.find();\n      \n      if(rawTag.equals(\"PUNC\") || !(hasRight || hasLeft)) {\n        node.firstChild().setValue(\"XSEG\");\n      \n      } else if(hasRight && hasLeft){\n        node.firstChild().setValue(\"SEGC\");\n      } else if(hasRight) {\n        node.firstChild().setValue(\"SEGL\");\n      } else if(hasLeft) {\n        node.firstChild().setValue(\"SEGR\");\n      } else {\n        throw new RuntimeException(\"Messy token: \" + rawWord);\n      }\n    }\n  }\n}\n","sourceCodeStart":73,"sourceCodeEnd":96,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/arabic/pipeline/LabeledATBDataset.java#L73-L96","documentation":"LabeledATBDataset's processPreterminal rewrites Arabic preterminal tokens to segmentation-prefixed forms (SEGC/SEGL/SEGR) based on which sides of the word have segment markers. If a token has neither left nor right boundary markers, the pipeline cannot classify it and throws this RuntimeException with the offending raw word.","triggerScenarios":"Processing an ATB-parsed Arabic tree whose preterminal leaves lack the expected segment continuation/boundary markers (# or equivalent), e.g. feeding unsegmented or differently-preprocessed text into the ATB dataset transformation.","commonSituations":"Using trees produced by a different tokenizer/segmenter, ATB releases preprocessed with non-standard scripts, or mixing raw and segmented trees in one corpus load.","solutions":["Inspect the reported rawWord for missing segment boundary markers","Pre-segment the input text with the same Arabic segmenter assumed by the ATB pipeline","Check the trees were produced from the same ATB release/preprocessing as the pipeline expects","Skip or sanitize tokens without markers before calling processAppliedTree/processPreterminal"],"exampleFix":"// before\ntree = dataset.processAppliedTree(tree);\n// after\nfor (Tree leaf : tree.getLeaves()) {\n  String w = leaf.value();\n  if (w == null || !(w.contains(\"+\") || w.endsWith(\"#\") || w.startsWith(\"#\"))) {\n    continue; // or log & fix segmentation\n  }\n}\ntree = dataset.processAppliedTree(tree);","handlingStrategy":"validation","validationCode":"for (Tree leaf : tree.getLeaves()) {\n  String w = leaf.value();\n  if (w == null || w.isEmpty()) throw new IllegalArgumentException(\"Empty leaf\");\n  boolean hasSegMark = w.contains(\"+\") || w.contains(\"#\");\n  if (!hasSegMark) throw new IllegalArgumentException(\"Leaf lacks segmentation markers: \" + w);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pre-segment Arabic text with the segmenter expected by the ATB pipeline","Validate tree leaves for segmentation markers before loading","Keep ATB release and preprocessing consistent across corpus files"],"tags":["arabic-nlp","treebank","preprocessing"],"backgroundTag":"invalid-argument-value","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}