{"record":{"id":"fa3eb42afde31b7c","repo":"stanfordnlp/CoreNLP","slug":"cannot-find-word-in-the-text-of-sentence","errorCode":null,"errorMessage":"Cannot find word  in the text of sentence ","messagePattern":"Cannot find word  in the text of sentence ","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/trees/ud/UniversalDependenciesConverter.java","lineNumber":236,"sourceCode":"      } catch (Exception ex) {\n        log.warn(\"Error running \" + NER_COMBINER_NAME + \" on Tree!  Not applying NER tags!\");\n      }\n    }\n  }\n\n  /**\n   * Break up a provided text input to match a tree's words,\n   * using the whitespace between the words to mark the AfterAnnotation.\n   * We assume a blank space at the end of the sentence\n   */\n  private static void addSpaceAfter(SemanticGraph sg, String text, int graphIdx) {\n    List<IndexedWord> tokens = sg.vertexListSorted();\n    int pos = tokens.get(0).word().length();\n    for (int i = 1; i < tokens.size(); ++i) {\n      String word = tokens.get(i).word();\n      int nextPos = text.indexOf(word, pos);\n      if (nextPos < 0) {\n        throw new RuntimeException(\"Cannot find word \" + word + \" in the text of sentence \" + graphIdx + \"\\n\" + text);\n      }\n      tokens.get(i-1).setAfter(text.substring(pos, nextPos));\n      pos = nextPos + word.length();\n    }\n    tokens.get(tokens.size() - 1).setAfter(\" \");\n  }\n\n  /**\n   * Converts a constituency tree to the English basic, enhanced, or\n   * enhanced++ Universal dependencies representation, or an English basic\n   * Universal dependencies tree to the enhanced or enhanced++ representation.\n   * <p>\n   * Command-line options:<br>\n   * {@code -treeFile}: File with PTB-formatted constituency trees<br>\n   * {@code -conlluFile}: File with basic dependency trees in CoNLL-U format<br>\n   * {@code -textFile}: A file with text to be used as a guide for SpaceAfter (optional)<br>\n   * {@code -outputRepresentation}: \"basic\" (default), \"enhanced\", or \"enhanced++\"<br>\n   * {@code -combineMWTs}: \"False\" (default), \"True\" marks things like it's as MWT","sourceCodeStart":218,"sourceCodeEnd":254,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/trees/ud/UniversalDependenciesConverter.java#L218-L254","documentation":"UniversalDependenciesConverter.addSpaceAfter aligns each token of a dependency graph against the raw sentence text using text.indexOf(word, pos). If a token's word string cannot be found at/after the current position in the text, it throws RuntimeException(\"Cannot find word <word> in the text of sentence <graphIdx>...\"). It indicates the tokenization and the raw text are out of sync.","triggerScenarios":"Converting CoNLL-U data where a token's word form does not appear in the sentence text at the expected offset — e.g. escaped characters, Unicode normalization differences, MWEs spanning spaces, or comments/text lines that don't match the token column.","commonSituations":"CoNLL-U files where # text metadata disagrees with the FORM column; multi-word tokens expanded into syntactic words (e.g. Spanish 'del' -> de+el); text with HTML entities or escaped characters; files edited so the text line no longer matches tokens.","solutions":["Fix the CoNLL-U file so the # text line matches the FORM tokens exactly","Check for Unicode normalization differences between text and word forms and normalize both","Handle multi-word tokens by using the surface form rather than the split word for offset search","Catch RuntimeException per sentence and skip/report the offending sentence instead of aborting conversion"],"exampleFix":"// before\nString word = tokens.get(i).word(); // 'del' split form\nint nextPos = text.indexOf(word, pos); // fails: surface text has 'del'\n// after\nString word = tokens.get(i).word();\nif (text.indexOf(word, pos) < 0) {\n  word = java.text.Normalizer.normalize(word, java.text.Normalizer.Form.NFC);\n  // or fall back to the multi-word-token surface form from the CoNLL-U line\n}\nint nextPos = text.indexOf(word, pos);","handlingStrategy":"try-catch","validationCode":"// verify each FORM token appears in the # text line before converting\nString text = sentenceText.toLowerCase(Locale.ROOT);\nfor (String form : forms) {\n  if (!text.contains(form.toLowerCase(Locale.ROOT))) throw new CoNLLUFormatException(\"FORM not in text: \" + form);\n}","typeGuard":"int idx = text.indexOf(word, pos);\nif (idx < 0) { normalizeBothSides(); idx = text.indexOf(word, pos); }\nif (idx < 0) { skipSentence(graphIdx); return; }","tryCatchPattern":"try { converter.convert(graph, text); } catch (RuntimeException e) { if (e.getMessage() != null && e.getMessage().startsWith(\"Cannot find word\")) { reportMisalignedSentence(graphIdx); } else throw e; }","preventionTips":["Ensure # text metadata matches the FORM column exactly","Normalize Unicode (NFC) in both text and token forms","Handle multi-word tokens using their surface forms for offsets","Validate CoNLL-U alignment in a pre-pass and report offending sentences"],"tags":["conllu","ud-conversion","tokenization-alignment","runtime-exception"],"backgroundTag":"resource-not-found","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}