{"record":{"id":"cc01d8b7771a7788","repo":"stanfordnlp/CoreNLP","slug":"allsentences-allwords","errorCode":null,"errorMessage":"allSentences != allWords","messagePattern":"allSentences != allWords","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/dcoref/MUCMentionExtractor.java","lineNumber":269,"sourceCode":"                m.goldCorefClusterID = m2.goldCorefClusterID;\n                break;\n              } else if (m2.originalRef == -1) {\n                m2.goldCorefClusterID = m2.mentionID;\n                m.goldCorefClusterID = m2.goldCorefClusterID;\n                break;\n              } else {\n                ref = m2.originalRef;\n              }\n            }\n          }\n        }\n      }\n    }\n\n    docAnno.set(CoreAnnotations.SentencesAnnotation.class, allSentences);\n    stanfordProcessor.annotate(docAnno);\n\n    if(allSentences.size()!=allWords.size()) throw new IllegalStateException(\"allSentences != allWords\");\n    for(int i = 0 ; i< allSentences.size(); i++){\n      List<CoreLabel> annotatedSent = allSentences.get(i).get(CoreAnnotations.TokensAnnotation.class);\n      List<CoreLabel> unannotatedSent = allWords.get(i);\n      List<Mention> mentionInSent = allGoldMentions.get(i);\n      for (Mention m : mentionInSent){\n        m.dependency = allSentences.get(i).get(SemanticGraphCoreAnnotations.EnhancedDependenciesAnnotation.class);\n      }\n      if(annotatedSent.size() != unannotatedSent.size()){\n        throw new IllegalStateException(\"annotatedSent != unannotatedSent\");\n      }\n      for (int j = 0, sz = annotatedSent.size(); j < sz; j++){\n        CoreLabel annotatedWord = annotatedSent.get(j);\n        CoreLabel unannotatedWord = unannotatedSent.get(j);\n        if ( ! annotatedWord.get(CoreAnnotations.TextAnnotation.class).equals(unannotatedWord.get(CoreAnnotations.TextAnnotation.class))) {\n          throw new IllegalStateException(\"annotatedWord != unannotatedWord\");\n        }\n      }\n      allWords.set(i, annotatedSent);","sourceCodeStart":251,"sourceCodeEnd":287,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/dcoref/MUCMentionExtractor.java#L251-L287","documentation":"MUCMentionExtractor.nextDoc() runs the Stanford coreference pipeline on a document: it tokenizes the raw text into allWords while gold annotations provide allSentences, then calls stanfordProcessor.annotate and requires that sentence splitting produced exactly one sentence per pre-split input sentence. When the sentence counts diverge the internal invariant is broken and an IllegalStateException is thrown, since the following loop zips the two lists index by index.","triggerScenarios":"Calling nextDoc() on a document where the annotator's sentence splitter produces a different number of sentences than the pre-split allWords list — e.g. MUC/ACE input files whose gold sentence boundaries disagree with what the tokenizer/sentence annotator derives after annotation, or a pipeline configuration that re-splits sentences.","commonSituations":"Running MUC/ACE coreference experiments with gold mentions but a pipeline whose ssplit settings differ from the ones used to build the gold data; documents with unusual formatting (empty lines, quote handling) that cause ssplit to merge or split sentences; version changes where the default sentence splitter behavior changed.","solutions":["Align the pipeline properties (tokenize.whitespace, ssplit.eolonly, etc.) with the configuration used to produce the gold allWords split","Verify input documents are not corrupted (missing sentence delimiters, truncated lines) in the MUC/ACE key files","Catch IllegalStateException and skip/flag the offending document, logging its ID for inspection","Ensure allSentences is populated only from the annotated document's SentencesAnnotation consistently with how allWords was built"],"exampleFix":"// before\nProperties props = new Properties();\nprops.setProperty(\"annotators\", \"tokenize,ssplit,pos,lemma,ner,parse\");\n// after: force the splitter to respect the original line-per-sentence split\nprops.setProperty(\"annotators\", \"tokenize,ssplit,pos,lemma,ner,parse\");\nprops.setProperty(\"ssplit.eolonly\", \"true\");","handlingStrategy":"validation","validationCode":"// before calling nextDoc, sanity-check document structure\nList<CoreLabel> words = /* allWords built from the document */;\nif (words == null || words.isEmpty()) throw new IllegalArgumentException(\"empty doc\");\n// keep the pipeline ssplit settings consistent with how allWords was split\nprops.setProperty(\"ssplit.eolonly\", \"true\");","typeGuard":null,"tryCatchPattern":"try {\n  Document doc = mentionExtractor.nextDoc();\n} catch (IllegalStateException e) {\n  logger.warning(\"Skipping document with sentence-count mismatch: \" + e.getMessage());\n  // advance to next document\n}","preventionTips":["Align ssplit/tokenize properties with the gold sentence split (e.g. ssplit.eolonly=true for line-per-sentence corpora)","Validate input key files for truncated or malformed sentences before batch runs","Run a small sample of the corpus first and assert allSentences.size()==allWords.size()","Pin your CoreNLP version; sentence-splitter behavior can change between releases"],"tags":["corenlp","coreference","sentence-splitting","invariant-violation"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}