{"record":{"id":"7cb5f78376934385","repo":"stanfordnlp/CoreNLP","slug":"new-chunk-started-prev-chunk-not-ended-yet","errorCode":null,"errorMessage":"New chunk started, prev chunk not ended yet!","messagePattern":"New chunk started, prev chunk not ended yet!","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/LabeledChunkIdentifier.java","lineNumber":136,"sourceCode":"        if (i > 0) {\n          prev = tokens.get(i-1);\n        }\n        Pair<CoreLabel,CoreLabel> p = Pair.makePair(token, prev);\n        isCompatible = checkTokensCompatible.test(p);\n      }\n      if (isEndOfChunk(prevTagType, curTagType) || !isCompatible) {\n        int tokenEnd = i;\n        if (tokenBegin >= 0 && tokenEnd > tokenBegin) {\n          CoreMap chunk = ChunkAnnotationUtils.getAnnotatedChunk(tokens, tokenBegin, tokenEnd, totalTokensOffset,\n              tokenChunkKey, textKey, tokenLabelKey);\n          chunk.set(labelKey, prevTagType.type);\n          chunks.add(chunk);\n          tokenBegin = -1;\n        }\n      }\n      if (isStartOfChunk(prevTagType, curTagType) || (!isCompatible && isChunk(curTagType))) {\n        if (tokenBegin >= 0) {\n          throw new RuntimeException(\"New chunk started, prev chunk not ended yet!\");\n        }\n        tokenBegin = i;\n      }\n      prevTagType = curTagType;\n    }\n    if (tokenBegin >= 0) {\n      CoreMap chunk = ChunkAnnotationUtils.getAnnotatedChunk(tokens, tokenBegin, tokens.size(), totalTokensOffset,\n          tokenChunkKey, textKey, tokenLabelKey);\n      chunk.set(labelKey, prevTagType.type);\n      chunks.add(chunk);\n    }\n//    System.out.println(\"number of chunks \" +  chunks.size());\n    return chunks;\n  }\n\n  /**\n   * Returns whether a chunk ended between the previous and current token.\n   *","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/LabeledChunkIdentifier.java#L118-L154","documentation":"LabeledChunkIdentifier.getAnnotatedChunks walks tokens tracking the index where the current chunk began (tokenBegin). If the tag sequence signals a new chunk start while tokenBegin is still >= 0 (the previous chunk was never closed), the tag sequence is internally inconsistent (e.g. I- tag following an incompatible O or B pattern) and a RuntimeException is thrown.","triggerScenarios":"Feeding a tag sequence where a chunk begins (B-X or I-X after a non-compatible tag) while the previous chunk never ended — typically an I-X tag that follows an incompatible tag type without an intervening B-X, given the isCompatible logic.","commonSituations":"Hand-crafted or edited IOB/IO tag sequences that violate the chunking grammar; outputs from a custom/buggy tagger; mixing tag schemes (IO vs IOBES) before chunk identification.","solutions":["Fix the tag sequence so every chunk is properly opened and closed (an I-X must follow a compatible B-X or I-X)","Normalize the tag scheme before calling getAnnotatedChunks (use CoNLL IOB1/IOB2 consistent sequences)","Check the annotator/tagger producing the labels for bugs or scheme mismatches","Sanitize sequences by converting orphan I-X tags to B-X before chunk identification"],"exampleFix":"// before: orphan I-NP after incompatible tag\n// tokens: [\"The\"] B-NP, [\"ran\"] B-VP, [\"dog\"] I-NP  <- new chunk while B-VP chunk open? ensure closure\n// after: sanitize orphan continuation tags\nif (prevTagType.equals(\"O\") && tag.equals(\"I-NP\")) tag = \"B-NP\";","handlingStrategy":"try-catch","validationCode":"static boolean tagsAreChunkConsistent(List<String> tags) {\n  boolean inChunk = false;\n  for (String t : tags) {\n    if (t.startsWith(\"I-\")) { if (!inChunk) return false; }\n    else if (t.startsWith(\"B-\")) { if (inChunk) return false; inChunk = true; }\n    else inChunk = false;\n  }\n  return true;\n}","typeGuard":null,"tryCatchPattern":"try {\n  List<CoreMap> chunks = identifier.getAnnotatedChunks(tokens, 0, tokens.size(), labelKey, \"B\", \"I\", 0);\n} catch (RuntimeException e) {\n  if (e.getMessage().equals(\"New chunk started, prev chunk not ended yet!\")) {\n    log.warning(\"Inconsistent IOB sequence; sanitizing orphan I- tags\");\n  } else throw e;\n}","preventionTips":["Sanitize orphan I-X tags to B-X before chunk identification","Use one tag scheme (IOB1 or IOB2) consistently across tagger and chunk identifier","Test getAnnotatedChunks against your tagger's full output before production","Avoid hand-editing tagged sequences without re-validating the chunk grammar"],"tags":["java","corenlp","chunking","iob-tags","invalid-state"],"backgroundTag":"invalid-state-transition","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}