stanfordnlp/CoreNLP · error · RuntimeException

New chunk started, prev chunk not ended yet!

Error message

New chunk started, prev chunk not ended yet!

What it means

LabeledChunkIdentifier.getAnnotatedChunks walks tokens tracking the index where the current chunk began (tokenBegin). If the tag sequence signals a new chunk start while tokenBegin is still >= 0 (the previous chunk was never closed), the tag sequence is internally inconsistent (e.g. I- tag following an incompatible O or B pattern) and a RuntimeException is thrown.

Solutions

1. Fix the tag sequence so every chunk is properly opened and closed (an I-X must follow a compatible B-X or I-X)
2. Normalize the tag scheme before calling getAnnotatedChunks (use CoNLL IOB1/IOB2 consistent sequences)
3. Check the annotator/tagger producing the labels for bugs or scheme mismatches
4. Sanitize sequences by converting orphan I-X tags to B-X before chunk identification

Example fix

// before: orphan I-NP after incompatible tag
// tokens: ["The"] B-NP, ["ran"] B-VP, ["dog"] I-NP  <- new chunk while B-VP chunk open? ensure closure
// after: sanitize orphan continuation tags
if (prevTagType.equals("O") && tag.equals("I-NP")) tag = "B-NP";

static boolean tagsAreChunkConsistent(List<String> tags) {
  boolean inChunk = false;
  for (String t : tags) {
    if (t.startsWith("I-")) { if (!inChunk) return false; }
    else if (t.startsWith("B-")) { if (inChunk) return false; inChunk = true; }
    else inChunk = false;
  }
  return true;
}

try {
  List<CoreMap> chunks = identifier.getAnnotatedChunks(tokens, 0, tokens.size(), labelKey, "B", "I", 0);
} catch (RuntimeException e) {
  if (e.getMessage().equals("New chunk started, prev chunk not ended yet!")) {
    log.warning("Inconsistent IOB sequence; sanitizing orphan I- tags");
  } else throw e;
}

When it happens

Trigger: Feeding a tag sequence where a chunk begins (B-X or I-X after a non-compatible tag) while the previous chunk never ended — typically an I-X tag that follows an incompatible tag type without an intervening B-X, given the isCompatible logic.

Common situations: Hand-crafted or edited IOB/IO tag sequences that violate the chunking grammar; outputs from a custom/buggy tagger; mixing tag schemes (IO vs IOBES) before chunk identification.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Shouldn't happen:
• Error reading saved links
• RuntimeIOException wrapping IOException
• Error creating data exporter
• Error setting up training

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/7cb5f78376934385. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/LabeledChunkIdentifier.java:136

        if (i > 0) {
          prev = tokens.get(i-1);
        }
        Pair<CoreLabel,CoreLabel> p = Pair.makePair(token, prev);
        isCompatible = checkTokensCompatible.test(p);
      }
      if (isEndOfChunk(prevTagType, curTagType) || !isCompatible) {
        int tokenEnd = i;
        if (tokenBegin >= 0 && tokenEnd > tokenBegin) {
          CoreMap chunk = ChunkAnnotationUtils.getAnnotatedChunk(tokens, tokenBegin, tokenEnd, totalTokensOffset,
              tokenChunkKey, textKey, tokenLabelKey);
          chunk.set(labelKey, prevTagType.type);
          chunks.add(chunk);
          tokenBegin = -1;
        }
      }
      if (isStartOfChunk(prevTagType, curTagType) || (!isCompatible && isChunk(curTagType))) {
        if (tokenBegin >= 0) {
          throw new RuntimeException("New chunk started, prev chunk not ended yet!");
        }
        tokenBegin = i;
      }
      prevTagType = curTagType;
    }
    if (tokenBegin >= 0) {
      CoreMap chunk = ChunkAnnotationUtils.getAnnotatedChunk(tokens, tokenBegin, tokens.size(), totalTokensOffset,
          tokenChunkKey, textKey, tokenLabelKey);
      chunk.set(labelKey, prevTagType.type);
      chunks.add(chunk);
    }
//    System.out.println("number of chunks " +  chunks.size());
    return chunks;
  }

  /**
   * Returns whether a chunk ended between the previous and current token.
   *

View on GitHub (pinned to 1b7edd19c4)