stanfordnlp/CoreNLP · error · FailedSerializationError

Could not find the token for index

Error message

Could not find the token for index ${index} empty ${emptyIndex}
(${size} known labels)

What it means

When converting a protobuf dependency node back into an IndexedWord, ProtobufAnnotationSerializer must locate the corresponding CoreLabel token. If no document is available it looks up the token by (index, emptyIndex) in the originalLabels map, and throws FailedSerializationError when that lookup returns null — the referenced token is not among the known labels.

Solutions

Serialize and deserialize the full document so the node lookup uses the document path instead of originalLabels.
Ensure every token referenced by the dependency graph is present in the labels passed to the deserializer (no dropped tokens, consistent 1-based indices).
Catch FailedSerializationError and fall back to document-level deserialization or re-run the dependency annotator.

Example fix

// before
SemanticGraph graph = serializer.fromProto(depProto, Optional.empty(), originalLabels); // token missing -> throws
// after
Annotation doc = serializer.fromProto(docProto); // full document carries all tokens; graph resolves correctly

Defensive patterns

Strategy: try-catch

Validate before calling

for (var node : depProto.getNodeList()) {
  if (!originalLabels.containsKey(Pair.makePair(node.getIndex(), node.getEmptyIndex()))) {
    throw new IllegalArgumentException("dependency node references unknown token index " + node.getIndex());
  }
}

Try / catch

try {
  graph = serializer.fromProto(depProto, documentOpt, originalLabels);
} catch (FailedSerializationError e) {
  log.warn("Token lookup failed for dependency node; falling back to full-document deserialization");
  graph = serializer.fromProto(docProto).get(SemanticGraphCoreAnnotations.BasicDependenciesAnnotation.class);
}

Prevention

Always serialize the full document with dependency graphs so the document lookup path is used
Never drop tokens between annotation and serialization
Keep 1-based token indices consistent; beware MWT/copied-token index shifts

When it happens

Trigger: Calling fromProto on a dependency node proto while deserializing without the full document context, where originalLabels lacks an entry for the node's token index — e.g. indices shifted because tokens were pruned, or the node references a token from a different sentence.

Common situations: Deserializing only semantic-graph/dependency protos without the accompanying token list; copied-token or MWT (multi-word token) expansions changing index numbering between serialize and deserialize; custom pipelines that drop tokens before serialization.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/4dd56423453dcecc. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/ProtobufAnnotationSerializer.java:2441

        Integer tokenIndex = token.get(IndexAnnotation.class);
        if (tokenIndex == null) {
          tokenIndex = index;
        }
        Integer emptyIndex = token.getEmptyIndex();
        if (emptyIndex == null) {
          emptyIndex = 0;
        }
        originalLabels.put(tokenIndex, emptyIndex, token);
      }
    }
    for(CoreNLPProtos.DependencyGraph.Node in: proto.getNodeList()){
      CoreLabel token;
      if (document.isPresent()) {
        token = document.get().get(SentencesAnnotation.class).get(in.getSentenceIndex()).get(TokensAnnotation.class).get(in.getIndex() - 1); // token index starts at 1!
      } else {
        token = originalLabels.get(in.getIndex(), in.getEmptyIndex());
        if (token == null) {
          throw new FailedSerializationError("Could not find the token for index " + in.getIndex() + " empty " + in.getEmptyIndex() + "\n(" + originalLabels.size() + " known labels)");
        }
      }
      IndexedWord word;
      if (in.hasCopyAnnotation() && in.getCopyAnnotation() > 0) {
        // TODO: if we make a copy wrapper CoreLabel, use it here instead
        word = new IndexedWord(new CoreLabel(token));
        word.setCopyCount(in.getCopyAnnotation());
      } else {
        word = new IndexedWord(token);
      }

      // for backwards compatibility - new annotations should have
      // these fields set, but annotations older than August 2014 might not
      if (word.docID() == null && docid != null) {
        word.setDocID(docid);
      }
      if (word.sentIndex() < 0 && in.getSentenceIndex() >= 0) {
        word.setSentIndex(in.getSentenceIndex());

View on GitHub (pinned to 1b7edd19c4)