{"record":{"id":"4dd56423453dcecc","repo":"stanfordnlp/CoreNLP","slug":"could-not-find-the-token-for-index-index-empty","errorCode":null,"errorMessage":"Could not find the token for index ${index} empty ${emptyIndex}\n(${size} known labels)","messagePattern":"Could not find the token for index (.+?) empty (.+?)\n\\((.+?) known labels\\)","errorType":"exception","errorClass":"FailedSerializationError","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/ProtobufAnnotationSerializer.java","lineNumber":2441,"sourceCode":"        Integer tokenIndex = token.get(IndexAnnotation.class);\n        if (tokenIndex == null) {\n          tokenIndex = index;\n        }\n        Integer emptyIndex = token.getEmptyIndex();\n        if (emptyIndex == null) {\n          emptyIndex = 0;\n        }\n        originalLabels.put(tokenIndex, emptyIndex, token);\n      }\n    }\n    for(CoreNLPProtos.DependencyGraph.Node in: proto.getNodeList()){\n      CoreLabel token;\n      if (document.isPresent()) {\n        token = document.get().get(SentencesAnnotation.class).get(in.getSentenceIndex()).get(TokensAnnotation.class).get(in.getIndex() - 1); // token index starts at 1!\n      } else {\n        token = originalLabels.get(in.getIndex(), in.getEmptyIndex());\n        if (token == null) {\n          throw new FailedSerializationError(\"Could not find the token for index \" + in.getIndex() + \" empty \" + in.getEmptyIndex() + \"\\n(\" + originalLabels.size() + \" known labels)\");\n        }\n      }\n      IndexedWord word;\n      if (in.hasCopyAnnotation() && in.getCopyAnnotation() > 0) {\n        // TODO: if we make a copy wrapper CoreLabel, use it here instead\n        word = new IndexedWord(new CoreLabel(token));\n        word.setCopyCount(in.getCopyAnnotation());\n      } else {\n        word = new IndexedWord(token);\n      }\n\n      // for backwards compatibility - new annotations should have\n      // these fields set, but annotations older than August 2014 might not\n      if (word.docID() == null && docid != null) {\n        word.setDocID(docid);\n      }\n      if (word.sentIndex() < 0 && in.getSentenceIndex() >= 0) {\n        word.setSentIndex(in.getSentenceIndex());","sourceCodeStart":2423,"sourceCodeEnd":2459,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/ProtobufAnnotationSerializer.java#L2423-L2459","documentation":"When converting a protobuf dependency node back into an IndexedWord, ProtobufAnnotationSerializer must locate the corresponding CoreLabel token. If no document is available it looks up the token by (index, emptyIndex) in the originalLabels map, and throws FailedSerializationError when that lookup returns null — the referenced token is not among the known labels.","triggerScenarios":"Calling fromProto on a dependency node proto while deserializing without the full document context, where originalLabels lacks an entry for the node's token index — e.g. indices shifted because tokens were pruned, or the node references a token from a different sentence.","commonSituations":"Deserializing only semantic-graph/dependency protos without the accompanying token list; copied-token or MWT (multi-word token) expansions changing index numbering between serialize and deserialize; custom pipelines that drop tokens before serialization.","solutions":["Serialize and deserialize the full document so the node lookup uses the document path instead of originalLabels.","Ensure every token referenced by the dependency graph is present in the labels passed to the deserializer (no dropped tokens, consistent 1-based indices).","Catch FailedSerializationError and fall back to document-level deserialization or re-run the dependency annotator."],"exampleFix":"// before\nSemanticGraph graph = serializer.fromProto(depProto, Optional.empty(), originalLabels); // token missing -> throws\n// after\nAnnotation doc = serializer.fromProto(docProto); // full document carries all tokens; graph resolves correctly","handlingStrategy":"try-catch","validationCode":"for (var node : depProto.getNodeList()) {\n  if (!originalLabels.containsKey(Pair.makePair(node.getIndex(), node.getEmptyIndex()))) {\n    throw new IllegalArgumentException(\"dependency node references unknown token index \" + node.getIndex());\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  graph = serializer.fromProto(depProto, documentOpt, originalLabels);\n} catch (FailedSerializationError e) {\n  log.warn(\"Token lookup failed for dependency node; falling back to full-document deserialization\");\n  graph = serializer.fromProto(docProto).get(SemanticGraphCoreAnnotations.BasicDependenciesAnnotation.class);\n}","preventionTips":["Always serialize the full document with dependency graphs so the document lookup path is used","Never drop tokens between annotation and serialization","Keep 1-based token indices consistent; beware MWT/copied-token index shifts"],"tags":["serialization","protobuf","dependencies","token-lookup"],"backgroundTag":"record-not-found","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}