stanfordnlp/CoreNLP · error · RuntimeIOException

ERROR: Invalid format token for serialized token

Error message

ERROR: Invalid format token for serialized token (only ${bits.length} tokens): ${line}

What it means

loadToken parses each token line by tab-splitting with a -1 limit (preserving empty trailing fields). A serialized token must contain at least 7 tab-separated fields (word, value, sentence index, etc.). If the split yields fewer than 7, the token line is truncated and a RuntimeIOException is thrown.

Solutions

1. Regenerate the serialized file with CustomAnnotationSerializer.write from a full pipeline run
2. Check the failing line for literal tab characters inside the word text (they must be encoded as the '##' SPACE_HOLDER)
3. Verify the file is not truncated (compare file size/checksum with the source)
4. Ensure any pre/post-processing scripts preserve all 7 tab-separated fields per token line

Example fix

// before: writing token text containing a raw tab, breaking field count on read
word = word.replace("##", ""); // loses placeholder escaping; tabs unescaped
// after: escape both tabs and the placeholder when serializing
word = word.replace("##", "####").replace("\t", "##");

String[] bits = tokenLine.split("\t", -1);
if (bits.length < 7) throw new IllegalArgumentException("Truncated token line: " + tokenLine);

static boolean isCompleteTokenLine(String line) {
  return line.split("\t", -1).length >= 7;
}

try {
  serializer.read(in);
} catch (RuntimeIOException e) {
  if (e.getMessage().startsWith("ERROR: Invalid format token for serialized token")) {
    log.severe("Token line truncated; regenerate .ser.gz");
  } else throw e;
}

When it happens

Trigger: A token line in the serialized annotation has fewer than 7 tab-separated fields — typically a truncated file, a line ending corrupted so two lines merged or a field was lost, or the raw word text contains a literal tab character.

Common situations: Files edited with tools that normalize tabs; documents containing literal tab characters in words that were not escaped to the SPACE_HOLDER placeholder; incomplete downloads of .ser.gz files.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• ERROR: Invalid dependency node line
• ERROR: Invalid format for dependency graph
• ERROR: Incorrect format for the serialized coref graph
• RuntimeIOException wrapping IOException
• Failed to save classifier

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/4854be874502f03c. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/CustomAnnotationSerializer.java:484

      sentence.set(SemanticGraphCoreAnnotations.CollapsedDependenciesAnnotation.class, collapsedDeps);
      SemanticGraph uncollapsedDeps = intermUncollapsedDeps.convertIntermediateGraph(tokens);
      sentence.set(SemanticGraphCoreAnnotations.BasicDependenciesAnnotation.class, uncollapsedDeps);
      SemanticGraph ccDeps = intermCcDeps.convertIntermediateGraph(tokens);
      sentence.set(SemanticGraphCoreAnnotations.CollapsedCCProcessedDependenciesAnnotation.class, ccDeps);

      sentences.add(sentence);
    }
    doc.set(CoreAnnotations.SentencesAnnotation.class, sentences);

    return Pair.makePair(doc, is);
  }

  private static final String SPACE_HOLDER = "##";

  private static CoreLabel loadToken(String line, boolean haveExplicitAntecedent) {
    CoreLabel token = new CoreLabel();
    String [] bits = line.split("\t", -1);
    if(bits.length < 7) throw new RuntimeIOException("ERROR: Invalid format token for serialized token (only " + bits.length + " tokens): " + line);

    // word
    String word = bits[0].replaceAll(SPACE_HOLDER, " ");
    token.set(CoreAnnotations.TextAnnotation.class, word);
    token.set(CoreAnnotations.ValueAnnotation.class, word);
    // if(word.length() == 0) log.info("FOUND 0-LENGTH TOKEN!");

    // lemma
    if(bits[1].length() > 0 || bits[0].length() == 0){
      String lemma = bits[1].replaceAll(SPACE_HOLDER, " ");
      token.set(CoreAnnotations.LemmaAnnotation.class, lemma);
    }
    // POS tag
    if(bits[2].length() > 0) token.set(CoreAnnotations.PartOfSpeechAnnotation.class, bits[2]);
    // NE tag
    if(bits[3].length() > 0) token.set(CoreAnnotations.NamedEntityTagAnnotation.class, bits[3]);
    // Normalized NE tag
    if(bits[4].length() > 0) token.set(CoreAnnotations.NormalizedNamedEntityTagAnnotation.class, bits[4]);

View on GitHub (pinned to 1b7edd19c4)