{"record":{"id":"4854be874502f03c","repo":"stanfordnlp/CoreNLP","slug":"error-invalid-format-token-for-serialized-token","errorCode":null,"errorMessage":"ERROR: Invalid format token for serialized token (only ${bits.length} tokens): ${line}","messagePattern":"ERROR: Invalid format token for serialized token \\(only (.+?) tokens\\): (.+?)","errorType":"exception","errorClass":"RuntimeIOException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/CustomAnnotationSerializer.java","lineNumber":484,"sourceCode":"      sentence.set(SemanticGraphCoreAnnotations.CollapsedDependenciesAnnotation.class, collapsedDeps);\n      SemanticGraph uncollapsedDeps = intermUncollapsedDeps.convertIntermediateGraph(tokens);\n      sentence.set(SemanticGraphCoreAnnotations.BasicDependenciesAnnotation.class, uncollapsedDeps);\n      SemanticGraph ccDeps = intermCcDeps.convertIntermediateGraph(tokens);\n      sentence.set(SemanticGraphCoreAnnotations.CollapsedCCProcessedDependenciesAnnotation.class, ccDeps);\n\n      sentences.add(sentence);\n    }\n    doc.set(CoreAnnotations.SentencesAnnotation.class, sentences);\n\n    return Pair.makePair(doc, is);\n  }\n\n  private static final String SPACE_HOLDER = \"##\";\n\n  private static CoreLabel loadToken(String line, boolean haveExplicitAntecedent) {\n    CoreLabel token = new CoreLabel();\n    String [] bits = line.split(\"\\t\", -1);\n    if(bits.length < 7) throw new RuntimeIOException(\"ERROR: Invalid format token for serialized token (only \" + bits.length + \" tokens): \" + line);\n\n    // word\n    String word = bits[0].replaceAll(SPACE_HOLDER, \" \");\n    token.set(CoreAnnotations.TextAnnotation.class, word);\n    token.set(CoreAnnotations.ValueAnnotation.class, word);\n    // if(word.length() == 0) log.info(\"FOUND 0-LENGTH TOKEN!\");\n\n    // lemma\n    if(bits[1].length() > 0 || bits[0].length() == 0){\n      String lemma = bits[1].replaceAll(SPACE_HOLDER, \" \");\n      token.set(CoreAnnotations.LemmaAnnotation.class, lemma);\n    }\n    // POS tag\n    if(bits[2].length() > 0) token.set(CoreAnnotations.PartOfSpeechAnnotation.class, bits[2]);\n    // NE tag\n    if(bits[3].length() > 0) token.set(CoreAnnotations.NamedEntityTagAnnotation.class, bits[3]);\n    // Normalized NE tag\n    if(bits[4].length() > 0) token.set(CoreAnnotations.NormalizedNamedEntityTagAnnotation.class, bits[4]);","sourceCodeStart":466,"sourceCodeEnd":502,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/CustomAnnotationSerializer.java#L466-L502","documentation":"loadToken parses each token line by tab-splitting with a -1 limit (preserving empty trailing fields). A serialized token must contain at least 7 tab-separated fields (word, value, sentence index, etc.). If the split yields fewer than 7, the token line is truncated and a RuntimeIOException is thrown.","triggerScenarios":"A token line in the serialized annotation has fewer than 7 tab-separated fields — typically a truncated file, a line ending corrupted so two lines merged or a field was lost, or the raw word text contains a literal tab character.","commonSituations":"Files edited with tools that normalize tabs; documents containing literal tab characters in words that were not escaped to the SPACE_HOLDER placeholder; incomplete downloads of .ser.gz files.","solutions":["Regenerate the serialized file with CustomAnnotationSerializer.write from a full pipeline run","Check the failing line for literal tab characters inside the word text (they must be encoded as the '##' SPACE_HOLDER)","Verify the file is not truncated (compare file size/checksum with the source)","Ensure any pre/post-processing scripts preserve all 7 tab-separated fields per token line"],"exampleFix":"// before: writing token text containing a raw tab, breaking field count on read\nword = word.replace(\"##\", \"\"); // loses placeholder escaping; tabs unescaped\n// after: escape both tabs and the placeholder when serializing\nword = word.replace(\"##\", \"####\").replace(\"\\t\", \"##\");","handlingStrategy":"validation","validationCode":"String[] bits = tokenLine.split(\"\\t\", -1);\nif (bits.length < 7) throw new IllegalArgumentException(\"Truncated token line: \" + tokenLine);","typeGuard":"static boolean isCompleteTokenLine(String line) {\n  return line.split(\"\\t\", -1).length >= 7;\n}","tryCatchPattern":"try {\n  serializer.read(in);\n} catch (RuntimeIOException e) {\n  if (e.getMessage().startsWith(\"ERROR: Invalid format token for serialized token\")) {\n    log.severe(\"Token line truncated; regenerate .ser.gz\");\n  } else throw e;\n}","preventionTips":["Escape literal tabs in token text with the SPACE_HOLDER before serializing","Never process .ser.gz files with tab-normalizing text tools","Verify download completeness (size/checksum) of serialized files","Round-trip test after any pre/post-processing scripts touch the file"],"tags":["java","serialization","corenlp","tokens","malformed-input"],"backgroundTag":"invalid-argument-format","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}