{"record":{"id":"e48236e5dab00805","repo":"stanfordnlp/CoreNLP","slug":"format-error-in-embeddings","errorCode":null,"errorMessage":"format error in embeddings","messagePattern":"format error in embeddings","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/ie/crf/CRFClassifier.java","lineNumber":2212,"sourceCode":"    line = br.readLine();\n\n    while (!line.equals(\"</flags>\")) {\n      // log.info(\"DEBUG: flags line: \"+line);\n      String[] keyValue = line.split(\"=\");\n      // System.err.printf(\"DEBUG: p.setProperty(%s,%s)%n\", keyValue[0],\n      // keyValue[1]);\n      p.setProperty(keyValue[0], keyValue[1]);\n      line = br.readLine();\n    }\n\n    // log.info(\"DEBUG: out from flags\");\n    flags = new SeqClassifierFlags(p);\n\n    if (flags.useEmbedding) {\n      line = br.readLine();\n      toks = line.split(\"\\\\t\");\n      if (!toks[0].equals(\"embeddings.size()=\")) {\n        throw new RuntimeException(\"format error in embeddings\");\n      }\n      int embeddingSize = Integer.parseInt(toks[1]);\n      embeddings = Generics.newHashMap(embeddingSize);\n      count = 0;\n      while (count < embeddingSize) {\n        line = br.readLine().trim();\n        toks = line.split(\"\\\\t\");\n        String word = toks[0];\n        double[] arr = ArrayUtils.toDoubleArray(toks[1].split(\" \"));\n        embeddings.put(word, arr);\n        count++;\n      }\n    }\n\n    // <featureFactory>\n    // edu.stanford.nlp.wordseg.Gale2007ChineseSegmenterFeatureFactory\n    // </featureFactory>\n    line = br.readLine();","sourceCodeStart":2194,"sourceCodeEnd":2230,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/ie/crf/CRFClassifier.java#L2194-L2230","documentation":"Thrown when flags.useEmbedding is true but the line expected after <flags> does not start with 'embeddings.size()='. The embedding section of the text-serialized model is missing or malformed even though the model flags declare embeddings.","triggerScenarios":"CRFClassifier.getClassifier on a model with useEmbedding=true whose embedding section header line is missing, altered, or whose contents were stripped (e.g. embeddings removed to shrink the file without updating flags).","commonSituations":"Manually deleting the embedding block to reduce file size; converting the model between versions losing the embeddings section; truncation at the end of the file where embeddings are stored.","solutions":["Restore the 'embeddings.size()=<int>' section after the <flags> block, matching useEmbedding=true","Either supply a complete model with embeddings or re-serialize with useEmbedding=false if embeddings are not needed","Re-export the model from the training run rather than editing it","Check for end-of-file truncation and compare with the original checksum"],"exampleFix":"// before: embeddings stripped from model while flags still say useEmbedding=true\n// after: re-serialize the full model (or disable embeddings before training/saving)\nflags.useEmbedding = false; // when re-training without embeddings\ncrf.writeModel(new File(\"model.txt\")); // keeps flags and sections consistent","handlingStrategy":"validation","validationCode":"// Verify embeddings section presence when the model declares useEmbedding\nstatic boolean embeddingsSectionConsistent(String modelPath) throws IOException {\n  boolean useEmbedding = false;\n  try (BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(modelPath), \"UTF-8\"))) {\n    String line;\n    while ((line = br.readLine()) != null) {\n      if (line.startsWith(\"useEmbedding=\")) useEmbedding = line.endsWith(\"true\");\n      if (line.startsWith(\"embeddings.size()=\")) return true;\n    }\n  }\n  return !useEmbedding;\n}","typeGuard":null,"tryCatchPattern":"try {\n  CRFClassifier<CoreLabel> model = CRFClassifier.getClassifier(modelPath);\n} catch (RuntimeException e) {\n  if (\"format error in embeddings\".equals(e.getMessage())) {\n    throw new IllegalStateException(\"Model declares useEmbedding=true but the embeddings section is missing/malformed. Re-export the full model.\", e);\n  }\n  throw e;\n}","preventionTips":["Do not strip the embedding section from models without changing useEmbedding=false","Re-export full models rather than editing them to shrink them","Verify the tail of the file is not truncated (embeddings live near the end)","Keep training-time flags consistent with the exported model"],"tags":["java","stanford-corenlp","model-loading","embeddings","file-format"],"backgroundTag":"schema-validation-failed","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}