{"record":{"id":"20b9c6d956fcab89","repo":"stanfordnlp/CoreNLP","slug":"unknown-word-vector-not-specified-in-the-word-vect-20b9c6","errorCode":null,"errorMessage":"Unknown word vector not specified in the word vector file","messagePattern":"Unknown word vector not specified in the word vector file","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/sentiment/SentimentModel.java","lineNumber":517,"sourceCode":"      wordVectors.put(word, vector);\n    }\n  }\n\n  void readWordVectors() {\n    Embedding embedding = new Embedding(op.wordVectors, op.numHid);\n    this.wordVectors = Generics.newTreeMap();\n//    Map<String, SimpleMatrix> rawWordVectors = NeuralUtils.readRawWordVectors(op.wordVectors, op.numHid);\n//    for (String word : rawWordVectors.keySet()) {\n    for (String word : embedding.keySet()) {\n      // TODO: factor out unknown word vector code from DVParser\n      wordVectors.put(word, embedding.get(word));\n    }\n\n    String unkWord = op.unkWord;\n    SimpleMatrix unknownWordVector = wordVectors.get(unkWord);\n    wordVectors.put(UNKNOWN_WORD, unknownWordVector);\n    if (unknownWordVector == null) {\n      throw new RuntimeException(\"Unknown word vector not specified in the word vector file\");\n    }\n\n  }\n\n  public int totalParamSize() {\n    int totalSize = 0;\n    // binaryTensorSize was set to 0 if useTensors=false\n    totalSize = numBinaryMatrices * (binaryTransformSize + binaryClassificationSize + binaryTensorSize);\n    totalSize += numUnaryMatrices * unaryClassificationSize;\n    totalSize += wordVectors.size() * numHid;\n    return totalSize;\n  }\n\n  public double[] paramsToVector() {\n    int totalSize = totalParamSize();\n    return NeuralUtils.paramsToVector(totalSize, binaryTransform.valueIterator(), binaryClassification.valueIterator(), SimpleTensor.iteratorSimpleMatrix(binaryTensors.valueIterator()), unaryClassification.values().iterator(), wordVectors.values().iterator());\n  }\n","sourceCodeStart":499,"sourceCodeEnd":535,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/sentiment/SentimentModel.java#L499-L535","documentation":"readWordVectors loads pretrained vectors from op.wordVectorFile and then looks up op.unkWord to serve as the vector for out-of-vocabulary words (stored under UNKNOWN_WORD). If the vector file does not contain an entry for the unkWord token, the unknown word vector is null and it throws RuntimeException.","triggerScenarios":"Constructing a SentimentModel with a wordVectorFile that lacks a line for the token configured in op.unkWord (default \"*UNKNOWN*\").","commonSituations":"Using custom word2vec/GloVe embeddings that don't include the default unknown token; changing op.unkWord without adding that token to the embeddings; trimming rare tokens from the embedding file during preprocessing.","solutions":["Add a line for the unknown token (e.g. \"*UNKNOWN*\" followed by numHid values) to the word vector file","Set op.unkWord to a token that actually exists in your word vector file","Generate and append a random or zero vector entry for the unknown word before training"],"exampleFix":"// before\nRNNOptions op = new RNNOptions();\nop.wordVectorFile = \"glove.txt\"; // has no *UNKNOWN* entry\nSentimentModel model = new SentimentModel(op, trees); // throws\n// after\necho \"*UNKNOWN* 0.01 0.02 ... (25 values)\" >> glove.txt\nRNNOptions op = new RNNOptions();\nop.wordVectorFile = \"glove.txt\";\nop.unkWord = \"*UNKNOWN*\";\nSentimentModel model = new SentimentModel(op, trees);","handlingStrategy":"validation","validationCode":"// Verify the unknown token exists in the embedding file before model construction\nString unk = op.unkWord; // default \"*UNKNOWN*\"\nboolean found = false;\ntry (BufferedReader r = Files.newBufferedReader(Paths.get(op.wordVectorFile))) {\n  String line;\n  while ((line = r.readLine()) != null) {\n    if (line.startsWith(unk + \" \")) { found = true; break; }\n  }\n}\nif (!found) throw new IllegalStateException(\"Word vector file missing unknown token: \" + unk);","typeGuard":null,"tryCatchPattern":"try {\n  model = new SentimentModel(op, trees);\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Unknown word vector\")) {\n    op.unkWord = resolveExistingUnkToken(op.wordVectorFile); // pick a token present in the file\n    model = new SentimentModel(op, trees);\n  } else { throw e; }\n}","preventionTips":["Include an explicit *UNKNOWN* vector in every embedding file you ship","Keep op.unkWord in sync with the token actually present in the vector file","Script your embedding preprocessing to always append an unknown-word entry","Smoke-test SentimentModel construction early in CI with the real vector file"],"tags":["java","stanford-nlp","sentiment","word-vectors"],"backgroundTag":"resource-not-found","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}