{"record":{"id":"fad8b7fd49f905a6","repo":"keras-team/keras","slug":"idf-weights-must-be-the-same-length-as-vocabular","errorCode":null,"errorMessage":"`idf_weights` must be the same length as vocabulary. len(idf_weights) is {len(idf_weights)}; len(vocabulary) is {len(vocabulary)}","messagePattern":"`idf_weights` must be the same length as vocabulary\\. len\\(idf_weights\\) is (.+?); len\\(vocabulary\\) is (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"keras/src/layers/preprocessing/index_lookup.py","lineNumber":560,"sourceCode":"                \"OOV tokens, or include them only at the start of the \"\n                f\"vocabulary in precisely this order: {special_tokens}. \"\n                f\"Received: oov_token={self.oov_token} at \"\n                f\"vocabulary index {oov_index}\"\n            )\n\n        new_vocab_size = token_start + len(tokens)\n        if self.max_tokens is not None and (new_vocab_size > self.max_tokens):\n            raise ValueError(\n                \"Attempted to set a vocabulary larger than the maximum vocab \"\n                f\"size. Received vocabulary size is {new_vocab_size}; \"\n                f\"`max_tokens` is {self.max_tokens}.\"\n            )\n        self.lookup_table = self._lookup_table_from_tokens(tokens)\n        self._record_vocabulary_size()\n\n        if self.output_mode == \"tf_idf\" and idf_weights is not None:\n            if len(vocabulary) != len(idf_weights):\n                raise ValueError(\n                    \"`idf_weights` must be the same length as vocabulary. \"\n                    f\"len(idf_weights) is {len(idf_weights)}; \"\n                    f\"len(vocabulary) is {len(vocabulary)}\"\n                )\n            idf_weights = self._convert_to_ndarray(idf_weights)\n            if idf_weights.ndim != 1:\n                raise ValueError(\n                    \"TF-IDF data must be a 1-index array. \"\n                    f\"Received: type(idf_weights)={type(idf_weights)}\"\n                )\n\n            # If the passed vocabulary has no special tokens, we need to pad the\n            # front of idf_weights. We don't have real document frequencies for\n            # these tokens so we will use an average of all idf_weights passed\n            # in as a reasonable default.\n            if found_special_tokens:\n                front_padding = 0\n                front_padding_value = 0","sourceCodeStart":542,"sourceCodeEnd":578,"githubUrl":"https://github.com/keras-team/keras/blob/7a34a03db60bf60042242d6a556fc3be119046a5/keras/src/layers/preprocessing/index_lookup.py#L542-L578","documentation":"In TF-IDF mode every token needs exactly one idf weight; a length mismatch would misalign the scaling, so set_vocabulary enforces `len(idf_weights) == len(vocabulary)`.","triggerScenarios":"`set_vocabulary(vocab, idf_weights=w)` where w was computed against a different version of the vocabulary (extra or missing tokens).","commonSituations":"Recomputing the vocabulary after idf weights were fitted; off-by-one from special-token slots present in one array but not the other.","solutions":["Recompute idf on the exact vocabulary being set — fit sklearn TfidfVectorizer on the same corpus and use its vocabulary and idf_ together.","Align explicitly before the call: `assert len(vocab) == len(idf_weights)`."],"exampleFix":"# before\nlayer.set_vocabulary(new_vocab, idf_weights=old_idf)\n\n# after\nassert len(new_vocab) == len(idf), (len(new_vocab), len(idf))\nlayer.set_vocabulary(new_vocab, idf_weights=idf)","handlingStrategy":"validation","validationCode":"assert len(vocab) == len(idf_weights), (len(vocab), len(idf_weights))\nlayer.set_vocabulary(vocab, idf_weights=idf_weights)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Fit and store vocabulary and idf weights as one atomic artifact.","When the corpus changes, refit both together — never reuse old weights with a new vocab."],"tags":["keras","preprocessing","index-lookup","tf-idf","length-mismatch"],"backgroundTag":"length-mismatch","analyzedSha":"7a34a03db60bf60042242d6a556fc3be119046a5","analyzedAt":"2026-08-25T21:25:25.994Z","schemaVersion":2},"datasetVersion":"2026-08-26T02:17:13.382Z"}