{"record":{"id":"7c24a90c615211f0","repo":"keras-team/keras","slug":"tf-idf-data-must-be-a-1-index-array-received-typ","errorCode":null,"errorMessage":"TF-IDF data must be a 1-index array. Received: type(idf_weights)={type(idf_weights)}","messagePattern":"TF-IDF data must be a 1-index array\\. Received: type\\(idf_weights\\)=(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"keras/src/layers/preprocessing/index_lookup.py","lineNumber":567,"sourceCode":"        if self.max_tokens is not None and (new_vocab_size > self.max_tokens):\n            raise ValueError(\n                \"Attempted to set a vocabulary larger than the maximum vocab \"\n                f\"size. Received vocabulary size is {new_vocab_size}; \"\n                f\"`max_tokens` is {self.max_tokens}.\"\n            )\n        self.lookup_table = self._lookup_table_from_tokens(tokens)\n        self._record_vocabulary_size()\n\n        if self.output_mode == \"tf_idf\" and idf_weights is not None:\n            if len(vocabulary) != len(idf_weights):\n                raise ValueError(\n                    \"`idf_weights` must be the same length as vocabulary. \"\n                    f\"len(idf_weights) is {len(idf_weights)}; \"\n                    f\"len(vocabulary) is {len(vocabulary)}\"\n                )\n            idf_weights = self._convert_to_ndarray(idf_weights)\n            if idf_weights.ndim != 1:\n                raise ValueError(\n                    \"TF-IDF data must be a 1-index array. \"\n                    f\"Received: type(idf_weights)={type(idf_weights)}\"\n                )\n\n            # If the passed vocabulary has no special tokens, we need to pad the\n            # front of idf_weights. We don't have real document frequencies for\n            # these tokens so we will use an average of all idf_weights passed\n            # in as a reasonable default.\n            if found_special_tokens:\n                front_padding = 0\n                front_padding_value = 0\n            else:\n                front_padding = token_start\n                front_padding_value = np.average(idf_weights)\n            # If pad_to_max_tokens is true, and max_tokens is greater than our\n            # total vocab size, we need to pad the back of idf_weights with\n            # zeros as well.\n            back_padding_value = 0","sourceCodeStart":549,"sourceCodeEnd":585,"githubUrl":"https://github.com/keras-team/keras/blob/7a34a03db60bf60042242d6a556fc3be119046a5/keras/src/layers/preprocessing/index_lookup.py#L549-L585","documentation":"IndexLookup.set_vocabulary() requires idf_weights to be a 1-D array when output_mode='tf_idf'. After converting the input to an ndarray, the layer checks ndim and rejects anything that is not a flat vector, because each weight must map to exactly one vocabulary entry.","triggerScenarios":"Calling layer.set_vocabulary(vocab, idf_weights=weights) or constructing IndexLookup/TextVectorization(output_mode='tf_idf', vocabulary=..., idf_weights=...) with idf_weights shaped 2-D or 0-D, e.g. a column vector [[1.2],[0.7]], a matrix, or a scalar.","commonSituations":"Loading IDF weights from a saved model or CSV where reshape introduced an extra axis; passing weights reshaped via .reshape(-1,1); computing weights with sklearn TfidfVectorizer.idf_ then reshaping incorrectly.","solutions":["Reshape the weights to 1-D before passing: np.asarray(idf_weights).ravel() or .reshape(-1)","Check the shape first: assert np.ndim(idf_weights) == 1","If weights come from a file, load with np.loadtxt (returns 1-D) rather than ndmin=2"],"exampleFix":"// before\nlayer.set_vocabulary(vocab, idf_weights=w.reshape(-1, 1))\n// after\nimport numpy as np\nlayer.set_vocabulary(vocab, idf_weights=np.asarray(w).ravel())","handlingStrategy":"validation","validationCode":"import numpy as np\nw = np.asarray(idf_weights)\nif w.ndim != 1:\n    w = w.ravel()\nassert len(w) == len(vocab)","typeGuard":"def is_flat_weights(w):\n    import numpy as np\n    return np.ndim(w) == 1","tryCatchPattern":"try:\n    layer.set_vocabulary(vocab, idf_weights=w)\nexcept ValueError as e:\n    if '1-index array' in str(e):\n        layer.set_vocabulary(vocab, idf_weights=np.asarray(w).ravel())\n    else:\n        raise","preventionTips":["Always ravel() idf weights before set_vocabulary","Assert len(idf_weights) == len(vocabulary) before the call"],"tags":["keras","preprocessing","tf-idf","shape-validation"],"backgroundTag":"array-shape-mismatch","analyzedSha":"7a34a03db60bf60042242d6a556fc3be119046a5","analyzedAt":"2026-08-25T21:25:25.994Z","schemaVersion":2},"datasetVersion":"2026-08-26T02:17:13.382Z"}