keras-team/keras · error · ValueError

TF-IDF data must be a 1-index array. Received: type(idf_weig

Error message

TF-IDF data must be a 1-index array. Received: type(idf_weights)={type(idf_weights)}

What it means

IndexLookup.set_vocabulary() requires idf_weights to be a 1-D array when output_mode='tf_idf'. After converting the input to an ndarray, the layer checks ndim and rejects anything that is not a flat vector, because each weight must map to exactly one vocabulary entry.

Source

Thrown at keras/src/layers/preprocessing/index_lookup.py:567

        if self.max_tokens is not None and (new_vocab_size > self.max_tokens):
            raise ValueError(
                "Attempted to set a vocabulary larger than the maximum vocab "
                f"size. Received vocabulary size is {new_vocab_size}; "
                f"`max_tokens` is {self.max_tokens}."
            )
        self.lookup_table = self._lookup_table_from_tokens(tokens)
        self._record_vocabulary_size()

        if self.output_mode == "tf_idf" and idf_weights is not None:
            if len(vocabulary) != len(idf_weights):
                raise ValueError(
                    "`idf_weights` must be the same length as vocabulary. "
                    f"len(idf_weights) is {len(idf_weights)}; "
                    f"len(vocabulary) is {len(vocabulary)}"
                )
            idf_weights = self._convert_to_ndarray(idf_weights)
            if idf_weights.ndim != 1:
                raise ValueError(
                    "TF-IDF data must be a 1-index array. "
                    f"Received: type(idf_weights)={type(idf_weights)}"
                )

            # If the passed vocabulary has no special tokens, we need to pad the
            # front of idf_weights. We don't have real document frequencies for
            # these tokens so we will use an average of all idf_weights passed
            # in as a reasonable default.
            if found_special_tokens:
                front_padding = 0
                front_padding_value = 0
            else:
                front_padding = token_start
                front_padding_value = np.average(idf_weights)
            # If pad_to_max_tokens is true, and max_tokens is greater than our
            # total vocab size, we need to pad the back of idf_weights with
            # zeros as well.
            back_padding_value = 0

View on GitHub (pinned to 7a34a03db6)

Solutions

1. Reshape the weights to 1-D before passing: np.asarray(idf_weights).ravel() or .reshape(-1)
2. Check the shape first: assert np.ndim(idf_weights) == 1
3. If weights come from a file, load with np.loadtxt (returns 1-D) rather than ndmin=2

Example fix

// before
layer.set_vocabulary(vocab, idf_weights=w.reshape(-1, 1))
// after
import numpy as np
layer.set_vocabulary(vocab, idf_weights=np.asarray(w).ravel())

import numpy as np
w = np.asarray(idf_weights)
if w.ndim != 1:
    w = w.ravel()
assert len(w) == len(vocab)

def is_flat_weights(w):
    import numpy as np
    return np.ndim(w) == 1

try:
    layer.set_vocabulary(vocab, idf_weights=w)
except ValueError as e:
    if '1-index array' in str(e):
        layer.set_vocabulary(vocab, idf_weights=np.asarray(w).ravel())
    else:
        raise

When it happens

Trigger: Calling layer.set_vocabulary(vocab, idf_weights=weights) or constructing IndexLookup/TextVectorization(output_mode='tf_idf', vocabulary=..., idf_weights=...) with idf_weights shaped 2-D or 0-D, e.g. a column vector [[1.2],[0.7]], a matrix, or a scalar.

Common situations: Loading IDF weights from a saved model or CSV where reshape introduced an extra axis; passing weights reshaped via .reshape(-1,1); computing weights with sklearn TfidfVectorizer.idf_ then reshaping incorrectly.

Related errors

• `idf_weights` should only be set if `output_mode` is `'tf_id
• When specifying the `vocabulary` argument, in TF-IDF output
• `idf_weights` must be set if output_mode is 'tf_idf'.
• `idf_weights` should only be set if output_mode is `'tf_idf'
• output_mode `'tf_idf'` does not support loading a vocabulary

AI-assisted analysis of keras-team/keras@7a34a03db6 (2026-08-25). Data as JSON: /api/errors/7c24a90c615211f0. Report an issue: GitHub.