{"record":{"id":"2460ef1789c1e9aa","repo":"keras-team/keras","slug":"fit-the-tokenizer-on-some-data-before-using-tfidf","errorCode":null,"errorMessage":"Fit the Tokenizer on some data before using tfidf mode.","messagePattern":"Fit the Tokenizer on some data before using tfidf mode\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"keras/src/legacy/preprocessing/text.py","lineNumber":251,"sourceCode":"\n    def texts_to_matrix(self, texts, mode=\"binary\"):\n        sequences = self.texts_to_sequences(texts)\n        return self.sequences_to_matrix(sequences, mode=mode)\n\n    def sequences_to_matrix(self, sequences, mode=\"binary\"):\n        if not self.num_words:\n            if self.word_index:\n                num_words = len(self.word_index) + 1\n            else:\n                raise ValueError(\n                    \"Specify a dimension (`num_words` argument), \"\n                    \"or fit on some text data first.\"\n                )\n        else:\n            num_words = self.num_words\n\n        if mode == \"tfidf\" and not self.document_count:\n            raise ValueError(\n                \"Fit the Tokenizer on some data before using tfidf mode.\"\n            )\n\n        x = np.zeros((len(sequences), num_words))\n        for i, seq in enumerate(sequences):\n            if not seq:\n                continue\n            counts = collections.defaultdict(int)\n            for j in seq:\n                if j >= num_words:\n                    continue\n                counts[j] += 1\n            for j, c in list(counts.items()):\n                if mode == \"count\":\n                    x[i][j] = c\n                elif mode == \"freq\":\n                    x[i][j] = c / len(seq)\n                elif mode == \"binary\":","sourceCodeStart":233,"sourceCodeEnd":269,"githubUrl":"https://github.com/keras-team/keras/blob/7a34a03db60bf60042242d6a556fc3be119046a5/keras/src/legacy/preprocessing/text.py#L233-L269","documentation":"tfidf mode needs corpus-level document frequencies (index_docs and document_count) computed during fit_on_texts. Requesting mode='tfidf' while document_count is 0 raises this ValueError, even if num_words is set.","triggerScenarios":"Tokenizer(num_words=5000).texts_to_matrix(texts, mode='tfidf') on a never-fitted tokenizer.","commonSituations":"Assuming num_words alone suffices; inference service reconstructs a fresh tokenizer and calls tfidf vectorization without loading fit state.","solutions":["Call fit_on_texts on the training corpus before using mode='tfidf'","Save and reload the fitted tokenizer rather than rebuilding it","If document statistics are unavailable, use mode='count' or 'binary'"],"exampleFix":"# before\ntok = Tokenizer(num_words=5000)\nX = tok.texts_to_matrix(texts, mode='tfidf')\n# after\ntok = Tokenizer(num_words=5000)\ntok.fit_on_texts(train_texts)\nX = tok.texts_to_matrix(texts, mode='tfidf')","handlingStrategy":"validation","validationCode":"if mode == 'tfidf' and not tok.document_count:\n    tok.fit_on_texts(corpus)","typeGuard":"def can_tfidf(tok):\n    return tok.document_count > 0","tryCatchPattern":null,"preventionTips":["Test the tfidf path in CI with a fitted tokenizer"],"tags":["keras","text","tokenizer","tfidf"],"backgroundTag":"unfitted-transformer","analyzedSha":"7a34a03db60bf60042242d6a556fc3be119046a5","analyzedAt":"2026-08-25T21:25:25.994Z","schemaVersion":2},"datasetVersion":"2026-08-26T02:17:13.382Z"}