{"record":{"id":"5f593cf32b0e535e","repo":"chroma-core/chroma","slug":"document-length-len-doc-tokens-ids-is-greater-t","errorCode":null,"errorMessage":"Document length {len(doc_tokens.ids)} is greater than the max tokens {self.max_tokens()}","messagePattern":"Document length (.+?) is greater than the max tokens (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"chromadb/utils/embedding_functions/onnx_mini_lm_l6_v2.py","lineNumber":166,"sourceCode":"\n        Args:\n            documents: The documents to generate embeddings for.\n            batch_size: The batch size to use when generating embeddings.\n\n        Returns:\n            The embeddings for the documents.\n        \"\"\"\n        all_embeddings = []\n        for i in range(0, len(documents), batch_size):\n            batch = documents[i : i + batch_size]\n\n            # Encode each document separately\n            encoded = [self.tokenizer.encode(d) for d in batch]\n\n            # Check if any document exceeds the max tokens\n            for doc_tokens in encoded:\n                if len(doc_tokens.ids) > self.max_tokens():\n                    raise ValueError(\n                        f\"Document length {len(doc_tokens.ids)} is greater than the max tokens {self.max_tokens()}\"\n                    )\n\n            input_ids = np.array([e.ids for e in encoded])\n            attention_mask = np.array([e.attention_mask for e in encoded])\n\n            onnx_input = {\n                \"input_ids\": np.array(input_ids, dtype=np.int64),\n                \"attention_mask\": np.array(attention_mask, dtype=np.int64),\n                \"token_type_ids\": np.array(\n                    [np.zeros(len(e), dtype=np.int64) for e in input_ids],\n                    dtype=np.int64,\n                ),\n            }\n\n            model_output = self.model.run(None, onnx_input)\n            last_hidden_state = model_output[0]\n","sourceCodeStart":148,"sourceCodeEnd":184,"githubUrl":"https://github.com/chroma-core/chroma/blob/aecdd12c8a891610db8653630b066b32ceb678b5/chromadb/utils/embedding_functions/onnx_mini_lm_l6_v2.py#L148-L184","documentation":"Before running the ONNX model, __call__ encodes each batch with the MiniLM tokenizer and raises ValueError when len(doc_tokens.ids) > self.max_tokens(), which is hard-coded to 256 for this model. This is a strict limit — unlike some transformers, there is no truncation=True here — because the model's positional embeddings only cover 256 tokens, so longer inputs would either crash the session or produce garbage embeddings.","triggerScenarios":"collection.add(documents=[long_text]) where the HuggingFace tokenizer yields more than 256 tokens (note: word-piece tokens, not words, so ~190+ English words can exceed it); ingesting full web pages, articles, PDFs, transcripts, or code files without chunking; a batch where just one document exceeds the limit fails the whole call.","commonSituations":"RAG pipelines ingesting raw documents without a text splitter; importing data from another EF that tolerated longer inputs (e.g. 512-token or 8192-token models); non-English text where tokenization inflates token counts (CJK, agglutinative languages); code/JSON blobs that tokenize densely.","solutions":["Chunk documents before add(): split into overlapping ~200-word / ~800-character pieces (e.g. LangChain RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)) and add each as its own document","Or truncate defensively: doc[:8000] characters is a rough guard, but tokenizer-based chunking is the correct fix since the limit is in tokens","If you must embed long documents whole, switch to an EF with a larger context window (e.g. OpenAI text-embedding-3, Ollama nomic-embed-text with 8192, or a 512-token sentence-transformers model)"],"exampleFix":"// before\ncollection.add(ids=[\"doc1\"], documents=[open(\"report.txt\").read()])  # >256 tokens -> ValueError\n\n// after\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nchunks = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100).split_text(open(\"report.txt\").read())\ncollection.add(ids=[f\"doc1_{i}\" for i in range(len(chunks))], documents=chunks)","handlingStrategy":"validation","validationCode":"MAX_CHARS = 800  # conservative: ~200 words < 256 word-piece tokens\ndef fit_chunks(text: str, chunk_size: int = 800, overlap: int = 100):\n    if len(text) <= chunk_size:\n        return [text]\n    return [text[i:i + chunk_size] for i in range(0, len(text), chunk_size - overlap)]\ndocs = [c for d in raw_docs for c in fit_chunks(d)]","typeGuard":null,"tryCatchPattern":"try:\n    collection.add(ids=ids, documents=docs)\nexcept ValueError as e:\n    if \"max tokens\" in str(e):\n        docs = [c for d in docs for c in fit_chunks(d)]  # split and retry once\n        collection.add(ids=new_ids, documents=docs)\n    else:\n        raise","preventionTips":["Always run a text splitter (chunk_size ~800 chars with overlap) before add() with this EF","Remember the 256 limit is in word-piece tokens, not words — non-English text inflates faster","Store chunk_index in metadata so you can reconstruct long documents from pieces","For long-document workloads choose an EF with a larger context window"],"tags":["onnx","embedding-function","token-limit","truncation","chunking","chroma"],"backgroundTag":"input-exceeds-token-limit","analyzedSha":"aecdd12c8a891610db8653630b066b32ceb678b5","analyzedAt":"2026-08-16T21:53:27.228Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}