{"record":{"id":"754a7eb7b33b3f67","repo":"run-llama/llama_index","slug":"document-doc-node-id-is-too-large-token-count","errorCode":null,"errorMessage":"Document {doc.node_id} is too large ({token_count} tokens) to be processed. Doc metadata: {doc.metadata}","messagePattern":"Document (.+?) is too large \\((.+?) tokens\\) to be processed\\. Doc metadata: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/extractors/document_context.py","lineNumber":281,"sourceCode":"            return None\n        if not is_text_node(doc):\n            logging.warning(f\"Document {doc_id} is not an instance of (TextNode, Node)\")\n            return None\n\n        # then truncate if necessary.\n        if self.max_context_length is not None:\n            strategy = self.oversized_document_strategy\n            token_count = self._count_tokens(doc.get_content())\n            if token_count > self.max_context_length:\n                message = (\n                    f\"Document {doc.node_id} is too large ({token_count} tokens) \"\n                    f\"to be processed. Doc metadata: {doc.metadata}\"\n                )\n\n                if strategy == \"warn\":\n                    logging.warning(message)\n                elif strategy == \"error\":\n                    raise ValueError(message)\n                elif strategy == \"ignore\":\n                    pass\n                else:\n                    raise ValueError(f\"Unknown oversized document strategy: {strategy}\")\n\n        return doc\n\n    async def aextract(self, nodes: Sequence[BaseNode]) -> List[Dict]:\n        \"\"\"\n        Extract context for multiple nodes asynchronously, optimized for loosely ordered nodes.\n        Processes each node independently without guaranteeing sequential document handling.\n        Nodes will be *mostly* processed in document-order assuming nodes get passed in document-order.\n\n        Args:\n            nodes: List of nodes to process, ideally grouped by source document\n\n        Returns:\n            List of metadata dictionaries with generated context","sourceCodeStart":263,"sourceCodeEnd":299,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/extractors/document_context.py#L263-L299","documentation":"Raised by DocumentContextExtractor._process_document when a document's token count exceeds max_context_length and oversized_document_strategy='error'. The extractor prepends document-level context (title/summary) by prompting an LLM over the whole document, so documents larger than the model's context window cannot be processed; the same message is only logged when strategy='warn' and silently skipped when strategy='ignore'.","triggerScenarios":"Running a transformation pipeline containing DocumentContextExtractor(max_context_length=N, oversized_document_strategy='error') over a document whose get_content() tokenizes to more than N tokens via the extractor's _count_tokens.","commonSituations":"Ingesting long PDFs or concatenated transcripts into a pipeline that assumed smaller docs; setting max_context_length to the embedding model's limit while the extractor uses the LLM's limit; batch ingestion jobs where one oversized file aborts the whole run.","solutions":["Split the document before extraction (SentenceSplitter/SemanticSplitter with a chunk cap) so each unit fits under max_context_length.","Raise max_context_length to your LLM's real context window if the count was conservative.","Change oversized_document_strategy to 'warn' (log and skip) or 'ignore' to let oversized docs pass through unenhanced."],"exampleFix":"# before\nextractor = DocumentContextExtractor(\n    max_context_length=2048, oversized_document_strategy=\"error\"\n)\n\n# after\nfrom llama_index.core.node_parser import SentenceSplitter\nsplit_docs = SentenceSplitter(chunk_size=1024).get_nodes_from_documents(docs)\nextractor = DocumentContextExtractor(\n    max_context_length=128000, oversized_document_strategy=\"warn\"\n)","handlingStrategy":"validation","validationCode":"token_count = extractor._count_tokens(doc.get_content())\nif extractor.max_context_length and token_count > extractor.max_context_length:\n    logging.warning(\"Skipping oversized doc %s (%d tokens)\", doc.node_id, token_count)\n    continue  # or split the doc first","typeGuard":null,"tryCatchPattern":"try:\n    doc = await extractor.aprocess_document(doc)\nexcept ValueError as e:\n    if \"too large\" in str(e):\n        logging.warning(\"Oversized document skipped: %s\", e)\n    else:\n        raise","preventionTips":["Pre-split documents below max_context_length before ingestion.","Use oversized_document_strategy='warn' in batch jobs so one big doc cannot abort the run.","Track token counts at ingestion time and alert on outliers."],"tags":["validation","context-window","document-processing","token-limit"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}