run-llama/llama_index · error · ValueError

Document {doc.node_id} is too large ({token_count} tokens) t

Error message

Document {doc.node_id} is too large ({token_count} tokens) to be processed. Doc metadata: {doc.metadata}

What it means

Raised by DocumentContextExtractor._process_document when a document's token count exceeds max_context_length and oversized_document_strategy='error'. The extractor prepends document-level context (title/summary) by prompting an LLM over the whole document, so documents larger than the model's context window cannot be processed; the same message is only logged when strategy='warn' and silently skipped when strategy='ignore'.

Source

Thrown at llama-index-core/llama_index/core/extractors/document_context.py:281

            return None
        if not is_text_node(doc):
            logging.warning(f"Document {doc_id} is not an instance of (TextNode, Node)")
            return None

        # then truncate if necessary.
        if self.max_context_length is not None:
            strategy = self.oversized_document_strategy
            token_count = self._count_tokens(doc.get_content())
            if token_count > self.max_context_length:
                message = (
                    f"Document {doc.node_id} is too large ({token_count} tokens) "
                    f"to be processed. Doc metadata: {doc.metadata}"
                )

                if strategy == "warn":
                    logging.warning(message)
                elif strategy == "error":
                    raise ValueError(message)
                elif strategy == "ignore":
                    pass
                else:
                    raise ValueError(f"Unknown oversized document strategy: {strategy}")

        return doc

    async def aextract(self, nodes: Sequence[BaseNode]) -> List[Dict]:
        """
        Extract context for multiple nodes asynchronously, optimized for loosely ordered nodes.
        Processes each node independently without guaranteeing sequential document handling.
        Nodes will be *mostly* processed in document-order assuming nodes get passed in document-order.

        Args:
            nodes: List of nodes to process, ideally grouped by source document

        Returns:
            List of metadata dictionaries with generated context

View on GitHub (pinned to afd0fef371)

Solutions

1. Split the document before extraction (SentenceSplitter/SemanticSplitter with a chunk cap) so each unit fits under max_context_length.
2. Raise max_context_length to your LLM's real context window if the count was conservative.
3. Change oversized_document_strategy to 'warn' (log and skip) or 'ignore' to let oversized docs pass through unenhanced.

Example fix

# before
extractor = DocumentContextExtractor(
    max_context_length=2048, oversized_document_strategy="error"
)

# after
from llama_index.core.node_parser import SentenceSplitter
split_docs = SentenceSplitter(chunk_size=1024).get_nodes_from_documents(docs)
extractor = DocumentContextExtractor(
    max_context_length=128000, oversized_document_strategy="warn"
)

token_count = extractor._count_tokens(doc.get_content())
if extractor.max_context_length and token_count > extractor.max_context_length:
    logging.warning("Skipping oversized doc %s (%d tokens)", doc.node_id, token_count)
    continue  # or split the doc first

try:
    doc = await extractor.aprocess_document(doc)
except ValueError as e:
    if "too large" in str(e):
        logging.warning("Oversized document skipped: %s", e)
    else:
        raise

When it happens

Trigger: Running a transformation pipeline containing DocumentContextExtractor(max_context_length=N, oversized_document_strategy='error') over a document whose get_content() tokenizes to more than N tokens via the extractor's _count_tokens.

Common situations: Ingesting long PDFs or concatenated transcripts into a pipeline that assumed smaller docs; setting max_context_length to the embedding model's limit while the extractor uses the LLM's limit; batch ingestion jobs where one oversized file aborts the whole run.

Related errors

• Unknown oversized document strategy: {strategy}
• Initial token count exceeds token limit
• Max iterations of {max_iterations} reached! Either something
• All agents must have a name in a multi-agent workflow
• All agents must have a description in a multi-agent workflow

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/754a7eb7b33b3f67. Report an issue: GitHub.