zylon-ai/private-gpt · error · ContentRequestLimitError

Document subtree could not be split within the requested tok

Error message

Document subtree could not be split within the requested token limit

What it means

ContentRequestLimitError raised as a post-condition check: after splitting, at least one produced TextNode still tokenizes above max_length. With chunk_overlap=0 the splitter should respect chunk_size, so this fires when individual tokens exceed chunk_size or the tokenizer used for validation differs from the splitter's tokenizer.

Source

Thrown at private_gpt/server/content/content_service.py:70

        tokenizer=tokenizer_fn,
        keep_whitespaces=True,
    )
    chunks = splitter.split_text(content)
    if not chunks:
        raise ContentRequestLimitError("Unable to split oversized document subtree")

    split_nodes = [
        TextNode(
            text=chunk,
            extra_info=dict(subtree.metadata),
            abs_idx=subtree.abs_idx,
            idx=subtree.idx,
        )
        for chunk in chunks
        if chunk
    ]
    if any(len(tokenizer_fn(node.text)) > max_length for node in split_nodes):
        raise ContentRequestLimitError(
            "Document subtree could not be split within the requested token limit"
        )
    return cast(list[BaseNode], split_nodes)


@singleton
class ContentService:
    @inject
    def __init__(
        self,
        settings: Settings,
        llm_component: LLMComponent,
        vector_store_component: VectorStoreComponent,
        embedding_component: EmbeddingComponent,
        node_store_component: NodeStoreComponent,
        ingest_component: IngestComponent,
        parse_component: ParseComponent,
    ) -> None:

View on GitHub (pinned to 4a030776a3)

Solutions

Increase max_length above the largest single token the documents contain.
Pre-process content to break up long unbroken strings (URLs, hashes) before splitting.
Ensure the same tokenizer_fn is passed to both the splitter and the length validation.
Skip or specially handle nodes that cannot be split (store raw, flag for manual processing).

Example fix

# before
splitter = splitter_class(chunk_size=max_length, chunk_overlap=0, tokenizer=tokenizer_fn, keep_whitespaces=True)

# after
if any(len(tokenizer_fn(t)) > max_length for t in content.split()):
    content = ' '.join(t[:max_length] for t in content.split())  # break unbreakable tokens
splitter = splitter_class(chunk_size=max_length, chunk_overlap=0, tokenizer=tokenizer_fn, keep_whitespaces=True)

Defensive patterns

Strategy: validation

Validate before calling

if any(len(tokenizer_fn(tok)) > max_length for tok in content.split()):
    content = break_long_tokens(content, max_length, tokenizer_fn)

Try / catch

try:
    nodes = split_oversized_subtree(subtree, tokenizer_fn, max_length)
except ContentRequestLimitError:
    # raise effective max_length for this node or skip with a warning

Prevention

Pre-split unbreakable tokens (URLs, hashes) before chunking
Use the same tokenizer for splitting and validation
Set max_length above the largest realistic single token

When it happens

Trigger: A single token (long URL, hash, CJK run, DNA-style string) longer than max_length tokens under the counting tokenizer; splitter.tokenizer and the validating tokenizer_fn disagreeing on counts.

Common situations: Small max_length settings (e.g. embedding-model chunk limits) with documents containing unbroken strings; tokenizer mismatch between component configuration and the splitter call.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/906ed8f401e7d613. Report an issue: GitHub.