{"record":{"id":"906ed8f401e7d613","repo":"zylon-ai/private-gpt","slug":"document-subtree-could-not-be-split-within-the-req","errorCode":null,"errorMessage":"Document subtree could not be split within the requested token limit","messagePattern":"Document subtree could not be split within the requested token limit","errorType":"error_code","errorClass":"ContentRequestLimitError","httpStatus":413,"severity":"error","filePath":"private_gpt/server/content/content_service.py","lineNumber":70,"sourceCode":"        tokenizer=tokenizer_fn,\n        keep_whitespaces=True,\n    )\n    chunks = splitter.split_text(content)\n    if not chunks:\n        raise ContentRequestLimitError(\"Unable to split oversized document subtree\")\n\n    split_nodes = [\n        TextNode(\n            text=chunk,\n            extra_info=dict(subtree.metadata),\n            abs_idx=subtree.abs_idx,\n            idx=subtree.idx,\n        )\n        for chunk in chunks\n        if chunk\n    ]\n    if any(len(tokenizer_fn(node.text)) > max_length for node in split_nodes):\n        raise ContentRequestLimitError(\n            \"Document subtree could not be split within the requested token limit\"\n        )\n    return cast(list[BaseNode], split_nodes)\n\n\n@singleton\nclass ContentService:\n    @inject\n    def __init__(\n        self,\n        settings: Settings,\n        llm_component: LLMComponent,\n        vector_store_component: VectorStoreComponent,\n        embedding_component: EmbeddingComponent,\n        node_store_component: NodeStoreComponent,\n        ingest_component: IngestComponent,\n        parse_component: ParseComponent,\n    ) -> None:","sourceCodeStart":52,"sourceCodeEnd":88,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/server/content/content_service.py#L52-L88","documentation":"ContentRequestLimitError raised as a post-condition check: after splitting, at least one produced TextNode still tokenizes above max_length. With chunk_overlap=0 the splitter should respect chunk_size, so this fires when individual tokens exceed chunk_size or the tokenizer used for validation differs from the splitter's tokenizer.","triggerScenarios":"A single token (long URL, hash, CJK run, DNA-style string) longer than max_length tokens under the counting tokenizer; splitter.tokenizer and the validating tokenizer_fn disagreeing on counts.","commonSituations":"Small max_length settings (e.g. embedding-model chunk limits) with documents containing unbroken strings; tokenizer mismatch between component configuration and the splitter call.","solutions":["Increase max_length above the largest single token the documents contain.","Pre-process content to break up long unbroken strings (URLs, hashes) before splitting.","Ensure the same tokenizer_fn is passed to both the splitter and the length validation.","Skip or specially handle nodes that cannot be split (store raw, flag for manual processing)."],"exampleFix":"# before\nsplitter = splitter_class(chunk_size=max_length, chunk_overlap=0, tokenizer=tokenizer_fn, keep_whitespaces=True)\n\n# after\nif any(len(tokenizer_fn(t)) > max_length for t in content.split()):\n    content = ' '.join(t[:max_length] for t in content.split())  # break unbreakable tokens\nsplitter = splitter_class(chunk_size=max_length, chunk_overlap=0, tokenizer=tokenizer_fn, keep_whitespaces=True)","handlingStrategy":"validation","validationCode":"if any(len(tokenizer_fn(tok)) > max_length for tok in content.split()):\n    content = break_long_tokens(content, max_length, tokenizer_fn)","typeGuard":null,"tryCatchPattern":"try:\n    nodes = split_oversized_subtree(subtree, tokenizer_fn, max_length)\nexcept ContentRequestLimitError:\n    # raise effective max_length for this node or skip with a warning","preventionTips":["Pre-split unbreakable tokens (URLs, hashes) before chunking","Use the same tokenizer for splitting and validation","Set max_length above the largest realistic single token"],"tags":["ingestion","chunking","tokens","validation"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}