{"record":{"id":"af5ed32e017ffeff","repo":"zylon-ai/private-gpt","slug":"unable-to-split-oversized-document-subtree","errorCode":null,"errorMessage":"Unable to split oversized document subtree","messagePattern":"Unable to split oversized document subtree","errorType":"error_code","errorClass":"ContentRequestLimitError","httpStatus":413,"severity":"error","filePath":"private_gpt/server/content/content_service.py","lineNumber":57,"sourceCode":"    tokenizer_fn: TokenizerFn | None,\n) -> list[BaseNode]:\n    if max_length is None or tokenizer_fn is None:\n        return [subtree]\n\n    content = subtree.get_content(TreeMetadataMode.LLM)\n    if len(tokenizer_fn(content)) <= max_length:\n        return [subtree]\n\n    splitter_class = cast(Any, TokenTextSplitterWithoutStripping)\n    splitter = splitter_class(\n        chunk_size=max_length,\n        chunk_overlap=0,\n        tokenizer=tokenizer_fn,\n        keep_whitespaces=True,\n    )\n    chunks = splitter.split_text(content)\n    if not chunks:\n        raise ContentRequestLimitError(\"Unable to split oversized document subtree\")\n\n    split_nodes = [\n        TextNode(\n            text=chunk,\n            extra_info=dict(subtree.metadata),\n            abs_idx=subtree.abs_idx,\n            idx=subtree.idx,\n        )\n        for chunk in chunks\n        if chunk\n    ]\n    if any(len(tokenizer_fn(node.text)) > max_length for node in split_nodes):\n        raise ContentRequestLimitError(\n            \"Document subtree could not be split within the requested token limit\"\n        )\n    return cast(list[BaseNode], split_nodes)\n\n","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/server/content/content_service.py#L39-L75","documentation":"ContentRequestLimitError raised when a document subtree whose content exceeds max_length tokens is passed to TokenTextSplitterWithoutStripping but split_text() returns zero chunks. Normally the splitter always produces at least one chunk for non-empty text, so an empty result implies empty/whitespace-only content after stripping-free splitting or splitter misconfiguration.","triggerScenarios":"An oversized subtree whose extracted content is empty or becomes empty after the splitter's handling, so chunks == [] and the guard fires.","commonSituations":"Ingesting documents with empty text nodes (binary/OCR-less PDFs, placeholder nodes) that nonetheless trip the length check; edge-case splitter behavior with keep_whitespaces=True and pathological input (e.g. a single unbreakable token larger than chunk_size in some configurations).","solutions":["Inspect the failing subtree's extracted text; skip nodes with empty/blank content before calling the splitter.","If content is one token longer than max_length and unbreakable, raise max_length (chunk_size) or pre-split the token.","Upgrade/patch if a splitter version regression returns empty lists for non-empty input."],"exampleFix":"# before\nchunks = splitter.split_text(content)\n\n# after\nchunks = splitter.split_text(content) if content and content.strip() else []\nif not chunks:\n    return []  # skip empty subtree instead of raising","handlingStrategy":"validation","validationCode":"content = subtree_text(subtree)\nif not content or not content.strip():\n    skip_subtree = True  # never reach the splitter","typeGuard":null,"tryCatchPattern":"try:\n    nodes = split_oversized_subtree(subtree, tokenizer_fn, max_length)\nexcept ContentRequestLimitError:\n    log.warning('skipping unsplittable subtree %s', subtree.id_)\n    return  # skip and continue ingestion","preventionTips":["Skip empty/blank text nodes during ingestion","Test the splitter path with edge-case content (empty, single giant token)","Log the subtree id when this fires to identify source documents"],"tags":["ingestion","chunking","tokens","content"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}