{"record":{"id":"04b53352ac051594","repo":"iflytek/astron-agent","slug":"fetch-all-document-chunks-exceeded-max-pages-max-pages-for","errorCode":null,"errorMessage":"fetch_all_document_chunks exceeded max_pages={max_pages} for doc={document_id}; server may be mis-reporting total","messagePattern":"fetch_all_document_chunks exceeded max_pages=(.+?) for doc=(.+?); server may be mis-reporting total","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"core/knowledge/infra/ragflow/ragflow_client.py","lineNumber":739,"sourceCode":"        batch = data.get(\"chunks\") or []\n        chunks.extend(batch)\n        # Missing/None/non-int => keep last-known good value.\n        raw_total = data.get(\"total\")\n        if isinstance(raw_total, int) and raw_total >= 0:\n            total = raw_total\n        if total is not None and len(chunks) >= total:\n            return chunks\n        if not batch:\n            # Protocol anomaly (stale pagination, mid-request deletion, or\n            # server mis-report): fail closed to avoid re-inserting the\n            # missing chunks as if they didn't exist.\n            raise RuntimeError(\n                f\"fetch_all_document_chunks: empty page {page} but only \"\n                f\"{len(chunks)}/{total if total is not None else '?'} \"\n                f\"chunks fetched for doc={document_id}\"\n            )\n        page += 1\n    raise RuntimeError(\n        f\"fetch_all_document_chunks exceeded max_pages={max_pages} for \"\n        f\"doc={document_id}; server may be mis-reporting total\"\n    )\n\n\nasync def get_document_info(dataset_id: str, doc_id: str) -> Optional[Dict[str, Any]]:\n    \"\"\"\n    Get detailed information for a single document via RAGFlow's id filter.\n\n    Uses the ``id`` query parameter on ``/api/v1/datasets/{dataset_id}/documents``,\n    which performs exact-match filtering server-side (verified against RAGFlow\n    v0.20.5 ~ v0.24.0: ``DocumentService.get_list`` applies\n    ``.where(cls.model.id == id)`` — peewee equality, not ``LIKE``).\n\n    Return contract:\n\n    - ``code == 0`` with matching doc: return the doc dict.\n    - ``code == 0`` with no matching doc: return ``None``.","sourceCodeStart":721,"sourceCodeEnd":757,"githubUrl":"https://github.com/iflytek/astron-agent/blob/5e758547a83371a5a4b29dadf4ac03e8dd527635/core/knowledge/infra/ragflow/ragflow_client.py#L721-L757","documentation":"Raised when fetch_all_document_chunks exhausts max_pages without reaching the reported total, indicating the server keeps claiming more chunks than the page cap allows. It is a safety valve against infinite pagination loops.","triggerScenarios":"Server continuously returning non-empty pages with total never reached: mis-reported/incorrect total field, extremely large document exceeding max_pages * page_size, or a server bug causing duplicated page data.","commonSituations":"Very large documents parsed into more chunks than max_pages*page_size allows with current settings; RAGFlow version with a total-count bug; documents being continuously re-chunked during retrieval.","solutions":["Increase max_pages (or page_size) in get_document_chunks to cover the document's actual chunk count.","Verify the real chunk count against the server (list chunks once and check total in the response).","Upgrade RAGFlow if total is mis-reported in that version.","Catch the error for oversized documents and handle them with an explicit chunked-export strategy."],"exampleFix":"# before\ndoc = await get_document_chunks(ds, doc_id)  # default max_pages too low\n# after\nchunks = await get_document_chunks(ds, doc_id, page_size=100, max_pages=200)","handlingStrategy":"validation","validationCode":"MAX_CHUNKS = max_pages * page_size\ninfo = await get_document_info(dataset_id, document_id)\nif info and info.get(\"chunk_count\", 0) > MAX_CHUNKS:\n    raise RuntimeError(f\"Document needs {info['chunk_count']} chunks > cap {MAX_CHUNKS}; raise max_pages/page_size\")","typeGuard":null,"tryCatchPattern":"try:\n    chunks = await fetch_all_document_chunks(dataset_id, document_id)\nexcept RuntimeError as e:\n    if \"exceeded max_pages\" in str(e):\n        logger.error(\"Pagination cap exhausted — increase max_pages or investigate total mis-report\")\n    raise","preventionTips":["Size max_pages*page_size above the largest expected document's chunk count","Spot-check the server-reported total against reality after RAGFlow upgrades","Treat repeated hits as a signal of server-side total mis-reporting"],"tags":["ragflow","pagination","limits"],"backgroundTag":"value-out-of-range","analyzedSha":"5e758547a83371a5a4b29dadf4ac03e8dd527635","analyzedAt":"2026-09-12T08:03:51.356Z","contentChangedAt":"2026-09-12T08:03:51.356Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}