{"record":{"id":"e8457a596139ee7b","repo":"iflytek/astron-agent","slug":"fetch-all-document-chunks-empty-page-page-but-only-len","errorCode":null,"errorMessage":"fetch_all_document_chunks: empty page {page} but only {len(chunks)}/{total if total is not None else '?'} chunks fetched for doc={document_id}","messagePattern":"fetch_all_document_chunks: empty page (.+?) but only (.+?)/(.+?) chunks fetched for doc=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"core/knowledge/infra/ragflow/ragflow_client.py","lineNumber":733,"sourceCode":"            raise RuntimeError(\n                f\"fetch_all_document_chunks failed on page {page} for \"\n                f\"doc={document_id}: code={resp.get('code')}, \"\n                f\"message={resp.get('message')}\"\n            )\n        data = resp.get(\"data\") or {}\n        batch = data.get(\"chunks\") or []\n        chunks.extend(batch)\n        # Missing/None/non-int => keep last-known good value.\n        raw_total = data.get(\"total\")\n        if isinstance(raw_total, int) and raw_total >= 0:\n            total = raw_total\n        if total is not None and len(chunks) >= total:\n            return chunks\n        if not batch:\n            # Protocol anomaly (stale pagination, mid-request deletion, or\n            # server mis-report): fail closed to avoid re-inserting the\n            # missing chunks as if they didn't exist.\n            raise RuntimeError(\n                f\"fetch_all_document_chunks: empty page {page} but only \"\n                f\"{len(chunks)}/{total if total is not None else '?'} \"\n                f\"chunks fetched for doc={document_id}\"\n            )\n        page += 1\n    raise RuntimeError(\n        f\"fetch_all_document_chunks exceeded max_pages={max_pages} for \"\n        f\"doc={document_id}; server may be mis-reporting total\"\n    )\n\n\nasync def get_document_info(dataset_id: str, doc_id: str) -> Optional[Dict[str, Any]]:\n    \"\"\"\n    Get detailed information for a single document via RAGFlow's id filter.\n\n    Uses the ``id`` query parameter on ``/api/v1/datasets/{dataset_id}/documents``,\n    which performs exact-match filtering server-side (verified against RAGFlow\n    v0.20.5 ~ v0.24.0: ``DocumentService.get_list`` applies","sourceCodeStart":715,"sourceCodeEnd":751,"githubUrl":"https://github.com/iflytek/astron-agent/blob/5e758547a83371a5a4b29dadf4ac03e8dd527635/core/knowledge/infra/ragflow/ragflow_client.py#L715-L751","documentation":"fetch_all_document_chunks fails closed when the server returns an empty chunk page while the accumulated chunk count is still below the reported total. The code treats this protocol anomaly as fatal rather than returning a silently incomplete chunk set.","triggerScenarios":"Server reports total=N but returns an empty page before all N chunks are collected: stale pagination metadata, document/chunks deleted mid-request, or server mis-reporting totals after concurrent re-chunking.","commonSituations":"Document re-parsed (chunks deleted/recreated) while another request was paginating through it; RAGFlow bugs or version quirks in the total field; heavy concurrency on the same document.","solutions":["Re-run the fetch; if it succeeds, the original failure was a concurrent re-chunk/delete race.","Ensure no other pipeline is re-parsing the document during chunk retrieval (serialize operations per document).","Check the RAGFlow server version for known pagination/total bugs and upgrade if affected.","If unavoidable, catch this error and treat the document fetch as failed rather than re-inserting partial chunks."],"exampleFix":"// before: partial chunks silently re-inserted\ntry { chunks = await fetchAllDocumentChunks(ds, doc); } catch { chunks = []; }\n// after: propagate — do not persist a partial set\ntry {\n  chunks = await fetchAllDocumentChunks(ds, doc);\n} catch (e) {\n  logger.error(\"chunk fetch incomplete, aborting upsert\", e);\n  throw e;\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    chunks = await fetch_all_document_chunks(dataset_id, document_id)\nexcept RuntimeError as e:\n    if \"empty page\" in str(e):\n        logger.error(\"Chunk fetch incomplete (concurrent re-parse or server bug); refusing partial upsert\")\n    raise  # never fall back to an empty/partial chunk list","preventionTips":["Never replace failed chunk fetches with empty lists before re-upsert","Serialize re-parsing and chunk reads per document","Add alerting on this error — it signals a real consistency race"],"tags":["ragflow","pagination","consistency"],"backgroundTag":"unexpected-response-shape","analyzedSha":"5e758547a83371a5a4b29dadf4ac03e8dd527635","analyzedAt":"2026-09-12T08:03:51.356Z","contentChangedAt":"2026-09-12T08:03:51.356Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}