iflytek/astron-agent · error · RuntimeError

fetch_all_document_chunks exceeded max_pages=

Error message

fetch_all_document_chunks exceeded max_pages={max_pages} for doc={document_id}; server may be mis-reporting total

What it means

Raised when fetch_all_document_chunks exhausts max_pages without reaching the reported total, indicating the server keeps claiming more chunks than the page cap allows. It is a safety valve against infinite pagination loops.

Solutions

Increase max_pages (or page_size) in get_document_chunks to cover the document's actual chunk count.
Verify the real chunk count against the server (list chunks once and check total in the response).
Upgrade RAGFlow if total is mis-reported in that version.
Catch the error for oversized documents and handle them with an explicit chunked-export strategy.

Example fix

# before
doc = await get_document_chunks(ds, doc_id)  # default max_pages too low
# after
chunks = await get_document_chunks(ds, doc_id, page_size=100, max_pages=200)

Defensive patterns

Strategy: validation

Validate before calling

MAX_CHUNKS = max_pages * page_size
info = await get_document_info(dataset_id, document_id)
if info and info.get("chunk_count", 0) > MAX_CHUNKS:
    raise RuntimeError(f"Document needs {info['chunk_count']} chunks > cap {MAX_CHUNKS}; raise max_pages/page_size")

Try / catch

try:
    chunks = await fetch_all_document_chunks(dataset_id, document_id)
except RuntimeError as e:
    if "exceeded max_pages" in str(e):
        logger.error("Pagination cap exhausted — increase max_pages or investigate total mis-report")
    raise

Prevention

Size max_pages*page_size above the largest expected document's chunk count
Spot-check the server-reported total against reality after RAGFlow upgrades
Treat repeated hits as a signal of server-side total mis-reporting

When it happens

Trigger: Server continuously returning non-empty pages with total never reached: mis-reported/incorrect total field, extremely large document exceeding max_pages * page_size, or a server bug causing duplicated page data.

Common situations: Very large documents parsed into more chunks than max_pages*page_size allows with current settings; RAGFlow version with a total-count bug; documents being continuously re-chunked during retrieval.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of iflytek/astron-agent@5e758547a8 (2026-09-12). Data as JSON: /api/errors/04b53352ac051594. Report an issue: GitHub.

Appendix: source

Thrown at core/knowledge/infra/ragflow/ragflow_client.py:739

        batch = data.get("chunks") or []
        chunks.extend(batch)
        # Missing/None/non-int => keep last-known good value.
        raw_total = data.get("total")
        if isinstance(raw_total, int) and raw_total >= 0:
            total = raw_total
        if total is not None and len(chunks) >= total:
            return chunks
        if not batch:
            # Protocol anomaly (stale pagination, mid-request deletion, or
            # server mis-report): fail closed to avoid re-inserting the
            # missing chunks as if they didn't exist.
            raise RuntimeError(
                f"fetch_all_document_chunks: empty page {page} but only "
                f"{len(chunks)}/{total if total is not None else '?'} "
                f"chunks fetched for doc={document_id}"
            )
        page += 1
    raise RuntimeError(
        f"fetch_all_document_chunks exceeded max_pages={max_pages} for "
        f"doc={document_id}; server may be mis-reporting total"
    )


async def get_document_info(dataset_id: str, doc_id: str) -> Optional[Dict[str, Any]]:
    """
    Get detailed information for a single document via RAGFlow's id filter.

    Uses the ``id`` query parameter on ``/api/v1/datasets/{dataset_id}/documents``,
    which performs exact-match filtering server-side (verified against RAGFlow
    v0.20.5 ~ v0.24.0: ``DocumentService.get_list`` applies
    ``.where(cls.model.id == id)`` — peewee equality, not ``LIKE``).

    Return contract:

    - ``code == 0`` with matching doc: return the doc dict.
    - ``code == 0`` with no matching doc: return ``None``.

View on GitHub (pinned to 5e758547a8)