Panniantong/Agent-Reach · error · ValueError

Jina Reader response exceeds {_MAX_RESPONSE_BYTES} byte limi

Error message

Jina Reader response exceeds {_MAX_RESPONSE_BYTES} byte limit

What it means

WebChannel.read (agent_reach/channels/web.py:59) wraps any URL with Jina Reader (https://r.jina.ai/...) and enforces _MAX_RESPONSE_BYTES = 5 MiB. If the Markdown that Jina returns exceeds 5 MiB, this ValueError is raised and the body is discarded. It protects the caller from unbounded memory use on huge pages.

Source

Thrown at agent_reach/channels/web.py:59

        return True  # Fallback — handles any URL

    def check(self, config=None):
        # 恒可用兜底渠道：无本地命令、不做网络探测（doctor 已有多个渠道触网），保持零开销
        self.active_backend = self.backends[0]
        return "ok", "通过 Jina Reader 读取任意网页（curl https://r.jina.ai/URL）"

    def read(self, url: str) -> str:
        """通过 Jina Reader 读取网页，返回 Markdown 全文。"""
        url = normalize_public_http_url(url)
        jina_url = f"https://r.jina.ai/{url}"
        req = urllib.request.Request(
            jina_url,
            headers={"User-Agent": _UA, "Accept": "text/plain"},
        )
        with urllib.request.urlopen(req, timeout=30) as resp:
            body = resp.read(_MAX_RESPONSE_BYTES + 1)
        if len(body) > _MAX_RESPONSE_BYTES:
            raise ValueError(
                f"Jina Reader response exceeds {_MAX_RESPONSE_BYTES} byte limit"
            )
        if _is_antibot_page(body):
            raise RuntimeError(
                "Jina Reader 返回了反爬验证页，未获取到目标内容；"
                "请改用站点专用工具或浏览器读取"
            )
        return body.decode("utf-8")

View on GitHub (pinned to 93ae1d18c3)

Solutions

Read a more specific URL (anchor/section page, paginated view) instead of the giant single page
Use a site-specific channel if one exists (youtube, reddit, etc.) which returns structured, smaller data
Catch the ValueError and degrade gracefully, e.g. report only the URL with a 'page too large' note
If you control the target site, publish a paginated or JSON variant of the content

Example fix

# before
from agent_reach.channels.web import WebChannel
md = WebChannel().read("https://example.com/huge-single-page-docs")

# after: fall back to the page URL when the body is too large
from agent_reach.channels.web import WebChannel

try:
    md = WebChannel().read("https://example.com/huge-single-page-docs")
except ValueError:
    md = f"(page exceeds Jina Reader 5 MiB limit, open directly: https://example.com/huge-single-page-docs)"

Defensive patterns

Strategy: try-catch

Try / catch

from agent_reach.channels.web import WebChannel

try:
    markdown = WebChannel().read(url)
except ValueError as exc:
    if "byte limit" in str(exc):
        markdown = None  # caller falls back to paginated/section URL
    else:
        raise

Prevention

Point agents at section/paginated URLs rather than single-page mega-docs
Check a site's sitemap for smaller pages when reads repeatedly blow the 5 MiB cap
Treat this ValueError as a signal about the target page, not about Jina being broken

When it happens

Trigger: web_channel.read(url) on extremely large pages: massive documentation single-page sites, giant forum threads, or data-dump pages whose rendered Markdown exceeds 5,242,880 bytes.

Common situations: Agents pointed at full-build docs sites (e.g. single-page API references), archived mailing-list mirrors, or machine-generated reports; also when Jina returns the page plus navigation boilerplate repeated thousands of times.

Related errors

AI-assisted analysis of Panniantong/Agent-Reach@93ae1d18c3 (2026-08-14). Data as JSON: /api/errors/0039bfd405d0b0cd. Report an issue: GitHub.