{"record":{"id":"0039bfd405d0b0cd","repo":"Panniantong/Agent-Reach","slug":"jina-reader-response-exceeds-max-response-bytes","errorCode":null,"errorMessage":"Jina Reader response exceeds {_MAX_RESPONSE_BYTES} byte limit","messagePattern":"Jina Reader response exceeds (.+?) byte limit","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"agent_reach/channels/web.py","lineNumber":59,"sourceCode":"        return True  # Fallback — handles any URL\n\n    def check(self, config=None):\n        # 恒可用兜底渠道：无本地命令、不做网络探测（doctor 已有多个渠道触网），保持零开销\n        self.active_backend = self.backends[0]\n        return \"ok\", \"通过 Jina Reader 读取任意网页（curl https://r.jina.ai/URL）\"\n\n    def read(self, url: str) -> str:\n        \"\"\"通过 Jina Reader 读取网页，返回 Markdown 全文。\"\"\"\n        url = normalize_public_http_url(url)\n        jina_url = f\"https://r.jina.ai/{url}\"\n        req = urllib.request.Request(\n            jina_url,\n            headers={\"User-Agent\": _UA, \"Accept\": \"text/plain\"},\n        )\n        with urllib.request.urlopen(req, timeout=30) as resp:\n            body = resp.read(_MAX_RESPONSE_BYTES + 1)\n        if len(body) > _MAX_RESPONSE_BYTES:\n            raise ValueError(\n                f\"Jina Reader response exceeds {_MAX_RESPONSE_BYTES} byte limit\"\n            )\n        if _is_antibot_page(body):\n            raise RuntimeError(\n                \"Jina Reader 返回了反爬验证页，未获取到目标内容；\"\n                \"请改用站点专用工具或浏览器读取\"\n            )\n        return body.decode(\"utf-8\")\n","sourceCodeStart":41,"sourceCodeEnd":68,"githubUrl":"https://github.com/Panniantong/Agent-Reach/blob/93ae1d18c37b707dec053c7c4f9d91cd8ef8943d/agent_reach/channels/web.py#L41-L68","documentation":"WebChannel.read (agent_reach/channels/web.py:59) wraps any URL with Jina Reader (https://r.jina.ai/...) and enforces _MAX_RESPONSE_BYTES = 5 MiB. If the Markdown that Jina returns exceeds 5 MiB, this ValueError is raised and the body is discarded. It protects the caller from unbounded memory use on huge pages.","triggerScenarios":"web_channel.read(url) on extremely large pages: massive documentation single-page sites, giant forum threads, or data-dump pages whose rendered Markdown exceeds 5,242,880 bytes.","commonSituations":"Agents pointed at full-build docs sites (e.g. single-page API references), archived mailing-list mirrors, or machine-generated reports; also when Jina returns the page plus navigation boilerplate repeated thousands of times.","solutions":["Read a more specific URL (anchor/section page, paginated view) instead of the giant single page","Use a site-specific channel if one exists (youtube, reddit, etc.) which returns structured, smaller data","Catch the ValueError and degrade gracefully, e.g. report only the URL with a 'page too large' note","If you control the target site, publish a paginated or JSON variant of the content"],"exampleFix":"# before\nfrom agent_reach.channels.web import WebChannel\nmd = WebChannel().read(\"https://example.com/huge-single-page-docs\")\n\n# after: fall back to the page URL when the body is too large\nfrom agent_reach.channels.web import WebChannel\n\ntry:\n    md = WebChannel().read(\"https://example.com/huge-single-page-docs\")\nexcept ValueError:\n    md = f\"(page exceeds Jina Reader 5 MiB limit, open directly: https://example.com/huge-single-page-docs)\"","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from agent_reach.channels.web import WebChannel\n\ntry:\n    markdown = WebChannel().read(url)\nexcept ValueError as exc:\n    if \"byte limit\" in str(exc):\n        markdown = None  # caller falls back to paginated/section URL\n    else:\n        raise","preventionTips":["Point agents at section/paginated URLs rather than single-page mega-docs","Check a site's sitemap for smaller pages when reads repeatedly blow the 5 MiB cap","Treat this ValueError as a signal about the target page, not about Jina being broken"],"tags":["web","jina-reader","response-size","safety-limit"],"backgroundTag":null,"analyzedSha":"93ae1d18c37b707dec053c7c4f9d91cd8ef8943d","analyzedAt":"2026-08-14T22:54:06.735Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}