{"record":{"id":"0ef46c04032d4235","repo":"Panniantong/Agent-Reach","slug":"v2ex-api-response-exceeds-the-1-mib-safety-limit","errorCode":null,"errorMessage":"V2EX API response exceeds the 1 MiB safety limit","messagePattern":"V2EX API response exceeds the 1 MiB safety limit","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"agent_reach/channels/v2ex.py","lineNumber":53,"sourceCode":"    if (\n        parsed.scheme.lower() != \"https\"\n        or (parsed.hostname or \"\").lower() not in {\"v2ex.com\", \"www.v2ex.com\"}\n        or port not in {None, 443}\n        or parsed.username is not None\n        or parsed.password is not None\n        or not parsed.path.startswith(\"/api/\")\n    ):\n        raise ValueError(\"only the V2EX HTTPS API is allowed\")\n\n\ndef _get_json_with_urllib(url: str) -> Any:\n    \"\"\"Fetch JSON with Python's standard HTTP stack.\"\"\"\n    _validate_api_url(url)\n    req = urllib.request.Request(url, headers={\"User-Agent\": _UA})\n    with urllib.request.urlopen(req, timeout=_TIMEOUT) as resp:\n        raw = resp.read(_MAX_RESPONSE_BYTES + 1)\n    if len(raw) > _MAX_RESPONSE_BYTES:\n        raise ValueError(\"V2EX API response exceeds the 1 MiB safety limit\")\n    return json.loads(raw.decode(\"utf-8\"))\n\n\ndef _is_unexpected_tls_eof(error: BaseException) -> bool:\n    \"\"\"Return whether an exception chain contains the retryable TLS EOF.\"\"\"\n    pending: list[BaseException] = [error]\n    seen: set[int] = set()\n    while pending:\n        current = pending.pop()\n        if id(current) in seen:\n            continue\n        seen.add(id(current))\n        if isinstance(current, ssl.SSLError) and not isinstance(\n            current, ssl.SSLCertVerificationError\n        ):\n            text = str(current).casefold()\n            if (\n                \"unexpected_eof_while_reading\" in text","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/Panniantong/Agent-Reach/blob/93ae1d18c37b707dec053c7c4f9d91cd8ef8943d/agent_reach/channels/v2ex.py#L35-L71","documentation":"Raised by _get_json_with_urllib() when a V2EX API response body exceeds _MAX_RESPONSE_BYTES (1 MiB). The read is deliberately capped at max+1 bytes so oversized bodies are detected without buffering them; this bounds memory and blocks response-splitting style abuse from a hostile endpoint.","triggerScenarios":"Any V2EX read/search whose response body crosses 1 MiB — e.g. /api/replies with a very long topic (thousands of replies), or a node listing returning a huge array. resp.read(_MAX_RESPONSE_BYTES + 1) returns 1 MiB + 1 bytes and the length check fires.","commonSituations":"Fetching replies for mega-threads; API changes increasing payload size; paging not applied so a single request returns everything.","solutions":["Request smaller scopes: use per-page/per-offset parameters V2EX supports (e.g. page=N on replies) instead of one giant call","Pick a narrower endpoint (topic-by-id rather than node-wide listings)","Retry — if it was an anomalous oversized payload (mitm/proxy injection), a clean retry may pass","Do not try to raise the cap from caller code; fork/patch is required, and 1 MiB is a safety invariant"],"exampleFix":"# before\n fetch(\"https://www.v2ex.com/api/replies.json?topic_id=1\")  # 5000+ replies -> >1MiB\n# after\n fetch(\"https://www.v2ex.com/api/replies.json?topic_id=1&page=1\")","handlingStrategy":"retry","validationCode":"# Caller cannot know size before the request; best pre-check is to\n# request scoped-down endpoints so responses stay small:\ndef build_paged_replies_url(topic_id: int, page: int = 1) -> str:\n    from urllib.parse import urlencode\n    return \"https://www.v2ex.com/api/replies.json?\" + urlencode({\"topic_id\": topic_id, \"page\": page})","typeGuard":null,"tryCatchPattern":"try:\n    data = _get_json_with_urllib(url)\nexcept ValueError as exc:\n    if \"1 MiB\" in str(exc):\n        data = _get_json_with_urllib(url + \"&page=1\")  # narrow the scope and retry once\n    else:\n        raise","preventionTips":["Always page large collections (replies, node topics) instead of fetching whole threads","Prefer by-id endpoints over broad listings","Treat the 1 MiB cap as a fixed invariant; design call patterns around it"],"tags":["v2ex","response-size","network","limits"],"backgroundTag":null,"analyzedSha":"93ae1d18c37b707dec053c7c4f9d91cd8ef8943d","analyzedAt":"2026-08-14T22:54:06.735Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}