Panniantong/Agent-Reach · error · ValueError

V2EX API response exceeds the 1 MiB safety limit

Error message

V2EX API response exceeds the 1 MiB safety limit

What it means

Raised by _get_json_with_urllib() when a V2EX API response body exceeds _MAX_RESPONSE_BYTES (1 MiB). The read is deliberately capped at max+1 bytes so oversized bodies are detected without buffering them; this bounds memory and blocks response-splitting style abuse from a hostile endpoint.

Source

Thrown at agent_reach/channels/v2ex.py:53

    if (
        parsed.scheme.lower() != "https"
        or (parsed.hostname or "").lower() not in {"v2ex.com", "www.v2ex.com"}
        or port not in {None, 443}
        or parsed.username is not None
        or parsed.password is not None
        or not parsed.path.startswith("/api/")
    ):
        raise ValueError("only the V2EX HTTPS API is allowed")


def _get_json_with_urllib(url: str) -> Any:
    """Fetch JSON with Python's standard HTTP stack."""
    _validate_api_url(url)
    req = urllib.request.Request(url, headers={"User-Agent": _UA})
    with urllib.request.urlopen(req, timeout=_TIMEOUT) as resp:
        raw = resp.read(_MAX_RESPONSE_BYTES + 1)
    if len(raw) > _MAX_RESPONSE_BYTES:
        raise ValueError("V2EX API response exceeds the 1 MiB safety limit")
    return json.loads(raw.decode("utf-8"))


def _is_unexpected_tls_eof(error: BaseException) -> bool:
    """Return whether an exception chain contains the retryable TLS EOF."""
    pending: list[BaseException] = [error]
    seen: set[int] = set()
    while pending:
        current = pending.pop()
        if id(current) in seen:
            continue
        seen.add(id(current))
        if isinstance(current, ssl.SSLError) and not isinstance(
            current, ssl.SSLCertVerificationError
        ):
            text = str(current).casefold()
            if (
                "unexpected_eof_while_reading" in text

View on GitHub (pinned to 93ae1d18c3)

Solutions

1. Request smaller scopes: use per-page/per-offset parameters V2EX supports (e.g. page=N on replies) instead of one giant call
2. Pick a narrower endpoint (topic-by-id rather than node-wide listings)
3. Retry — if it was an anomalous oversized payload (mitm/proxy injection), a clean retry may pass
4. Do not try to raise the cap from caller code; fork/patch is required, and 1 MiB is a safety invariant

Example fix

# before
 fetch("https://www.v2ex.com/api/replies.json?topic_id=1")  # 5000+ replies -> >1MiB
# after
 fetch("https://www.v2ex.com/api/replies.json?topic_id=1&page=1")

# Caller cannot know size before the request; best pre-check is to
# request scoped-down endpoints so responses stay small:
def build_paged_replies_url(topic_id: int, page: int = 1) -> str:
    from urllib.parse import urlencode
    return "https://www.v2ex.com/api/replies.json?" + urlencode({"topic_id": topic_id, "page": page})

try:
    data = _get_json_with_urllib(url)
except ValueError as exc:
    if "1 MiB" in str(exc):
        data = _get_json_with_urllib(url + "&page=1")  # narrow the scope and retry once
    else:
        raise

When it happens

Trigger: Any V2EX read/search whose response body crosses 1 MiB — e.g. /api/replies with a very long topic (thousands of replies), or a node listing returning a huge array. resp.read(_MAX_RESPONSE_BYTES + 1) returns 1 MiB + 1 bytes and the length check fires.

Common situations: Fetching replies for mega-threads; API changes increasing payload size; paging not applied so a single request returns everything.

Related errors

• invalid V2EX API URL
• only the V2EX HTTPS API is allowed
• curl is unavailable for the V2EX TLS fallback
• curl could not complete the V2EX TLS fallback
• Jina Reader response exceeds {_MAX_RESPONSE_BYTES} byte limi

AI-assisted analysis of Panniantong/Agent-Reach@93ae1d18c3 (2026-08-14). Data as JSON: /api/errors/0ef46c04032d4235. Report an issue: GitHub.