{"record":{"id":"72fc9a279c080134","repo":"Graphify-Labs/graphify","slug":"ingest-failed-to-fetch-url-r-exc","errorCode":null,"errorMessage":"ingest: failed to fetch {url!r}: {exc}","messagePattern":"ingest: failed to fetch (.+?): (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"graphify/ingest.py","lineNumber":257,"sourceCode":"            suffix = Path(urllib.parse.urlparse(url).path).suffix or \".jpg\"\n            out = _download_binary(url, suffix, target_dir)\n            print(f\"Downloaded image: {out.name}\")\n            return out\n\n        if url_type == \"youtube\":\n            from graphify.transcribe import download_audio\n            out = download_audio(url, target_dir)\n            print(f\"Downloaded audio: {out.name}\")\n            return out\n\n        if url_type == \"tweet\":\n            content, filename = _fetch_tweet(url, author, contributor)\n        elif url_type == \"arxiv\":\n            content, filename = _fetch_arxiv(url, author, contributor)\n        else:\n            content, filename = _fetch_webpage(url, author, contributor)\n    except (urllib.error.HTTPError, urllib.error.URLError, OSError) as exc:\n        raise RuntimeError(f\"ingest: failed to fetch {url!r}: {exc}\") from exc\n\n    out_path = target_dir / filename\n    # Avoid overwriting - append counter if needed\n    counter = 1\n    while out_path.exists() and counter < 1000:\n        stem = Path(filename).stem\n        out_path = target_dir / f\"{stem}_{counter}.md\"\n        counter += 1\n\n    out_path.write_text(content, encoding=\"utf-8\")\n    print(f\"Saved {url_type}: {out_path.name}\")\n    return out_path\n\nOUTCOMES = (\"useful\", \"dead_end\", \"corrected\")\n\n\ndef save_query_result(\n    question: str,","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/Graphify-Labs/graphify/blob/7fe58b0b0f3873be9a21c30106b8b8527c353aa6/graphify/ingest.py#L239-L275","documentation":"Raised by ingest when the fetch phase throws a network-level exception: urllib HTTPError (4xx/5xx), URLError (DNS failure, refused connection, TLS problem), or OSError (local I/O during download). Only the download/fetch block is guarded; validation failures have already been handled separately, so this error specifically means the request was attempted and the network or remote end failed.","triggerScenarios":"Calling ingest() where _fetch_webpage/_fetch_tweet/_fetch_arxiv or a binary download raises: 404/403 from the server, DNS not resolving, connection refused behind a proxy, TLS certificate errors, or disk OSError writing temp files.","commonSituations":"Dead or typo'd links; sites blocking the default user-agent; corporate proxies/SSL interception breaking urllib; offline runs; rate-limited endpoints returning 429.","solutions":["Check the embedded exception: HTTP 4xx/5xx means the URL is bad or blocked; URLError/OSError usually means proxy, DNS, or TLS","Test the URL directly: `curl -I <url>` from the same machine","Set proxy/SSL env vars if behind a corporate proxy (HTTPS_PROXY, REQUESTS_CA_bundle-style fixes)","Retry transient failures (5xx, timeouts) — the error does not write partial files"],"exampleFix":"# before\ningest(\"https://example.com/gone.pdf\", out_dir)\n# RuntimeError: ingest: failed to fetch 'https://example.com/gone.pdf': HTTP Error 404\n\n# after\ningest(\"https://example.com/paper-v2.pdf\", out_dir)","handlingStrategy":"retry","validationCode":"from urllib.parse import urlparse\nimport socket\n\nhost = urlparse(url).hostname or \"\"\ntry:\n    socket.getaddrinfo(host, 443)\nexcept socket.gaierror:\n    raise SystemExit(f\"host {host!r} does not resolve - check the URL or DNS\")","typeGuard":null,"tryCatchPattern":"import time\nfor attempt in range(3):\n    try:\n        ingest(url, target_dir)\n        break\n    except RuntimeError as e:\n        if \"failed to fetch\" not in str(e):\n            raise\n        if attempt == 2 or \"HTTP Error 4\" in str(e):\n            raise  # client errors are permanent - stop retrying\n        time.sleep(2 ** attempt)","preventionTips":["Retry only transient classes (5xx, URLError, timeouts) with backoff; never retry 4xx","Set proxy/TLS env vars in corporate networks before batch ingests","URL-check inputs (curl preflight) in bulk ingestion jobs to prune dead links early"],"tags":["network","http","ingest","url"],"backgroundTag":null,"analyzedSha":"7fe58b0b0f3873be9a21c30106b8b8527c353aa6","analyzedAt":"2026-08-14T19:23:21.323Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}