Graphify-Labs/graphify · error · RuntimeError

ingest: failed to fetch {url!r}: {exc}

Error message

ingest: failed to fetch {url!r}: {exc}

What it means

Raised by ingest when the fetch phase throws a network-level exception: urllib HTTPError (4xx/5xx), URLError (DNS failure, refused connection, TLS problem), or OSError (local I/O during download). Only the download/fetch block is guarded; validation failures have already been handled separately, so this error specifically means the request was attempted and the network or remote end failed.

Source

Thrown at graphify/ingest.py:257

            suffix = Path(urllib.parse.urlparse(url).path).suffix or ".jpg"
            out = _download_binary(url, suffix, target_dir)
            print(f"Downloaded image: {out.name}")
            return out

        if url_type == "youtube":
            from graphify.transcribe import download_audio
            out = download_audio(url, target_dir)
            print(f"Downloaded audio: {out.name}")
            return out

        if url_type == "tweet":
            content, filename = _fetch_tweet(url, author, contributor)
        elif url_type == "arxiv":
            content, filename = _fetch_arxiv(url, author, contributor)
        else:
            content, filename = _fetch_webpage(url, author, contributor)
    except (urllib.error.HTTPError, urllib.error.URLError, OSError) as exc:
        raise RuntimeError(f"ingest: failed to fetch {url!r}: {exc}") from exc

    out_path = target_dir / filename
    # Avoid overwriting - append counter if needed
    counter = 1
    while out_path.exists() and counter < 1000:
        stem = Path(filename).stem
        out_path = target_dir / f"{stem}_{counter}.md"
        counter += 1

    out_path.write_text(content, encoding="utf-8")
    print(f"Saved {url_type}: {out_path.name}")
    return out_path

OUTCOMES = ("useful", "dead_end", "corrected")


def save_query_result(
    question: str,

View on GitHub (pinned to 7fe58b0b0f)

Solutions

1. Check the embedded exception: HTTP 4xx/5xx means the URL is bad or blocked; URLError/OSError usually means proxy, DNS, or TLS
2. Test the URL directly: `curl -I <url>` from the same machine
3. Set proxy/SSL env vars if behind a corporate proxy (HTTPS_PROXY, REQUESTS_CA_bundle-style fixes)
4. Retry transient failures (5xx, timeouts) — the error does not write partial files

Example fix

# before
ingest("https://example.com/gone.pdf", out_dir)
# RuntimeError: ingest: failed to fetch 'https://example.com/gone.pdf': HTTP Error 404

# after
ingest("https://example.com/paper-v2.pdf", out_dir)

from urllib.parse import urlparse
import socket

host = urlparse(url).hostname or ""
try:
    socket.getaddrinfo(host, 443)
except socket.gaierror:
    raise SystemExit(f"host {host!r} does not resolve - check the URL or DNS")

import time
for attempt in range(3):
    try:
        ingest(url, target_dir)
        break
    except RuntimeError as e:
        if "failed to fetch" not in str(e):
            raise
        if attempt == 2 or "HTTP Error 4" in str(e):
            raise  # client errors are permanent - stop retrying
        time.sleep(2 ** attempt)

When it happens

Trigger: Calling ingest() where _fetch_webpage/_fetch_tweet/_fetch_arxiv or a binary download raises: 404/403 from the server, DNS not resolving, connection refused behind a proxy, TLS certificate errors, or disk OSError writing temp files.

Common situations: Dead or typo'd links; sites blocking the default user-agent; corporate proxies/SSL interception breaking urllib; offline runs; rate-limited endpoints returning 429.

Related errors

• ingest: {exc}
• DNS resolution failed for '{hostname}': {exc}. Got: {url!r}
• {self.status_code} Error
• Proxy error: {exc}
• could not read Google Workspace shortcut {path}: {exc}

AI-assisted analysis of Graphify-Labs/graphify@7fe58b0b0f (2026-08-14). Data as JSON: /api/errors/72fc9a279c080134. Report an issue: GitHub.