{"record":{"id":"ec5f218937464954","repo":"Graphify-Labs/graphify","slug":"ingest-exc","errorCode":null,"errorMessage":"ingest: {exc}","messagePattern":"ingest: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"graphify/ingest.py","lineNumber":230,"sourceCode":"    filename = _safe_filename(url, suffix)\n    out_path = target_dir / filename\n    out_path.write_bytes(safe_fetch(url))\n    return out_path\n\n\ndef ingest(url: str, target_dir: Path, author: str | None = None, contributor: str | None = None) -> Path:\n    \"\"\"\n    Fetch a URL and save it into target_dir as a graphify-ready file.\n\n    Returns the path of the saved file.\n    \"\"\"\n    target_dir.mkdir(parents=True, exist_ok=True)\n    url_type = _detect_url_type(url)\n\n    try:\n        validate_url(url)\n    except ValueError as exc:\n        raise ValueError(f\"ingest: {exc}\") from exc\n\n    try:\n        if url_type == \"pdf\":\n            out = _download_binary(url, \".pdf\", target_dir)\n            print(f\"Downloaded PDF: {out.name}\")\n            return out\n\n        if url_type == \"image\":\n            suffix = Path(urllib.parse.urlparse(url).path).suffix or \".jpg\"\n            out = _download_binary(url, suffix, target_dir)\n            print(f\"Downloaded image: {out.name}\")\n            return out\n\n        if url_type == \"youtube\":\n            from graphify.transcribe import download_audio\n            out = download_audio(url, target_dir)\n            print(f\"Downloaded audio: {out.name}\")\n            return out","sourceCodeStart":212,"sourceCodeEnd":248,"githubUrl":"https://github.com/Graphify-Labs/graphify/blob/7fe58b0b0f3873be9a21c30106b8b8527c353aa6/graphify/ingest.py#L212-L248","documentation":"Raised by graphify's URL ingest when validate_url(url) rejects the input before any network activity. The ingest pipeline classifies the URL (pdf/image/audio/tweet/arxiv/webpage) and first runs strict validation; a ValueError from that check is re-raised prefixed with 'ingest:' and chained to the original, preserving the specific reason (bad scheme, malformed host, SSRF-guard rejection, etc.).","triggerScenarios":"Calling ingest(url, target_dir) — or `graphify ingest <url>` — with a URL validate_url refuses: missing scheme, non-http(s) protocol, unparsable host, or an address the SSRF protections block.","commonSituations":"Pasting bare domains ('example.com/doc.pdf') without https://; passing file:// or other disallowed schemes; trailing punctuation from copy-paste; URLs blocked because they resolve to private/loopback ranges.","solutions":["Read the tail of the message — the original ValueError names the exact validation failure","Normalize the URL: add the https:// scheme, strip stray whitespace/quotes","If the target is legitimately internal, check validate_url's SSRG/allowlist options before relaxing anything — the block is a security guard"],"exampleFix":"# before\ningest(\"example.com/paper.pdf\", out_dir)   # ValueError: ingest: missing scheme\n\n# after\ningest(\"https://example.com/paper.pdf\", out_dir)","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\nu = urlparse(url)\nif u.scheme not in (\"http\", \"https\") or not u.hostname:\n    raise SystemExit(f\"refusing URL {url!r}: needs an http(s) scheme and host\")\n\nfrom graphify.ingest import ingest\ningest(url, target_dir)","typeGuard":null,"tryCatchPattern":"try:\n    ingest(url, target_dir)\nexcept ValueError as e:\n    if str(e).startswith(\"ingest:\"):\n        raise SystemExit(f\"bad URL {url!r}: {e}\")  # fix the input, don't retry\n    raise","preventionTips":["Normalize URLs (add scheme, strip whitespace) before passing to ingest","Never disable the SSRF validation to 'make it work' - rewrite the input instead","Fail fast on ValueError in batch ingesters: validation errors never self-heal"],"tags":["validation","url","ingest","security"],"backgroundTag":null,"analyzedSha":"7fe58b0b0f3873be9a21c30106b8b8527c353aa6","analyzedAt":"2026-08-14T19:23:21.323Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}