{"record":{"id":"3b9c9874c82bed20","repo":"ComposioHQ/composio","slug":"refusing-to-fetch-a-malformed-or-non-http-s-url","errorCode":null,"errorMessage":"Refusing to fetch a malformed or non-http(s) URL","messagePattern":"Refusing to fetch a malformed or non-http\\(s\\) URL","errorType":"exception","errorClass":"BlockedInternalUrlError","httpStatus":null,"severity":"error","filePath":"python/composio/utils/url_safety.py","lineNumber":92,"sourceCode":"def assert_safe_fetch_target(url: str) -> t.List[str]:\n    \"\"\"Refuse non-HTTP(S) URLs and hosts that resolve to internal addresses.\n\n    Parse the URL after Requests prepares it so validation uses the same\n    canonical hostname that the eventual connection will use.\n\n    :returns: The validated addresses to connect to, in resolver order.\n        Callers must connect to *these* rather than re-resolving the hostname;\n        see :func:`safe_get`.\n    \"\"\"\n    try:\n        prepared_url = requests.Request(method=\"GET\", url=url).prepare().url\n        if prepared_url is None:\n            raise ValueError(\"Prepared URL is missing\")\n        parsed = urlparse(prepared_url)\n        if parsed.scheme not in {\"http\", \"https\"} or not parsed.hostname:\n            raise ValueError(\"URL must use HTTP(S) and include a hostname\")\n    except (requests.exceptions.RequestException, ValueError):\n        raise BlockedInternalUrlError(\n            \"Refusing to fetch a malformed or non-http(s) URL\"\n        ) from None\n\n    try:\n        # Resolver order is kept: it encodes the system's address preference\n        # (RFC 6724), and connecting walks it the way urllib3 would.\n        addresses: t.List[str] = []\n        for result in socket.getaddrinfo(parsed.hostname, None):\n            address = result[4][0]\n            if not isinstance(address, str):\n                raise BlockedInternalUrlError(\n                    f'Could not resolve host \"{parsed.hostname}\"'\n                )\n            if address not in addresses:\n                addresses.append(address)\n    except socket.gaierror as error:\n        raise BlockedInternalUrlError(\n            f'Could not resolve host \"{parsed.hostname}\"'","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/ComposioHQ/composio/blob/64b1b85502b1beeb2379e6c9e8bf1104504fa637/python/composio/utils/url_safety.py#L74-L110","documentation":"assert_safe_fetch_target prepares the URL with requests and requires an http/https scheme plus a hostname; if URL preparation/parsing fails or those checks fail, BlockedInternalUrlError is raised wrapping the underlying error. This is SSRF protection — the SDK refuses to fetch malformed or non-HTTP URLs (ftp://, file://, bare hostnames without parseable hostname, etc.).","triggerScenarios":"Passing a URL like \"ftp://example.com/file\", \"file:///etc/passwd\", \"example.com/path\" (no scheme, so hostname parsing fails), or a malformed URL that requests' preparation rejects (e.g. invalid characters, bad percent-encoding). The check runs before any DNS resolution in _pinned_request.","commonSituations":"User- or LLM-supplied URLs not normalized (missing https:// prefix); configs containing internal schemes; copy-pasted URLs with stray whitespace/quotes that break parsing; data sources mixing URNs/paths with URLs.","solutions":["Normalize URLs before fetching: strip whitespace/quotes, prepend https:// when the scheme is missing, and reject non-http(s) schemes early","Validate with urlparse yourself: scheme in {http, https} and a non-empty hostname","If the target is genuinely http(s) but still blocked, print the exact string being passed — hidden characters (zero-width, \\r) are a common cause","Whitelist expected hosts in your own layer and construct URLs from trusted templates rather than raw input"],"exampleFix":"# before\nfetch(\"example.com/data.json\")\n# after\nfrom urllib.parse import urlparse\nurl = \"example.com/data.json\"\nif not urlparse(url).scheme:\n    url = \"https://\" + url\nassert urlparse(url).scheme in {\"http\", \"https\"} and urlparse(url).hostname\nfetch(url)","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\ndef normalize_and_check(url):\n    url = url.strip().strip('\"\\'')\n    if not urlparse(url).scheme:\n        url = \"https://\" + url\n    p = urlparse(url)\n    if p.scheme not in {\"http\", \"https\"} or not p.hostname:\n        raise ValueError(f\"unsafe/malformed URL: {url!r}\")\n    return url","typeGuard":"def is_fetchable_url(u: str) -> bool:\n    try:\n        p = urlparse(u.strip())\n        return p.scheme in {\"http\", \"https\"} and bool(p.hostname)\n    except ValueError:\n        return False","tryCatchPattern":"from composio.utils.url_safety import BlockedInternalUrlError\ntry:\n    fetch(url)\nexcept BlockedInternalUrlError:\n    url = normalize_and_check(url)\n    fetch(url)","preventionTips":["Normalize user/LLM-supplied URLs (strip, add scheme) before fetching","Allowlist expected hostnames in your own layer","Construct URLs from trusted templates; reject non-http schemes early"],"tags":["ssrf","url-validation","network-fetch","security"],"backgroundTag":"blocked-unsafe-url-fetch","analyzedSha":"64b1b85502b1beeb2379e6c9e8bf1104504fa637","analyzedAt":"2026-08-28T15:39:33.623Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}