{"record":{"id":"83aa777eb48cdede","repo":"stamparm/maltrail","slug":"s-s-s","errorCode":null,"errorMessage":"%s (%s): %s","messagePattern":"%s \\(%s\\): %s","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sensor/tools/check_redistribution.py","lineNumber":103,"sourceCode":"# Piles whose whole purpose is to name shared infrastructure. update_trails() spares them and so\n# does this: reporting them would be reporting the intent.\nEXEMPT_INFO = (\"parking\", \"sinkhole\", \"cdn\", \"mass scanner\")\n\n\ndef fetch(url):\n    request = Request(url, headers={\"User-agent\": \"maltrail-redistribution-check\"})\n    return urlopen(request, timeout=TIMEOUT).read().decode(\"utf8\", \"replace\")\n\n\ndef networks():\n    \"\"\"{provider: [ip_network, ...]} from the providers' own published lists.\"\"\"\n\n    retval = {}\n    for provider, url, kind in SOURCES:\n        try:\n            body = fetch(url)\n        except Exception as ex:\n            raise RuntimeError(\"%s (%s): %s\" % (provider, url, ex))\n\n        prefixes = []\n        if kind == \"lines\":\n            prefixes = [_.strip() for _ in body.splitlines() if _.strip()]\n        elif kind == \"aws\":\n            data = json.loads(body)\n            prefixes = [_[\"ip_prefix\"] for _ in data.get(\"prefixes\", [])\n                        if _.get(\"service\") in AWS_SHARED_SERVICES]\n            prefixes += [_[\"ipv6_prefix\"] for _ in data.get(\"ipv6_prefixes\", [])\n                         if _.get(\"service\") in AWS_SHARED_SERVICES]\n        elif kind == \"gcp\":\n            data = json.loads(body)\n            for entry in data.get(\"prefixes\", []):\n                prefixes.append(entry.get(\"ipv4Prefix\") or entry.get(\"ipv6Prefix\"))\n        elif kind == \"fastly\":\n            data = json.loads(body)\n            prefixes = list(data.get(\"addresses\", [])) + list(data.get(\"ipv6_addresses\", []))\n","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/stamparm/maltrail/blob/77cfb06d7606506d101bbcec0786c77166c4255e/sensor/tools/check_redistribution.py#L85-L121","documentation":"In check_redistribution.py, main() iterates SOURCES (provider, url, kind) and calls fetch(url) for each redistribution-list provider. Any exception raised while fetching (network failure, DNS error, HTTP error, TLS problem) is caught and re-raised as a RuntimeError formatted '%s (%s): %s' — provider name, URL, and the underlying exception — so the caller knows exactly which source could not be retrieved and why.","triggerScenarios":"fetch(url) raises any Exception for one of the SOURCES entries: the URL is unreachable, DNS resolution fails, the server returns a non-200 status that fetch turns into an exception, the connection times out, or TLS handshake fails.","commonSituations":"Running the check in a CI container or air-gapped network without internet access; a provider changing or deprecating its URL; a corporate proxy or firewall blocking the host; transient upstream outage of a blocklist provider (e.g. a CDN or government feed being temporarily down).","solutions":["Read the inner exception in the message to identify the cause (connection refused, timeout, HTTP status) and fix network access or proxy settings.","Verify the provider URL in SOURCES is still valid — open it in a browser or curl it, and update it if the provider moved its feed.","Re-run later if the failure is a transient upstream outage; consider adding retry logic around fetch.","In restricted environments, run from a machine with egress access or pre-download the lists and adapt fetch to read from local files."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import urllib.request\ndef reachable(url):\n    try:\n        with urllib.request.urlopen(url, timeout=10) as r:\n            return r.status == 200\n    except Exception:\n        return False\n\nfor provider, url, kind in SOURCES:\n    if not reachable(url):\n        print(f\"skipping {provider}: {url} unreachable\")","typeGuard":null,"tryCatchPattern":"try:\n    body = fetch(url)\nexcept RuntimeError as ex:\n    provider, url, cause = ex.args[0] if False else str(ex).split(' (', 1)\n    log.warning(\"source fetch failed, will retry/fallback: %s\", ex)\n    body = cached_or_skip(provider)","preventionTips":["Pre-check provider URLs with a lightweight HEAD request before running the full redistribution check.","Cache successful fetches so a single transient outage doesn't fail the whole run.","Set explicit timeouts in fetch and add bounded retries with backoff for flaky providers.","Monitor provider feed URLs (they move or get deprecated) and keep the SOURCES list up to date.","Run network-dependent checks from environments with confirmed egress; skip or mock in air-gapped CI."],"tags":["network","http","fetch","third-party-data"],"backgroundTag":"http-request-failed","analyzedSha":"77cfb06d7606506d101bbcec0786c77166c4255e","analyzedAt":"2026-09-13T03:50:16.010Z","contentChangedAt":"2026-09-13T03:50:16.010Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}