{"record":{"id":"d7b2a5dbe604b23a","repo":"invoke-ai/InvokeAI","slug":"label-network-error-exc","errorCode":null,"errorMessage":"{label} network error: {exc}","messagePattern":"(.+?) network error: (.+?)","errorType":"exception","errorClass":"ExternalProviderRequestError","httpStatus":null,"severity":"error","filePath":"invokeai/app/services/external_generation/providers/alibabacloud.py","lineNumber":370,"sourceCode":"        json: dict | None = None,\n    ) -> requests.Response:\n        \"\"\"Issue a request with limited retries on transient failures (429/5xx, network errors).\n\n        Honors `Retry-After` for 429 responses when present. Non-retryable errors\n        (4xx other than 429, parse failures) are returned to the caller, which is\n        responsible for raising a meaningful ExternalProviderRequestError.\n        \"\"\"\n        last_exc: Exception | None = None\n        for attempt in range(_MAX_RETRIES + 1):\n            try:\n                if method == \"POST\":\n                    response = requests.post(url, headers=headers, json=json, timeout=timeout)\n                else:\n                    response = requests.get(url, headers=headers, timeout=timeout)\n            except requests.RequestException as exc:\n                last_exc = exc\n                if attempt >= _MAX_RETRIES:\n                    raise ExternalProviderRequestError(f\"{label} network error: {exc}\") from exc\n                delay = _RETRY_BACKOFF_BASE * (2**attempt)\n                self._logger.warning(\n                    \"%s network error on attempt %d/%d: %s — retrying in %.1fs\",\n                    label,\n                    attempt + 1,\n                    _MAX_RETRIES + 1,\n                    exc,\n                    delay,\n                )\n                time.sleep(delay)\n                continue\n\n            if response.status_code in _RETRY_STATUS_CODES and attempt < _MAX_RETRIES:\n                delay = self._retry_delay(response, attempt)\n                self._logger.warning(\n                    \"%s got status %d on attempt %d/%d — retrying in %.1fs\",\n                    label,\n                    response.status_code,","sourceCodeStart":352,"sourceCodeEnd":388,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/app/services/external_generation/providers/alibabacloud.py#L352-L388","documentation":"_request_with_retry wraps every DashScope HTTP call (sync POST, async submit POST, task poll GET). If requests raises a RequestException (connection refused, DNS failure, timeout, SSL error) on the final allowed attempt (attempt >= _MAX_RETRIES, i.e. 3 total tries), it raises ExternalProviderRequestError labeled '<label> network error' with the underlying exception.","triggerScenarios":"Connection reset/refused to dashscope-intl.aliyuncs.com; DNS resolution failure; request timeout (120s sync, 60s async submit, 30s poll); TLS handshake failure; repeated transient network errors across all 3 attempts with exponential backoff (2s, 4s).","commonSituations":"Egress firewall blocking the DashScope endpoint from the container; DNS misconfiguration in Kubernetes; DashScope regional outage; long generations exceeding the 120s request timeout; proxy environments with unstable connections.","solutions":["Read the wrapped exception (e.g. ConnectionError vs Timeout vs SSLError) to identify the network layer at fault","Verify outbound HTTPS access to your external_alibabacloud_base_url from the app host (curl the endpoint)","Check DNS configuration and proxy env vars (HTTP_PROXY/HTTPS_PROXY/NO_PROXY)","If timeouts dominate, increase the timeout passed to _post_with_retry or reduce generation size/complexity","Implement higher-level retry/backoff in the calling service for sustained outages"],"exampleFix":"// before\nresponse = self._post_with_retry(endpoint, headers=headers, json=payload, timeout=120, label=\"DashScope sync\")\n// after\ntry:\n    response = self._post_with_retry(endpoint, headers=headers, json=payload, timeout=120, label=\"DashScope sync\")\nexcept ExternalProviderRequestError as e:\n    logger.warning(\"DashScope unreachable, aborting generation: %s\", e)\n    raise","handlingStrategy":"retry","validationCode":"import requests\ndef check_connectivity(base_url: str) -> bool:\n    try:\n        requests.get(base_url, timeout=10)\n        return True\n    except requests.RequestException:\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    result = provider.generate(request)\nexcept ExternalProviderRequestError as e:\n    if \"network error\" in str(e):\n        log.warning(\"DashScope unreachable after 3 attempts: %s\", e)\n        # schedule the job for retry rather than failing permanently\n        requeue(request)\n    else:\n        raise","preventionTips":["Verify outbound HTTPS/DNS to the DashScope endpoint from the deployment environment","Set proxy env vars correctly in containers/Kubernetes","Size generation work so requests finish within the per-call timeouts","Add job-level retry with longer backoff for sustained outages","Monitor DashScope status pages during regional incidents"],"tags":["network","retry","timeout","http","dashscope"],"backgroundTag":"connection-refused","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}