{"record":{"id":"fee1600b04a7a072","repo":"invoke-ai/InvokeAI","slug":"label-failed-after-retries-last-exc","errorCode":null,"errorMessage":"{label} failed after retries: {last_exc}","messagePattern":"(.+?) failed after retries: (.+?)","errorType":"exception","errorClass":"ExternalProviderRequestError","httpStatus":null,"severity":"error","filePath":"invokeai/app/services/external_generation/providers/alibabacloud.py","lineNumber":400,"sourceCode":"\n            if response.status_code in _RETRY_STATUS_CODES and attempt < _MAX_RETRIES:\n                delay = self._retry_delay(response, attempt)\n                self._logger.warning(\n                    \"%s got status %d on attempt %d/%d — retrying in %.1fs\",\n                    label,\n                    response.status_code,\n                    attempt + 1,\n                    _MAX_RETRIES + 1,\n                    delay,\n                )\n                time.sleep(delay)\n                continue\n\n            return response\n\n        # Unreachable: the loop either returns a response or raises.\n        assert last_exc is not None\n        raise ExternalProviderRequestError(f\"{label} failed after retries: {last_exc}\") from last_exc\n\n    @staticmethod\n    def _retry_delay(response: requests.Response, attempt: int) -> float:\n        retry_after = response.headers.get(\"Retry-After\")\n        if retry_after:\n            try:\n                return max(0.0, float(retry_after))\n            except ValueError:\n                pass\n        return _RETRY_BACKOFF_BASE * (2**attempt)\n","sourceCodeStart":382,"sourceCodeEnd":411,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/app/services/external_generation/providers/alibabacloud.py#L382-L411","documentation":"ExternalProviderRequestError raised by the AlibabaCloud provider's _request_with_retry helper after every retry attempt has been exhausted without receiving an HTTP response it could return. The provider retries on transient failures (429/5xx, network errors) with backoff, and when even the final attempt fails, last_exc is re-raised wrapped in this message. It means the AlibabaCloud endpoint was persistently unreachable or kept returning retryable errors for the whole retry window.","triggerScenarios":"All retry attempts to the AlibabaCloud external-generation endpoint fail: connection errors, timeouts, or the server keeps returning 429/5xx status codes past the final attempt in _request_with_retry (called via _post_with_retry / _get_with_retry).","commonSituations":"AlibabaCloud regional outage or degradation; wrong external_alibabacloud_base_url pointing at an unreachable host; aggressive rate limiting from many queued jobs; firewall/proxy blocking egress; transient DNS failure during a burst of generations.","solutions":["Check whether the AlibabaCloud status page / region reports an outage and retry the generation later.","Verify external_alibabacloud_base_url (and credentials) in InvokeAI config point to the correct reachable endpoint.","Inspect the wrapped last_exc message to identify the underlying cause (timeout vs HTTP status) and fix accordingly.","Reduce request concurrency or add queueing if you are being rate-limited (429) on every attempt.","Increase retry count/backoff in _retry_delay/_request_with_retry if transient network flakiness is expected."],"exampleFix":"// before: no pre-check, request fails only after exhausting retries\nresult = provider.generate(request)\n// after: probe reachability first and fail fast with a clearer message\nif not provider.is_configured():\n    raise RuntimeError(\"AlibabaCloud provider is not configured\")\ntry:\n    result = provider.generate(request)\nexcept ExternalProviderRequestError as e:\n    logger.warning(\"AlibabaCloud unreachable after retries: %s\", e)\n    result = fallback_provider.generate(request)","handlingStrategy":"retry","validationCode":"import requests\ntry:\n    requests.get(base_url, timeout=5)\nexcept requests.RequestException as e:\n    raise RuntimeError(f\"AlibabaCloud endpoint unreachable before generation: {e}\")","typeGuard":null,"tryCatchPattern":"try:\n    result = provider.generate(request)\nexcept ExternalProviderRequestError as e:\n    logger.warning(\"AlibabaCloud failed after retries: %s\", e)\n    schedule_retry_with_backoff(request)  # or fall back to another provider","preventionTips":["Monitor provider endpoint health before enqueueing generation jobs.","Cap job concurrency so you don't trip rate limits that exhaust retries.","Log the wrapped last_exc to distinguish timeouts from HTTP failures.","Configure a fallback provider for outages."],"tags":["network","retry-exhausted","external-api","http"],"backgroundTag":"retries-exhausted","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}