invoke-ai/InvokeAI · error · ExternalProviderRequestError

{label} failed after retries: {last_exc}

Error message

{label} failed after retries: {last_exc}

What it means

ExternalProviderRequestError raised by the AlibabaCloud provider's _request_with_retry helper after every retry attempt has been exhausted without receiving an HTTP response it could return. The provider retries on transient failures (429/5xx, network errors) with backoff, and when even the final attempt fails, last_exc is re-raised wrapped in this message. It means the AlibabaCloud endpoint was persistently unreachable or kept returning retryable errors for the whole retry window.

Source

Thrown at invokeai/app/services/external_generation/providers/alibabacloud.py:400

            if response.status_code in _RETRY_STATUS_CODES and attempt < _MAX_RETRIES:
                delay = self._retry_delay(response, attempt)
                self._logger.warning(
                    "%s got status %d on attempt %d/%d — retrying in %.1fs",
                    label,
                    response.status_code,
                    attempt + 1,
                    _MAX_RETRIES + 1,
                    delay,
                )
                time.sleep(delay)
                continue

            return response

        # Unreachable: the loop either returns a response or raises.
        assert last_exc is not None
        raise ExternalProviderRequestError(f"{label} failed after retries: {last_exc}") from last_exc

    @staticmethod
    def _retry_delay(response: requests.Response, attempt: int) -> float:
        retry_after = response.headers.get("Retry-After")
        if retry_after:
            try:
                return max(0.0, float(retry_after))
            except ValueError:
                pass
        return _RETRY_BACKOFF_BASE * (2**attempt)

View on GitHub (pinned to 0b6a024f2f)

Solutions

1. Check whether the AlibabaCloud status page / region reports an outage and retry the generation later.
2. Verify external_alibabacloud_base_url (and credentials) in InvokeAI config point to the correct reachable endpoint.
3. Inspect the wrapped last_exc message to identify the underlying cause (timeout vs HTTP status) and fix accordingly.
4. Reduce request concurrency or add queueing if you are being rate-limited (429) on every attempt.
5. Increase retry count/backoff in _retry_delay/_request_with_retry if transient network flakiness is expected.

Example fix

// before: no pre-check, request fails only after exhausting retries
result = provider.generate(request)
// after: probe reachability first and fail fast with a clearer message
if not provider.is_configured():
    raise RuntimeError("AlibabaCloud provider is not configured")
try:
    result = provider.generate(request)
except ExternalProviderRequestError as e:
    logger.warning("AlibabaCloud unreachable after retries: %s", e)
    result = fallback_provider.generate(request)

import requests
try:
    requests.get(base_url, timeout=5)
except requests.RequestException as e:
    raise RuntimeError(f"AlibabaCloud endpoint unreachable before generation: {e}")

try:
    result = provider.generate(request)
except ExternalProviderRequestError as e:
    logger.warning("AlibabaCloud failed after retries: %s", e)
    schedule_retry_with_backoff(request)  # or fall back to another provider

When it happens

Trigger: All retry attempts to the AlibabaCloud external-generation endpoint fail: connection errors, timeouts, or the server keeps returning 429/5xx status codes past the final attempt in _request_with_retry (called via _post_with_retry / _get_with_retry).

Common situations: AlibabaCloud regional outage or degradation; wrong external_alibabacloud_base_url pointing at an unreachable host; aggressive rate limiting from many queued jobs; firewall/proxy blocking egress; transient DNS failure during a burst of generations.

Related errors

• {reason}
• Failed to download image from DashScope: {exc}
• {label} network error: {exc}
• Resume refused by server. Restart required.
• Download interrupted. Resume required.

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/fee1600b04a7a072. Report an issue: GitHub.