invoke-ai/InvokeAI · error · ExternalProviderRequestError

failed after retries

Error message

{label} failed after retries: {last_exc}

What it means

ExternalProviderRequestError raised by the AlibabaCloud provider's _request_with_retry helper after every retry attempt has been exhausted without receiving an HTTP response it could return. The provider retries on transient failures (429/5xx, network errors) with backoff, and when even the final attempt fails, last_exc is re-raised wrapped in this message. It means the AlibabaCloud endpoint was persistently unreachable or kept returning retryable errors for the whole retry window.

Solutions

  1. Check whether the AlibabaCloud status page / region reports an outage and retry the generation later.
  2. Verify external_alibabacloud_base_url (and credentials) in InvokeAI config point to the correct reachable endpoint.
  3. Inspect the wrapped last_exc message to identify the underlying cause (timeout vs HTTP status) and fix accordingly.
  4. Reduce request concurrency or add queueing if you are being rate-limited (429) on every attempt.
  5. Increase retry count/backoff in _retry_delay/_request_with_retry if transient network flakiness is expected.

Example fix

// before: no pre-check, request fails only after exhausting retries
result = provider.generate(request)
// after: probe reachability first and fail fast with a clearer message
if not provider.is_configured():
    raise RuntimeError("AlibabaCloud provider is not configured")
try:
    result = provider.generate(request)
except ExternalProviderRequestError as e:
    logger.warning("AlibabaCloud unreachable after retries: %s", e)
    result = fallback_provider.generate(request)
Defensive patterns

Strategy: retry

Validate before calling

import requests
try:
    requests.get(base_url, timeout=5)
except requests.RequestException as e:
    raise RuntimeError(f"AlibabaCloud endpoint unreachable before generation: {e}")

Try / catch

try:
    result = provider.generate(request)
except ExternalProviderRequestError as e:
    logger.warning("AlibabaCloud failed after retries: %s", e)
    schedule_retry_with_backoff(request)  # or fall back to another provider

Prevention

When it happens

Trigger: All retry attempts to the AlibabaCloud external-generation endpoint fail: connection errors, timeouts, or the server keeps returning 429/5xx status codes past the final attempt in _request_with_retry (called via _post_with_retry / _get_with_retry).

Common situations: AlibabaCloud regional outage or degradation; wrong external_alibabacloud_base_url pointing at an unreachable host; aggressive rate limiting from many queued jobs; firewall/proxy blocking egress; transient DNS failure during a burst of generations.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/fee1600b04a7a072. Report an issue: GitHub.

Appendix: source

Thrown at invokeai/app/services/external_generation/providers/alibabacloud.py:400

            if response.status_code in _RETRY_STATUS_CODES and attempt < _MAX_RETRIES:
                delay = self._retry_delay(response, attempt)
                self._logger.warning(
                    "%s got status %d on attempt %d/%d — retrying in %.1fs",
                    label,
                    response.status_code,
                    attempt + 1,
                    _MAX_RETRIES + 1,
                    delay,
                )
                time.sleep(delay)
                continue

            return response

        # Unreachable: the loop either returns a response or raises.
        assert last_exc is not None
        raise ExternalProviderRequestError(f"{label} failed after retries: {last_exc}") from last_exc

    @staticmethod
    def _retry_delay(response: requests.Response, attempt: int) -> float:
        retry_after = response.headers.get("Retry-After")
        if retry_after:
            try:
                return max(0.0, float(retry_after))
            except ValueError:
                pass
        return _RETRY_BACKOFF_BASE * (2**attempt)

View on GitHub (pinned to 0b6a024f2f)