{"record":{"id":"c52241514ed64770","repo":"decolua/9router","slug":"all-accounts-unavailable-c52241","errorCode":null,"errorMessage":"All accounts unavailable","messagePattern":"All accounts unavailable","errorType":"http","errorClass":null,"httpStatus":503,"severity":"error","filePath":"src/sse/handlers/fetch.js","lineNumber":175,"sourceCode":"  const excludeConnectionIds = new Set();\n  let lastError = null;\n  let lastStatus = null;\n\n  while (true) {\n    const credentials = await getProviderCredentials(providerId, excludeConnectionIds);\n\n    if (!credentials || credentials.allRateLimited) {\n      if (credentials?.allRateLimited) {\n        const errorMsg = lastError || credentials.lastError || \"Unavailable\";\n        const status = lastStatus || Number(credentials.lastErrorCode) || HTTP_STATUS.SERVICE_UNAVAILABLE;\n        log.warn(\"FETCH\", `[${providerId}] ${errorMsg} (${credentials.retryAfterHuman})`);\n        return unavailableResponse(status, `[${providerId}] ${errorMsg}`, credentials.retryAfter, credentials.retryAfterHuman);\n      }\n      if (excludeConnectionIds.size === 0) {\n        log.error(\"AUTH\", `No credentials for provider: ${providerId}`);\n        return errorResponse(HTTP_STATUS.BAD_REQUEST, `No credentials for provider: ${providerId}`);\n      }\n      log.warn(\"FETCH\", \"No more accounts available\", { provider: providerId });\n      return errorResponse(lastStatus || HTTP_STATUS.SERVICE_UNAVAILABLE, lastError || \"All accounts unavailable\");\n    }\n\n    log.info(\"AUTH\", `\\x1b[32mUsing ${providerId} account: ${credentials.connectionName}\\x1b[0m`);\n\n    const refreshedCredentials = await checkAndRefreshToken(providerId, credentials);\n\n    const result = await handleFetchCore({\n      url: targetUrl,\n      format,\n      maxCharacters,\n      provider: resolvedProvider.id,\n      providerConfig,\n      credentials: refreshedCredentials,\n      log,\n      onCredentialsRefreshed: async (newCreds) => {\n        await updateProviderCredentials(credentials.connectionId, {\n          accessToken: newCreds.accessToken,","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/decolua/9router/blob/90b52e06ffd666b7929554211474d01588f6b1f8/src/sse/handlers/fetch.js#L157-L193","documentation":"Raised in handleSingleProviderFetch (src/sse/handlers/fetch.js:175-176) when the credential fallback loop is exhausted: excludeConnectionIds is non-empty (at least one account was already tried and failed) and getProviderCredentials can no longer return a usable connection. The gateway has tried each stored account for the provider, each was marked unavailable via markAccountUnavailable, and there is nothing left to try. It returns the last upstream error/status if available, otherwise 503 'All accounts unavailable'.","triggerScenarios":"POST /v1/fetch against a provider with multiple connections where every connection fails in sequence: the first upstream attempt returns a fallback-worthy status (e.g. 429/401/5xx), shouldFallback is true, the connection id is added to excludeConnectionIds, the loop continues, and the next getProviderCredentials call finds no remaining eligible account.","commonSituations":"Multi-account setups where all keys for a provider expired, were revoked, or hit quota simultaneously (often from a shared IP being throttled upstream); a combo spreading load across accounts of the same dying provider; OAuth connections whose refresh tokens silently expired so every account fails auth on use.","solutions":["Open the dashboard and inspect all connections for that provider; re-authenticate or replace the failing credentials.","Fix the underlying lastError reported in the response (quota, expired token, bad key) for each account before retrying.","Add healthy accounts for the provider so the fallback loop has more than one candidate.","Use a combo spanning different providers so a total failure of one provider falls through to another.","If the accounts are actually fine and were locked by a transient upstream outage, clear the account error state and retry after the upstream recovers."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// Verify the provider has healthy accounts before batching work:\nconst status = await fetch(base + '/dashboard/api/providers').then(r => r.json());\nconst p = status.providers?.find(p => p.id === 'exa');\nif (!p || p.connections?.every(c => c.status !== 'active')) throw new Error('No healthy accounts for provider');","typeGuard":null,"tryCatchPattern":"const res = await doFetch();\nif (res.status === 503 && (await res.text()).includes('All accounts unavailable')) {\n  return fallbackToAlternateProvider(request);\n}\nreturn res;","preventionTips":["Keep at least two active connections for any business-critical provider.","Re-authenticate OAuth connections before their tokens expire (watch dashboard warnings).","Spread load across accounts/combos instead of exhausting one provider.","Alert on lastError/lastErrorCode stored per account so the first failure gets fixed before all accounts lock out."],"tags":["credentials","multi-account","fallback-exhausted","web-fetch"],"backgroundTag":"no-accounts-available","analyzedSha":"90b52e06ffd666b7929554211474d01588f6b1f8","analyzedAt":"2026-08-30T21:05:45.952Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}