decolua/9router · warning

[${provider}/${model}] ${errorMsg}

Error message

[${provider}/${model}] ${errorMsg}

What it means

HTTP 429/503-style unavailableResponse returned when every credential account for the target provider is currently rate-limited or unavailable. The message embeds the last upstream error and includes Retry-After info (credentials.retryAfter / retryAfterHuman). It is the rate-limited branch of the credentials loop in handleEmbeddings.

Source

Thrown at src/sse/handlers/embeddings.js:105

    log.info("ROUTING", `${modelStr} → ${provider}/${model}`);
  } else {
    log.info("ROUTING", `Provider: ${provider}, Model: ${model}`);
  }

  // Credential + fallback loop (mirrors handleChat)
  const excludeConnectionIds = new Set();
  let lastError = null;
  let lastStatus = null;

  while (true) {
    const credentials = await getProviderCredentials(provider, excludeConnectionIds, model);

    // All accounts unavailable
    if (!credentials || credentials.allRateLimited) {
      if (credentials?.allRateLimited) {
        const errorMsg = lastError || credentials.lastError || "Unavailable";
        const status = lastStatus || Number(credentials.lastErrorCode) || HTTP_STATUS.SERVICE_UNAVAILABLE;
        log.warn("EMBEDDINGS", `[${provider}/${model}] ${errorMsg} (${credentials.retryAfterHuman})`);
        return unavailableResponse(status, `[${provider}/${model}] ${errorMsg}`, credentials.retryAfter, credentials.retryAfterHuman);
      }
      if (excludeConnectionIds.size === 0) {
        log.error("AUTH", `No credentials for provider: ${provider}`);
        return errorResponse(HTTP_STATUS.BAD_REQUEST, `No credentials for provider: ${provider}`);
      }
      log.warn("EMBEDDINGS", "No more accounts available", { provider });
      return errorResponse(lastStatus || HTTP_STATUS.SERVICE_UNAVAILABLE, lastError || "All accounts unavailable");
    }

    log.info("AUTH", `\x1b[32mUsing ${provider} account: ${credentials.connectionName}\x1b[0m`);

    const refreshedCredentials = await checkAndRefreshToken(provider, credentials);

    const result = await handleEmbeddingsCore({
      body: { ...body, model: `${provider}/${model}` },
      modelInfo: { provider, model },
      credentials: refreshedCredentials,

View on GitHub (pinned to 90b52e06ff)

Solutions

1. Wait for the retryAfter period indicated in the response/log before retrying
2. Add more provider accounts/connections in the dashboard to spread load
3. Route through a combo with fallback providers for embeddings
4. Clear the account error state in the dashboard if the failure was transient and already recovered
5. Check upstream provider quotas and reduce request concurrency

Example fix

// before
await embed(model, inputs); // no backoff
// after
const res = await embed(model, inputs);
if (res.status === 429) { await sleep(res.retryAfterMs ?? 30000); return embed(model, inputs); }

// Pre-check on the client: spread heavy embedding jobs over time
const BATCH_DELAY_MS = 1000;
for (const batch of batches) {
  await embed(batch);
  await new Promise(r => setTimeout(r, BATCH_DELAY_MS));
}

const res = await post('/v1/embeddings', payload);
if (res.status === 429 || res.status === 503) {
  const retryAfter = Number(res.headers.get('retry-after')) * 1000 || 30000;
  await sleep(retryAfter);
  return post('/v1/embeddings', payload); // bounded retries
}

When it happens

Trigger: All connections for the provider were marked unavailable by markAccountUnavailable (429s, quota errors, upstream auth failures), so getProviderCredentials returns { allRateLimited: true } on the next loop iteration.

Common situations: Burst embedding jobs exhausting per-account quotas; single account provider hitting its rate cap; upstream provider outage causing all accounts to be flagged unavailable; expired tokens repeatedly failing and tripping the unavailable flag.

Related errors

• [AutoPing] ${provider}:${connection.id}: ping failed (reset
• MiMo bootstrap failed: ${response.status}
• NanoBanana status ${r.status}
• Runway status ${r.status}
• Bing translator fetch failed: ${res.status}

AI-assisted analysis of decolua/9router@90b52e06ff (2026-08-30). Data as JSON: /api/errors/73bbd64e01068a41. Report an issue: GitHub.