{"record":{"id":"8721ca922d62df12","repo":"santifer/career-ops","slug":"all-active-length-active-models-failed-last-er","errorCode":null,"errorMessage":"All ${active.length} active models failed. Last error: ${lastError?.message}","messagePattern":"All (.+?) active models failed\\. Last error: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"critical","filePath":"openrouter-runner.mjs","lineNumber":368,"sourceCode":"        console.log(`SKIP (blacklisted: ${msg})`);\n      } else if (is429) {\n        rateLimitCounts[model] = (rateLimitCounts[model] ?? 0) + 1;\n        if (rateLimitCounts[model] >= 3) {\n          blacklistedModels.add(model);\n          saveBlacklist(blacklistedModels);\n          console.log(`SKIP (auto-blacklisted: persistent 429)`);\n        } else {\n          console.log(`FAILED (HTTP 429 [${rateLimitCounts[model]}/3])`);\n          await new Promise(r => setTimeout(r, 800));\n        }\n      } else {\n        console.log(`FAILED (${msg})`);\n        await new Promise(r => setTimeout(r, 800));\n      }\n    }\n  }\n\n  throw new Error(`All ${active.length} active models failed. Last error: ${lastError?.message}`);\n}\n\n// ---------------------------------------------------------------------------\n// Context loading\n// ---------------------------------------------------------------------------\nfunction loadContext() {\n  return {\n    cv:          readFile('cv.md')               ?? 'CV not found.',\n    profile:     readFile('config/profile.yml')  ?? '',\n    shared:      readFile('modes/_shared.md')    ?? '',\n    profileMode: readFile('modes/_profile.md')   ?? '',\n  };\n}\n\nexport function buildSystemPrompt(modeContent, ctx) {\n  const languageInstruction = outputLanguageInstruction(parseOutputLanguage(ctx.profile));\n  return [\n    ctx.shared,","sourceCodeStart":350,"sourceCodeEnd":386,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/openrouter-runner.mjs#L350-L386","documentation":"callOpenRouter iterates over the pool of enabled OpenRouter models (after blacklisting), retrying each with rotation on failure. If every active model attempt fails — due to HTTP errors, rate limits, timeouts, or empty responses — it abandons retrying and throws this aggregate error carrying the last underlying error message. It means the LLM call could not be completed by any model in the configured pool, so the caller has no usable completion.","triggerScenarios":"Any callOpenRouter invocation where all models in the active pool fail: OpenRouter API returns non-2xx for every model (401 bad OPENROUTER_API_KEY, 402 insufficient credits, 403 model moderation/availability, 429 persistent rate limit), every request exceeds MODEL_TIMEOUT_MS, every response has data.error set, or every response has an empty choices[0].message.content.","commonSituations":"Expired or missing OpenRouter API key, exhausted credits, all configured models deprecated/removed from OpenRouter, no network access or corporate proxy blocking openrouter.ai, prompts too large for every model in the pool, or a persistent 429 that blacklisted all models one by one until the pool was empty.","solutions":["Read the 'Last error:' suffix in the message — it names the real cause (e.g. HTTP 401 vs 402 vs 429) and fix that first","Verify OPENROUTER_API_KEY is set and valid, and that the account has credits (OpenRouter dashboard)","Check the configured model names against https://openrouter.ai/models — replace any deprecated slugs","Inspect and reset the persisted model blacklist (saveBlacklist target file); blacklisted models are excluded from 'active' and may leave the pool too small","Confirm outbound network access to openrouter.ai (proxy/firewall/DNS) and retry later if the cause was a transient rate limit"],"exampleFix":"// before: single model pool, all fail\nMODELS = ['openai/gpt-4o'];\n\n// after: diversify the pool and keep it valid\nMODELS = [\n  'openai/gpt-4o',\n  'anthropic/claude-3.5-sonnet',\n  'google/gemini-2.0-flash-001',\n];\n// plus: export OPENROUTER_API_KEY=sk-or-... (fresh key, funded account)","handlingStrategy":"retry","validationCode":"function canAttemptCall() {\n  if (!process.env.OPENROUTER_API_KEY) throw new Error('OPENROUTER_API_KEY not set');\n  if (activeModels.length === 0) throw new Error('No active models left in pool (all blacklisted?)');\n  return true;\n}","typeGuard":null,"tryCatchPattern":"try {\n  const { content } = await callOpenRouter(prompt);\n} catch (e) {\n  if (e.message.startsWith('All ') && e.message.includes('active models failed')) {\n    // inspect lastError suffix: 401/402 → fix key/credits; 429 → back off longer;\n    // timeout → shrink prompt. Then retry with fresh models or abort the run.\n    console.error('LLM pool exhausted:', e.message.match(/Last error: (.*)/)?.[1]);\n  } else throw e;\n}","preventionTips":["Check the OpenRouter key and credit balance at startup before batch runs","Keep a diverse model pool (different providers) so a single provider outage doesn't empty it","Periodically clear stale entries from the persisted blacklist file","Watch for repeated 'FAILED (HTTP 429)' logs and add long backoff before the pool drains","Size prompts below the context limit of the smallest model in the pool"],"tags":["llm","openrouter","api","retry-exhausted","network"],"backgroundTag":"api-request-failed","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}