classify-eval: expected one openrouter classify provider…

Error message

classify-eval: expected one openrouter classify provider, found ${entries.length}

What it means

extractRelayOpenRouterModel reads only the `openrouter` entry of CLASSIFY_LLM_PROVIDERS from the relay source, splitting the array into entries by name. Because this model value is what --capture sends and records, an ambiguous or missing read would pin the wrong literal, so it throws when the number of matched openrouter entries is not exactly one. The error reports the actual count found.

Solutions

Ensure CLASSIFY_LLM_PROVIDERS contains exactly one entry with name: 'openrouter' — remove duplicates or restore a removed entry.
If the provider was renamed, update the regex filter (/name: 'openrouter',/) in classify-eval.mjs.
Check that the array literal still matches /const CLASSIFY_LLM_PROVIDERS = \[...\n\];/ and that formatting changes did not confuse entry splitting.
Confirm the quoted model field on that entry so the subsequent mustMatch can read it.

Example fix

// before (duplicated)
const CLASSIFY_LLM_PROVIDERS = [{ name: 'openrouter', model: 'a' }, { name: 'openrouter', model: 'b' }];
// after
const CLASSIFY_LLM_PROVIDERS = [{ name: 'openrouter', model: 'a' }];

Defensive patterns

Strategy: validation

Validate before calling

const count = (relaySrc.match(/name: 'openrouter',/g) || []).length;
if (count !== 1) throw new Error(`expected exactly one openrouter provider, found ${count}`);

Try / catch

try { const model = extractRelayOpenRouterModel(src); } catch (e) { if (e.message.includes('openrouter classify provider')) { console.error('Fix CLASSIFY_LLM_PROVIDERS to hold exactly one openrouter entry'); } throw e; }

Prevention

Keep exactly one 'openrouter' provider entry in CLASSIFY_LLM_PROVIDERS
If a second openrouter-backed model is needed, use a distinct provider name or extend the eval extractor
Update classify-eval filters in the same PR that renames providers
Run classify-eval --capture in CI to catch extraction drift early

When it happens

Trigger: Running classify-eval when CLASSIFY_LLM_PROVIDERS contains zero openrouter entries (renamed, removed, or the array regex matched the wrong region) or more than one (duplicated provider blocks, e.g. two openrouter entries for different tiers).

Common situations: Renaming the provider from 'openrouter' in the relay config; copy-pasting a provider block to add a second model; reformatting the array (extra blank lines/comments) so the split-by-name heuristic miscounts.

Related errors

AI-assisted analysis of koala73/worldmonitor@e586b8b4b8 (2026-09-22). Data as JSON: /api/errors/0840054c80fe6f00. Report an issue: GitHub.

Appendix: source

Thrown at scripts/lib/classify-eval.mjs:34

// A prompt is hashed and sent as source text, so it must BE the text production sends:
// an interpolated template would pin (and send) a literal `${...}`.
function staticTemplate(text, what) {
  if (text.includes('${')) throw new Error(`classify-eval: ${what} is interpolated; the eval can only read a static template literal`);
  return text;
}

// scripts/ais-relay.cjs — the batch prompt (50 titles per completion).
export function extractRelayPrompt(relaySrc) {
  return staticTemplate(mustMatch(relaySrc, /const CLASSIFY_SYSTEM_PROMPT = `([\s\S]*?)`;/, 'CLASSIFY_SYSTEM_PROMPT'), 'CLASSIFY_SYSTEM_PROMPT');
}

// Reads ONLY the `openrouter` entry, comments stripped. A model held in a constant, or
// missing, throws rather than falling through to a later provider's literal: this value
// is what `--capture` sends and records, so a wrong read would pin itself.
export function extractRelayOpenRouterModel(relaySrc) {
  const list = mustMatch(relaySrc, /const CLASSIFY_LLM_PROVIDERS = \[([\s\S]*?)\n\];/, 'CLASSIFY_LLM_PROVIDERS');
  const entries = list.split(/(?=\{\s*\n\s*name: ')/).filter((e) => /^\{\s*\n\s*name: 'openrouter',/.test(e));
  if (entries.length !== 1) throw new Error(`classify-eval: expected one openrouter classify provider, found ${entries.length}`);
  const code = entries[0].replace(/\/\/[^\n]*/g, '');
  return mustMatch(code, /\n\s*model: '([^']+)',/, "a quoted model on the relay's openrouter classify provider");
}

// server/worldmonitor/intelligence/v1/classify-event.ts — one title per completion.
export function extractRpcPrompt(rpcSrc) {
  return staticTemplate(mustMatch(rpcSrc, /const systemPrompt = `([\s\S]*?)`;/, 'classify-event systemPrompt'), 'classify-event systemPrompt');
}

export function extractRpcOpenRouterModel(rpcSrc) {
  return mustMatch(rpcSrc, /^const CLASSIFY_OPENROUTER_MODEL = '([^']+)';/m, 'CLASSIFY_OPENROUTER_MODEL');
}

// The block both prompts must carry verbatim, delimited by its first and last line.
export function extractHighBoundaryBlock(prompt) {
  return mustMatch(prompt, /(Do not under-rate "high"\.[\s\S]*?not for the event itself\.)/, 'the "high" boundary block');
}

View on GitHub (pinned to e586b8b4b8)