{"record":{"id":"bc60bf1757c99a06","repo":"koala73/worldmonitor","slug":"classify-eval-could-not-find-what","errorCode":null,"errorMessage":"classify-eval: could not find ${what}","messagePattern":"classify-eval: could not find (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scripts/lib/classify-eval.mjs","lineNumber":12,"sourceCode":"// Pure helpers behind scripts/eval-classify-labels.mjs and its tests: read the two\n// headline-classification prompts and their OpenRouter model straight out of the\n// source files (so the eval can never drift from what production sends), and score a\n// set of labels against blind-judged levels.\nimport { createHash } from 'node:crypto';\n\nexport const ALERT_LEVELS = ['critical', 'high'];\nexport const isAlertLevel = (level) => ALERT_LEVELS.includes(level);\n\nfunction mustMatch(src, re, what) {\n  const m = src.match(re);\n  if (!m) throw new Error(`classify-eval: could not find ${what}`);\n  return m[1];\n}\n\n// A prompt is hashed and sent as source text, so it must BE the text production sends:\n// an interpolated template would pin (and send) a literal `${...}`.\nfunction staticTemplate(text, what) {\n  if (text.includes('${')) throw new Error(`classify-eval: ${what} is interpolated; the eval can only read a static template literal`);\n  return text;\n}\n\n// scripts/ais-relay.cjs — the batch prompt (50 titles per completion).\nexport function extractRelayPrompt(relaySrc) {\n  return staticTemplate(mustMatch(relaySrc, /const CLASSIFY_SYSTEM_PROMPT = `([\\s\\S]*?)`;/, 'CLASSIFY_SYSTEM_PROMPT'), 'CLASSIFY_SYSTEM_PROMPT');\n}\n\n// Reads ONLY the `openrouter` entry, comments stripped. A model held in a constant, or\n// missing, throws rather than falling through to a later provider's literal: this value\n// is what `--capture` sends and records, so a wrong read would pin itself.","sourceCodeStart":1,"sourceCodeEnd":30,"githubUrl":"https://github.com/koala73/worldmonitor/blob/e586b8b4b80f595aa7ece295eec10d76f2921240/scripts/lib/classify-eval.mjs#L1-L30","documentation":"mustMatch is a helper that extracts a capture group from source text using a regex and throws 'classify-eval: could not find <what>' when the regex does not match. The classify-eval harness uses it to locate exact prompt/provider literals (e.g. CLASSIFY_SYSTEM_PROMPT) in production source files; if the anchor text has drifted, extraction fails loudly rather than pinning the wrong prompt.","triggerScenarios":"Running classify-eval when the target source file no longer contains the expected literal, e.g. the regex /const CLASSIFY_SYSTEM_PROMPT = `([\\s\\S]*?)`;/ no longer matches because the constant was renamed, reformatted, moved, or changed to an interpolated value.","commonSituations":"Refactoring the relay or classify-event source (renaming constants, changing template syntax) without updating classify-eval's regexes; running the eval against a stale or wrong file path.","solutions":["Update the regex in classify-eval.mjs to match the renamed/reformatted literal in the source file.","Verify you are pointing classify-eval at the correct current source file (scripts/ais-relay.cjs or the classify-event.ts path).","If the constant is now interpolated or dynamic, restore a static template literal so the eval can hash the exact production text.","Run the eval again after aligning the source and the extraction patterns."],"exampleFix":"// before (source renamed)\nconst CLASSIFY_PROMPT = `...`;\n// after (align source with eval regex)\nconst CLASSIFY_SYSTEM_PROMPT = `...`;","handlingStrategy":"validation","validationCode":"if (!/const CLASSIFY_SYSTEM_PROMPT = `[\\s\\S]*?`;/.test(relaySrc)) throw new Error('relay source missing CLASSIFY_SYSTEM_PROMPT literal');","typeGuard":null,"tryCatchPattern":"try { const prompt = extractRelayPrompt(relaySrc); } catch (e) { if (e.message.startsWith('classify-eval: could not find')) { console.error('Source literal drifted; update classify-eval regexes or restore the constant'); } throw e; }","preventionTips":["Treat the prompt constant names referenced by classify-eval as a stable public contract","Add a CI check that runs classify-eval extraction after refactors of the relay/classify sources","When renaming constants, grep classify-eval.mjs for the old name and update the regexes together","Keep prompts as static template literals at a fixed location in the file"],"tags":["build-script","regex","eval-harness","source-extraction"],"backgroundTag":"resource-not-found","analyzedSha":"e586b8b4b80f595aa7ece295eec10d76f2921240","analyzedAt":"2026-09-22T01:50:49.965Z","contentChangedAt":"2026-09-22T01:50:49.965Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}