{"record":{"id":"86274abb5972abad","repo":"koala73/worldmonitor","slug":"classify-eval-what-is-interpolated-the-eval-can-only-read-a","errorCode":null,"errorMessage":"classify-eval: ${what} is interpolated; the eval can only read a static template literal","messagePattern":"classify-eval: (.+?) is interpolated; the eval can only read a static template literal","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scripts/lib/classify-eval.mjs","lineNumber":19,"sourceCode":"// Pure helpers behind scripts/eval-classify-labels.mjs and its tests: read the two\n// headline-classification prompts and their OpenRouter model straight out of the\n// source files (so the eval can never drift from what production sends), and score a\n// set of labels against blind-judged levels.\nimport { createHash } from 'node:crypto';\n\nexport const ALERT_LEVELS = ['critical', 'high'];\nexport const isAlertLevel = (level) => ALERT_LEVELS.includes(level);\n\nfunction mustMatch(src, re, what) {\n  const m = src.match(re);\n  if (!m) throw new Error(`classify-eval: could not find ${what}`);\n  return m[1];\n}\n\n// A prompt is hashed and sent as source text, so it must BE the text production sends:\n// an interpolated template would pin (and send) a literal `${...}`.\nfunction staticTemplate(text, what) {\n  if (text.includes('${')) throw new Error(`classify-eval: ${what} is interpolated; the eval can only read a static template literal`);\n  return text;\n}\n\n// scripts/ais-relay.cjs — the batch prompt (50 titles per completion).\nexport function extractRelayPrompt(relaySrc) {\n  return staticTemplate(mustMatch(relaySrc, /const CLASSIFY_SYSTEM_PROMPT = `([\\s\\S]*?)`;/, 'CLASSIFY_SYSTEM_PROMPT'), 'CLASSIFY_SYSTEM_PROMPT');\n}\n\n// Reads ONLY the `openrouter` entry, comments stripped. A model held in a constant, or\n// missing, throws rather than falling through to a later provider's literal: this value\n// is what `--capture` sends and records, so a wrong read would pin itself.\nexport function extractRelayOpenRouterModel(relaySrc) {\n  const list = mustMatch(relaySrc, /const CLASSIFY_LLM_PROVIDERS = \\[([\\s\\S]*?)\\n\\];/, 'CLASSIFY_LLM_PROVIDERS');\n  const entries = list.split(/(?=\\{\\s*\\n\\s*name: ')/).filter((e) => /^\\{\\s*\\n\\s*name: 'openrouter',/.test(e));\n  if (entries.length !== 1) throw new Error(`classify-eval: expected one openrouter classify provider, found ${entries.length}`);\n  const code = entries[0].replace(/\\/\\/[^\\n]*/g, '');\n  return mustMatch(code, /\\n\\s*model: '([^']+)',/, \"a quoted model on the relay's openrouter classify provider\");\n}","sourceCodeStart":1,"sourceCodeEnd":37,"githubUrl":"https://github.com/koala73/worldmonitor/blob/e586b8b4b80f595aa7ece295eec10d76f2921240/scripts/lib/classify-eval.mjs#L1-L37","documentation":"staticTemplate guards classify-eval's core invariant: prompts are hashed and sent as literal source text, so a template literal containing '${' would hash and send a literal interpolation placeholder instead of the real runtime prompt. If the extracted prompt text includes '${', it throws naming the offending item. This prevents the eval from silently pinning an incorrect prompt.","triggerScenarios":"Running classify-eval when the production prompt constant was converted into an interpolated template literal, e.g. `Prompt for ${modelName}` or a template embedding config values, so the extracted source contains '${'.","commonSituations":"A developer refactoring prompts to interpolate model names, dates, or thresholds; templating added for localization; generated prompt files where an outer template leaks inner '${'.","solutions":["Rewrite the production prompt as a static template literal with no ${...} interpolation.","If runtime values are needed, move them into the completion request as separate parameters/messages, not into the system prompt text.","Concatenate fixed string pieces instead of interpolating, only if the concatenation itself is static at the definition site.","Re-run classify-eval to confirm the extracted prompt is static before trusting captured results."],"exampleFix":"// before\nconst CLASSIFY_SYSTEM_PROMPT = `Classify for model ${MODEL_NAME}.`;\n// after\nconst CLASSIFY_SYSTEM_PROMPT = `Classify the title.`; // model passed as a request parameter","handlingStrategy":"validation","validationCode":"const m = relaySrc.match(/const CLASSIFY_SYSTEM_PROMPT = `([\\s\\S]*?)`;/);\nif (m && m[1].includes('${')) throw new Error('CLASSIFY_SYSTEM_PROMPT is interpolated');","typeGuard":"const isStaticTemplate = (text) => typeof text === 'string' && !text.includes('${');","tryCatchPattern":"try { const prompt = extractRelayPrompt(src); } catch (e) { if (e.message.includes('is interpolated')) { console.error('Remove ${...} from the production prompt; pass dynamic values as request parameters'); } throw e; }","preventionTips":["Never interpolate values into system prompt constants","Pass dynamic data (model name, thresholds) as separate API parameters or user messages","Add a lint/test that scans prompt constants for '${'","Review prompt-related PRs for template interpolation before merge"],"tags":["eval-harness","llm","prompt-validation","invariant"],"backgroundTag":"unsupported-operation","analyzedSha":"e586b8b4b80f595aa7ece295eec10d76f2921240","analyzedAt":"2026-09-22T01:50:49.965Z","contentChangedAt":"2026-09-22T01:50:49.965Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}