{"record":{"id":"9b3f9328f2396074","repo":"santifer/career-ops","slug":"missing-replay-fixture-fixture-record-it-or","errorCode":null,"errorMessage":"missing replay fixture: ${fixture} — record it or run --live","messagePattern":"missing replay fixture: (.+?) — record it or run --live","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"eval-golden.mjs","lineNumber":145,"sourceCode":"\n/**\n * Return the candidate model's raw evaluation text for one golden case.\n *\n * In replay mode this reads a recorded fixture so the gate is offline and\n * deterministic; in live mode it shells out to openai-eval.mjs, reusing the\n * real prompt-assembly path rather than duplicating it here.\n *\n * @param {{id: string, jd: string}} testCase - The golden case being run.\n * @returns {string} Raw model output (expected to contain a SCORE_SUMMARY block).\n */\nfunction getCompletion(testCase) {\n  if (mode === 'replay') {\n    // Slash-form provider ids (e.g. \"deepseek/deepseek-chat\") must not become\n    // path separators, or the fixture lands in a phantom subdirectory. Sanitize\n    // to a flat filename — record fixtures under the same sanitized name.\n    const fixture = join(fixtureDir, `${testCase.id}__${fixtureModelId(model)}.txt`);\n    if (!existsSync(fixture)) {\n      throw new Error(`missing replay fixture: ${fixture} — record it or run --live`);\n    }\n    return readFileSync(fixture, 'utf8');\n  }\n\n  // live: write the JD to a temp file and run the existing evaluator.\n  const dir = mkdtempSync(join(tmpdir(), 'eval-golden-'));\n  try {\n    const jdFile = join(dir, 'jd.txt');\n    writeFileSync(jdFile, testCase.jd);\n    const res = spawnSync(process.execPath,\n      [join(ROOT, 'openai-eval.mjs'), '--file', jdFile, '--model', model, '--no-save'],\n      { encoding: 'utf8', env: process.env, timeout: 360000 });\n    if (res.status !== 0) {\n      throw new Error(`openai-eval.mjs exited ${res.status}: ${(res.stderr || '').slice(0, 200)}`);\n    }\n    return res.stdout || '';\n  } finally {\n    rmSync(dir, { recursive: true, force: true });","sourceCodeStart":127,"sourceCodeEnd":163,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/eval-golden.mjs#L127-L163","documentation":"In replay mode, getCompletion serves recorded LLM outputs from fixture files instead of calling the provider. Each golden test case maps to a fixture at fixtureDir/<caseId>__<sanitizedModelId>.txt; when that file does not exist the evaluation cannot proceed deterministically, so it throws telling you to either record the fixture or run with --live (which hits the real API).","triggerScenarios":"Running eval-golden.mjs (default replay mode) when: (1) a new golden case was added to the golden directory without recording its fixture; (2) the fixture filename's model id doesn't match the model being evaluated (model renamed, e.g. deepseek/deepseek-chat vs an alias); (3) the fixture file was moved/deleted or fixtureDir was relocated; (4) a slash-containing model id created a nested path expectation in an older run.","commonSituations":"Contributor adds a new test case in CI where fixtures aren't committed; developer switches default model in config so the fixture lookup key changes; running on a fresh clone where fixtures were gitignored by mistake; running replay offline expecting a fixture that only exists on another machine.","solutions":["Record the fixture by running the eval once with --live for that case/model so the output is written to fixtureDir/<id>__<model>.txt, then re-run in replay mode.","Check the exact expected path in the error and confirm the file exists there — if it's under a subdirectory (slash-form model id), move it to a flat sanitized filename.","Verify the model id passed matches the one used when the fixture was recorded; re-record if the provider renamed the model.","If you intentionally want a real API call, pass --live (requires API credentials) instead of replay.","Ensure fixture files are committed/not gitignored so CI and fresh clones have them."],"exampleFix":"# before: replay fails because fixture was never recorded\nnode eval-golden.mjs   # Error: missing replay fixture: .../case07__deepseek_deepseek-chat.txt\n\n# after: record it live once, then replay\nnode eval-golden.mjs --live   # writes the fixture\nnode eval-golden.mjs          # replay passes","handlingStrategy":"validation","validationCode":"import { existsSync } from 'node:fs';\nconst fixture = join(fixtureDir, `${caseId}__${model.replace(/\\//g, '_')}.txt`);\nif (!existsSync(fixture)) console.error(`Fixture missing: ${fixture} — run --live once first`);","typeGuard":null,"tryCatchPattern":"try {\n  const completion = getCompletion(testCase, model, { mode: 'replay' });\n} catch (err) {\n  if (err.message.startsWith('missing replay fixture')) {\n    console.error('Record it: node eval-golden.mjs --live  (or commit the fixture)');\n    process.exit(1);\n  }\n  throw err;\n}","preventionTips":["Record a fixture with --live in the same commit that adds a new golden case","Never gitignore the fixture directory; fixtures are test data, not artifacts","When changing the default model id, re-record all fixtures for the new key","Keep model ids flat/sanitized consistently so lookups and writes use the same filename"],"tags":["replay","fixture","testing","file-not-found"],"backgroundTag":"file-not-found","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}