santifer/career-ops · error

missing replay fixture

Error message

missing replay fixture: ${fixture} — record it or run --live

What it means

In replay mode, getCompletion serves recorded LLM outputs from fixture files instead of calling the provider. Each golden test case maps to a fixture at fixtureDir/<caseId>__<sanitizedModelId>.txt; when that file does not exist the evaluation cannot proceed deterministically, so it throws telling you to either record the fixture or run with --live (which hits the real API).

Solutions

1. Record the fixture by running the eval once with --live for that case/model so the output is written to fixtureDir/<id>__<model>.txt, then re-run in replay mode.
2. Check the exact expected path in the error and confirm the file exists there — if it's under a subdirectory (slash-form model id), move it to a flat sanitized filename.
3. Verify the model id passed matches the one used when the fixture was recorded; re-record if the provider renamed the model.
4. If you intentionally want a real API call, pass --live (requires API credentials) instead of replay.
5. Ensure fixture files are committed/not gitignored so CI and fresh clones have them.

Example fix

# before: replay fails because fixture was never recorded
node eval-golden.mjs   # Error: missing replay fixture: .../case07__deepseek_deepseek-chat.txt

# after: record it live once, then replay
node eval-golden.mjs --live   # writes the fixture
node eval-golden.mjs          # replay passes

import { existsSync } from 'node:fs';
const fixture = join(fixtureDir, `${caseId}__${model.replace(/\//g, '_')}.txt`);
if (!existsSync(fixture)) console.error(`Fixture missing: ${fixture} — run --live once first`);

try {
  const completion = getCompletion(testCase, model, { mode: 'replay' });
} catch (err) {
  if (err.message.startsWith('missing replay fixture')) {
    console.error('Record it: node eval-golden.mjs --live  (or commit the fixture)');
    process.exit(1);
  }
  throw err;
}

When it happens

Trigger: Running eval-golden.mjs (default replay mode) when: (1) a new golden case was added to the golden directory without recording its fixture; (2) the fixture filename's model id doesn't match the model being evaluated (model renamed, e.g. deepseek/deepseek-chat vs an alias); (3) the fixture file was moved/deleted or fixtureDir was relocated; (4) a slash-containing model id created a nested path expectation in an older run.

Common situations: Contributor adds a new test case in CI where fixtures aren't committed; developer switches default model in config so the fixture lookup key changes; running on a fresh clone where fixtures were gitignored by mistake; running replay offline expecting a fixture that only exists on another machine.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• config/profile.yml not found: fill it in first
• CV file not found at
• ENOENT
• invalid golden case : need string id/jd and label.
• ⚠️ not found at

AI-assisted analysis of santifer/career-ops@aac998c7ed (2026-09-16). Data as JSON: /api/errors/9b3f9328f2396074. Report an issue: GitHub.

Appendix: source

Thrown at eval-golden.mjs:145

/**
 * Return the candidate model's raw evaluation text for one golden case.
 *
 * In replay mode this reads a recorded fixture so the gate is offline and
 * deterministic; in live mode it shells out to openai-eval.mjs, reusing the
 * real prompt-assembly path rather than duplicating it here.
 *
 * @param {{id: string, jd: string}} testCase - The golden case being run.
 * @returns {string} Raw model output (expected to contain a SCORE_SUMMARY block).
 */
function getCompletion(testCase) {
  if (mode === 'replay') {
    // Slash-form provider ids (e.g. "deepseek/deepseek-chat") must not become
    // path separators, or the fixture lands in a phantom subdirectory. Sanitize
    // to a flat filename — record fixtures under the same sanitized name.
    const fixture = join(fixtureDir, `${testCase.id}__${fixtureModelId(model)}.txt`);
    if (!existsSync(fixture)) {
      throw new Error(`missing replay fixture: ${fixture} — record it or run --live`);
    }
    return readFileSync(fixture, 'utf8');
  }

  // live: write the JD to a temp file and run the existing evaluator.
  const dir = mkdtempSync(join(tmpdir(), 'eval-golden-'));
  try {
    const jdFile = join(dir, 'jd.txt');
    writeFileSync(jdFile, testCase.jd);
    const res = spawnSync(process.execPath,
      [join(ROOT, 'openai-eval.mjs'), '--file', jdFile, '--model', model, '--no-save'],
      { encoding: 'utf8', env: process.env, timeout: 360000 });
    if (res.status !== 0) {
      throw new Error(`openai-eval.mjs exited ${res.status}: ${(res.stderr || '').slice(0, 200)}`);
    }
    return res.stdout || '';
  } finally {
    rmSync(dir, { recursive: true, force: true });

View on GitHub (pinned to aac998c7ed)