What causes "Immutable Evalbook record changed: ${path}" in paperclipai/paperclip?

Use a new output/attempt path (new revision or run id) instead of re-rendering an existing attempt

paperclipai/paperclip · error · Error

Immutable Evalbook record changed: ${path}

Error message

Immutable Evalbook record changed: ${path}

What it means

writeImmutable enforces that once an Evalbook record exists on disk it never changes: if the serialized new content differs from the file's existing content it throws, preserving the append-only/immutable guarantee of the Evalbook history. ENOENT is the only tolerated case (write a new file with mode 0600).

Source

Thrown at packages/paperclip-runner/scripts/render-runner-workflow-evalbook.mjs:217

  const config = {
    schema: "paperclip-runner/workflow-eval-config/v1",
    id: result.candidateId,
    model,
    provider: result.observation.provider,
    driver,
    runnerVersion: report.bundle.runnerVersion,
    runnerBuild: report.bundle.runnerBuild,
    promptPolicyId: report.bundle.promptPolicyId,
  };
  return { attemptId, artifact, score, case: evalCase, config };
}

async function writeImmutable(path, value) {
  const content = json(value);
  try {
    const existing = await readFile(path, "utf8");
    if (existing !== content) {
      throw new Error(`Immutable Evalbook record changed: ${path}`);
    }
  } catch (error) {
    if (error?.code !== "ENOENT") throw error;
    await writeFile(path, content, { flag: "wx", mode: 0o600 });
  }
}

export async function writeRunnerWorkflowEvalbookAttempts({
  report,
  runsRoot,
  caseForId,
}) {
  await mkdir(runsRoot, { recursive: true });
  const attempts = [];
  for (const result of report.results) {
    const attempt = runnerWorkflowEvalbookAttempt({
      report,
      result,

View on GitHub (pinned to 01ad858492)

Solutions

Use a new output/attempt path (new revision or run id) instead of re-rendering an existing attempt
Revert the underlying data change so the record matches what was previously written
Diff the existing file against the new json(value) to find the source of drift (often timestamps or ordering)
If drift is from tooling upgrades, render into a fresh directory rather than over an old Evalbook

Example fix

// before
await writeImmutable('attempts/run-1.json', recordWithNewTimestamp);
// after
await writeImmutable('attempts/run-2.json', recordWithNewTimestamp);

Defensive patterns

Strategy: try-catch

Validate before calling

const next = json(value); const prev = await readFile(path,'utf8').catch(e => e.code === 'ENOENT' ? null : e); if (prev !== null && prev !== next) throw new Error('Record would change: ' + path);

Type guard

null

Try / catch

try { await writeImmutable(path, value); } catch (e) { if (String(e.message).startsWith('Immutable Evalbook record changed')) { console.error('Diff existing file vs new content for', path); } throw e; }

Prevention

Keep Evalbook records deterministic (no timestamps or unstable ordering)
Render new attempts to new paths, never rewrite existing ones
Diff before writing when data may have changed

When it happens

Trigger: Re-rendering an attempt whose recorded JSON differs from what's on disk — e.g. rerunning with different input data, a changed renderer version, or nondeterministic content (timestamps, ordering) embedded in the record; writing to the same attempt path with updated results.

Common situations: Editing a historical attempt's data then re-running the renderer; upgrading the eval program so record fields change for an existing attempt; two runs racing on the same output directory producing different content for the same path.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/c57363654d919ec2. Report an issue: GitHub.