paperclipai/paperclip · error · Error

Canonical Evalbook generator not found. Set PAPERCLIP_EVALBO

Error message

Canonical Evalbook generator not found. Set PAPERCLIP_EVALBOOK_PROGRAM to ${EVAL_PROGRAM_RELATIVE_PATH} in a paperclip-evals checkout.

What it means

resolveCanonicalEvalProgram searches known locations (including a sibling paperclip-evals checkout) for the canonical Evalbook generator script and throws when none exists. The error means the environment does not have the paperclip-evals repo checked out next to the package and PAPERCLIP_EVALBOOK_PROGRAM was not set to override the location.

Source

Thrown at packages/paperclip-runner/scripts/render-runner-workflow-evalbook.mjs:283

    ? resolve(environment.PAPERCLIP_EVALS_ROOT, EVAL_PROGRAM_RELATIVE_PATH)
    : null;
  const program = await existingPath([
    environment.PAPERCLIP_EVALBOOK_PROGRAM,
    fromRoot,
    resolve(packageRoot, "../../.paperclip-evals", EVAL_PROGRAM_RELATIVE_PATH),
    resolve(
      packageRoot,
      "../../../paperclip-evals",
      EVAL_PROGRAM_RELATIVE_PATH,
    ),
    resolve(
      packageRoot,
      "../../../../paperclip-evals",
      EVAL_PROGRAM_RELATIVE_PATH,
    ),
  ]);
  if (program !== null) return program;
  throw new Error(
    `Canonical Evalbook generator not found. Set PAPERCLIP_EVALBOOK_PROGRAM to ${EVAL_PROGRAM_RELATIVE_PATH} in a paperclip-evals checkout.`,
  );
}

async function run(command, args) {
  await new Promise((accept, reject) => {
    const child = spawn(command, args, { stdio: "inherit" });
    child.once("error", reject);
    child.once("exit", (code, signal) => {
      if (code === 0) accept();
      else
        reject(
          new Error(
            `${command} exited ${code ?? `for signal ${signal ?? "unknown"}`}`,
          ),
        );
    });
  });

View on GitHub (pinned to 01ad858492)

Solutions

1. Clone the paperclip-evals repository as a sibling checkout at the expected relative path
2. Set PAPERCLIP_EVALBOOK_PROGRAM to the absolute path of the generator (EVAL_PROGRAM_RELATIVE_PATH inside a paperclip-evals checkout)
3. In CI, add a checkout step for paperclip-evals into the expected adjacent directory
4. Verify the resolved path with ls and adjust PAPERCLIP_EVALBOOK_PROGRAM accordingly

Example fix

// before
node render-runner-workflow-evalbook.mjs ...
// after
PAPERCLIP_EVALBOOK_PROGRAM=/srv/checkouts/paperclip-evals/evals/paperclip-runner/tools/eval_program.py \
  node render-runner-workflow-evalbook.mjs ...

const program = process.env.PAPERCLIP_EVALBOOK_PROGRAM; if (!program || !existsSync(program)) throw new Error('Set PAPERCLIP_EVALBOOK_PROGRAM to the eval generator path');

null

try { await renderRunnerWorkflowEvalbook(...); } catch (e) { if (e.message.includes('Canonical Evalbook generator not found')) console.error('Clone paperclip-evals or set PAPERCLIP_EVALBOOK_PROGRAM'); throw e; }

When it happens

Trigger: Running render-runner-workflow-evalbook.mjs in a fresh clone or CI container that only contains the paperclip monorepo; PAPERCLIP_EVALBOOK_PROGRAM unset and no ../../../../paperclip-evals directory present.

Common situations: Contributors cloning only the main repo; CI runners missing the sibling checkout step; moving the repo so the relative ../../.. path no longer resolves to the evals checkout.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• Managed installs require Node.js ${MINIMUM_NODE_VERSION} or
• ANTHROPIC_API_KEY is required in the CLI process environment
• Probe found no matching Anthropic Environment
• Environment driver "${driver.driver}" does not support duple
• Managed worktree seed registration is incomplete; project wo

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/7a5998333c31ca4f. Report an issue: GitHub.