{"record":{"id":"eecb0b56fa9ae853","repo":"vercel/next.js","slug":"next-eval-tarball-not-set-run-evals-via-pnpm-eva","errorCode":null,"errorMessage":"NEXT_EVAL_TARBALL not set. Run evals via `pnpm eval` from the repo root.","messagePattern":"NEXT_EVAL_TARBALL not set\\. Run evals via `pnpm eval` from the repo root\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"evals/lib/setup.ts","lineNumber":15,"sourceCode":"import { readFileSync } from 'node:fs'\nimport type { Sandbox } from '@vercel/agent-eval'\n\n/**\n * Install the locally-built Next.js into the sandbox.\n *\n * The tarball path comes from run-evals.js via NEXT_EVAL_TARBALL, the same\n * env-var handoff that run-tests.js uses for NEXT_TEST_PKG_PATHS. We hard-fail\n * if it's missing rather than falling back to npm — silently testing the\n * published canary instead of your local build defeats the point.\n */\nexport async function installNextJs(sandbox: Sandbox): Promise<void> {\n  const tarball = process.env.NEXT_EVAL_TARBALL\n  if (!tarball) {\n    throw new Error(\n      'NEXT_EVAL_TARBALL not set. Run evals via `pnpm eval` from the repo root.'\n    )\n  }\n  await sandbox.writeFiles({\n    // @ts-expect-error — upstream types writeFiles as Record<string, string>\n    // but the runtime accepts Buffer. Tarballs are binary; can't send as string.\n    'next.tgz': readFileSync(tarball),\n  })\n  const { exitCode, stderr } = await sandbox.runCommand('npm', [\n    'install',\n    './next.tgz',\n  ])\n  if (exitCode !== 0) {\n    throw new Error(\n      `npm install ./next.tgz failed (exit ${exitCode}):\\n${stderr}`\n    )\n  }\n}","sourceCodeStart":1,"sourceCodeEnd":33,"githubUrl":"https://github.com/vercel/next.js/blob/0ae8c72462952df163f1b1e0726641bc5b40dc93/evals/lib/setup.ts#L1-L33","documentation":"Thrown by the eval harness setup (`evals/lib/setup.ts`) when installing the locally-built Next.js into an evaluation sandbox. The path to the locally-built `next.tgz` is passed via the `NEXT_EVAL_TARBALL` env var by the `pnpm eval` runner; the setup intentionally hard-fails when it is unset rather than silently falling back to the published canary, which would test the wrong build and defeat the purpose of running evals.","triggerScenarios":"Running the eval setup directly (e.g. importing `installNextJs` from `evals/lib/setup.ts` or invoking the eval script by hand) without going through `pnpm eval`; or `pnpm eval` failing to export NEXT_EVAL_TARBALL because the tarball build step was skipped or failed.","commonSituations":"Trying to run a single eval file with `node`/`tsx` instead of the `pnpm eval` wrapper; CI missing the build-then-eval sequence; local shell where the env var was set in a different session.","solutions":["Run evals through the repo-root wrapper: `pnpm eval` — it builds the tarball and sets NEXT_EVAL_TARBALL automatically.","If invoking setup programmatically, build the tarball first (`pnpm --filter=next build` then pack) and `export NEXT_EVAL_TARBALL=/abs/path/next.tgz` before running.","Confirm the env var is exported in the same process that runs the eval (print `process.env.NEXT_EVAL_TARBALL` to verify).","Ensure the build step actually produced the tarball; rebuild if it is missing."],"exampleFix":"# before (fails)\nnode evals/run.ts\n\n# after\npnpm eval  # builds next.tgz and sets NEXT_EVAL_TARBALL","handlingStrategy":"validation","validationCode":"// Always enter evals via the wrapper. If scripting manually:\nconst tarball = process.env.NEXT_EVAL_TARBALL\nif (!tarball || !require('fs').existsSync(tarball)) {\n  throw new Error('Build the next tarball first: pnpm --filter=next build, then run via `pnpm eval`.')\n}","typeGuard":"function isTarballPath(s: string | undefined): s is string {\n  return typeof s === 'string' && s.endsWith('.tgz')\n}","tryCatchPattern":"try {\n  await installNextJs(sandbox)\n} catch (e) {\n  if (/NEXT_EVAL_TARBALL not set/.test(e.message)) {\n    console.error('Run via `pnpm eval` from repo root.')\n  }\n  throw e\n}","preventionTips":["Always invoke evals through `pnpm eval`.","Confirm the tarball build step ran before eval in CI.","Document NEXT_EVAL_TARBALL in the eval README."],"tags":["evals","tooling","environment","ci","next-build"],"analyzedSha":"0ae8c72462952df163f1b1e0726641bc5b40dc93","analyzedAt":"2026-08-06T19:44:29.143Z","schemaVersion":2},"datasetVersion":"2026-08-07T02:17:10.218Z"}