NEXT_EVAL_TARBALL not set. Run evals via `pnpm eval` from th

Error message

NEXT_EVAL_TARBALL not set. Run evals via `pnpm eval` from the repo root.

What it means

Thrown by the eval harness setup (`evals/lib/setup.ts`) when installing the locally-built Next.js into an evaluation sandbox. The path to the locally-built `next.tgz` is passed via the `NEXT_EVAL_TARBALL` env var by the `pnpm eval` runner; the setup intentionally hard-fails when it is unset rather than silently falling back to the published canary, which would test the wrong build and defeat the purpose of running evals.

Source

Thrown at evals/lib/setup.ts:15

import { readFileSync } from 'node:fs'
import type { Sandbox } from '@vercel/agent-eval'

/**
 * Install the locally-built Next.js into the sandbox.
 *
 * The tarball path comes from run-evals.js via NEXT_EVAL_TARBALL, the same
 * env-var handoff that run-tests.js uses for NEXT_TEST_PKG_PATHS. We hard-fail
 * if it's missing rather than falling back to npm — silently testing the
 * published canary instead of your local build defeats the point.
 */
export async function installNextJs(sandbox: Sandbox): Promise<void> {
  const tarball = process.env.NEXT_EVAL_TARBALL
  if (!tarball) {
    throw new Error(
      'NEXT_EVAL_TARBALL not set. Run evals via `pnpm eval` from the repo root.'
    )
  }
  await sandbox.writeFiles({
    // @ts-expect-error — upstream types writeFiles as Record<string, string>
    // but the runtime accepts Buffer. Tarballs are binary; can't send as string.
    'next.tgz': readFileSync(tarball),
  })
  const { exitCode, stderr } = await sandbox.runCommand('npm', [
    'install',
    './next.tgz',
  ])
  if (exitCode !== 0) {
    throw new Error(
      `npm install ./next.tgz failed (exit ${exitCode}):\n${stderr}`
    )
  }
}

View on GitHub (pinned to 0ae8c72462)

Solutions

Run evals through the repo-root wrapper: `pnpm eval` — it builds the tarball and sets NEXT_EVAL_TARBALL automatically.
If invoking setup programmatically, build the tarball first (`pnpm --filter=next build` then pack) and `export NEXT_EVAL_TARBALL=/abs/path/next.tgz` before running.
Confirm the env var is exported in the same process that runs the eval (print `process.env.NEXT_EVAL_TARBALL` to verify).
Ensure the build step actually produced the tarball; rebuild if it is missing.

Example fix

# before (fails)
node evals/run.ts

# after
pnpm eval  # builds next.tgz and sets NEXT_EVAL_TARBALL

Defensive patterns

Strategy: validation

Validate before calling

// Always enter evals via the wrapper. If scripting manually:
const tarball = process.env.NEXT_EVAL_TARBALL
if (!tarball || !require('fs').existsSync(tarball)) {
  throw new Error('Build the next tarball first: pnpm --filter=next build, then run via `pnpm eval`.')
}

Type guard

function isTarballPath(s: string | undefined): s is string {
  return typeof s === 'string' && s.endsWith('.tgz')
}

Try / catch

try {
  await installNextJs(sandbox)
} catch (e) {
  if (/NEXT_EVAL_TARBALL not set/.test(e.message)) {
    console.error('Run via `pnpm eval` from repo root.')
  }
  throw e
}

Prevention

Always invoke evals through `pnpm eval`.
Confirm the tarball build step ran before eval in CI.
Document NEXT_EVAL_TARBALL in the eval README.

When it happens

Trigger: Running the eval setup directly (e.g. importing `installNextJs` from `evals/lib/setup.ts` or invoking the eval script by hand) without going through `pnpm eval`; or `pnpm eval` failing to export NEXT_EVAL_TARBALL because the tarball build step was skipped or failed.

Common situations: Trying to run a single eval file with `node`/`tsx` instead of the `pnpm eval` wrapper; CI missing the build-then-eval sequence; local shell where the env var was set in a different session.

Related errors

AI-assisted analysis of vercel/next.js@0ae8c72462 (2026-08-06). Data as JSON: /api/errors/eecb0b56fa9ae853. Report an issue: GitHub.