ruvnet/ruflo · error

frozen human eval set not found

Error message

frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})

What it means

ADR-176 freezes a public human-labeled eval set shipped inside the package at `.claude/eval/human-relevance-frozen-v1.json`; it is the anti-overfitting anchor for the flywheel. `locate()` probes candidate roots (the resolved package root, `__dirname/../../..` for dist, `__dirname/../..` for src) and if none contains the file, `loadFrozenHumanEval` throws — evaluation cannot proceed without the frozen set.

Solutions

1. Reinstall the package (`npm i @claude-flow/cli`) so `.claude/eval/human-relevance-frozen-v1.json` exists under the package root
2. If you bundle or repackage the CLI, copy the `.claude/eval/` directory next to the compiled output so the `__dirname`-relative probes find it
3. Verify the file ships: inspect `npm pack` output or the package.json `files` list
4. Make sure you execute the installed package (its resolved root contains the file), not a moved copy of dist

Example fix

# before: Docker image copies only compiled JS
COPY dist /app/dist

# after: package assets included so locate() probes succeed
COPY package.json /app/
COPY .claude/eval/human-relevance-frozen-v1.json /app/.claude/eval/

import { existsSync } from 'node:fs';
import { createRequire } from 'node:module';
import { fileURLToPath } from 'node:url';
import { dirname, join, resolve } from 'node:path';

const __dirname = dirname(fileURLToPath(import.meta.url));
const REL = join('.claude', 'eval', 'human-relevance-frozen-v1.json');

// Mirrors locate(): true only if some candidate exists.
function frozenEvalAvailable(): boolean {
  const candidates: string[] = [];
  try {
    const req = createRequire(import.meta.url);
    candidates.push(join(dirname(req.resolve('@claude-flow/cli/package.json')), REL));
  } catch { /* unresolvable */ }
  candidates.push(resolve(__dirname, '..', '..', '..', REL));
  candidates.push(resolve(__dirname, '..', '..', REL));
  return candidates.some((c) => existsSync(c));
}

if (!frozenEvalAvailable()) throw new Error('frozen eval asset missing — reinstall the package or fix bundling');

try {
  const frozen = loadFrozenHumanEval();
} catch (e) {
  if (e instanceof Error && e.message.includes('frozen human eval set not found')) {
    // packaging defect, not runtime state: fail with actionable guidance
    throw new Error('Install incomplete: .claude/eval/human-relevance-frozen-v1.json is missing from the package. Reinstall @claude-flow/cli or copy the asset next to the build output.');
  }
  throw e;
}

When it happens

Trigger: Running a code path that loads the frozen set (e.g. the built-in anchor selection for the ruflo repository itself, or human-relevance deltas in receipts) from an install where the JSON is absent: bundlers that inline dist without sibling assets, Docker images copying only compiled JS, `files` allowlists in package.json omitting `.claude/eval`, or a pruned node_modules.

Common situations: Single-file bundles (esbuild/webpack) that drop non-JS assets; minimal container images; downstream forks repackaging the CLI; broken mirrors or partial installs; running a relocated copy of `dist/` without the package root.

Related errors

• archive did not contain the expected binary at its root
• build evidence path is not a file or symlink
• build input bytes must be a non-negative safe integer
• build input escapes repository
• case-fold collision in declared build inputs

AI-assisted analysis of ruvnet/ruflo@fa13ee4ad6 (2026-08-18). Data as JSON: /api/errors/8aaa5e6c9bb3b5f3. Report an issue: GitHub.

Appendix: source

Thrown at v3/@claude-flow/cli/src/services/harness-frozen-eval.ts:61

function locate(): string | null {
  const candidates: string[] = [];
  try {
    const req = createRequire(import.meta.url);
    candidates.push(path.join(path.dirname(req.resolve('@claude-flow/cli/package.json')), FROZEN_HUMAN_EVAL_FILE));
  } catch { /* not resolvable in this context */ }
  candidates.push(path.resolve(__dirname, '..', '..', '..', FROZEN_HUMAN_EVAL_FILE)); // dist/src/services → pkg root
  candidates.push(path.resolve(__dirname, '..', '..', FROZEN_HUMAN_EVAL_FILE));        // src/services → pkg root
  for (const c of candidates) if (fs.existsSync(c)) return c;
  return null;
}

/**
 * Load + verify the frozen human eval set. Throws if missing or if its content
 * hash != the pinned FROZEN_HUMAN_EVAL_HASH (the "frozen" guarantee).
 */
export function loadFrozenHumanEval(): FrozenHumanEval {
  const p = locate();
  if (!p) throw new Error(`frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})`);
  const parsed = JSON.parse(fs.readFileSync(p, 'utf-8')) as { version?: string; tasks?: HumanEvalTask[] };
  const tasks = parsed.tasks ?? [];
  const corpusHash = humanEvalHash(tasks);
  if (corpusHash !== FROZEN_HUMAN_EVAL_HASH) {
    throw new Error(`frozen human eval hash mismatch — set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit`);
  }
  return { version: parsed.version ?? FROZEN_HUMAN_EVAL_VERSION, tasks, corpusHash };
}

View on GitHub (pinned to fa13ee4ad6)