ruvnet/ruflo · error · Error

frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})

Error message

frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})

What it means

Thrown by loadFrozenHumanEval() when the frozen human-relevance eval set file (.claude/eval/human-relevance-frozen-v1.json) cannot be located via any candidate path: package-resolve, dist-relative, or src-relative. This file is the anti-overfitting anchor the flywheel must never regress on; downstream/foreign projects must supply their own anchor instead.

Source

Thrown at v3/@claude-flow/cli/src/services/harness-frozen-eval.ts:61

function locate(): string | null {
  const candidates: string[] = [];
  try {
    const req = createRequire(import.meta.url);
    candidates.push(path.join(path.dirname(req.resolve('@claude-flow/cli/package.json')), FROZEN_HUMAN_EVAL_FILE));
  } catch { /* not resolvable in this context */ }
  candidates.push(path.resolve(__dirname, '..', '..', '..', FROZEN_HUMAN_EVAL_FILE)); // dist/src/services → pkg root
  candidates.push(path.resolve(__dirname, '..', '..', FROZEN_HUMAN_EVAL_FILE));        // src/services → pkg root
  for (const c of candidates) if (fs.existsSync(c)) return c;
  return null;
}

/**
 * Load + verify the frozen human eval set. Throws if missing or if its content
 * hash != the pinned FROZEN_HUMAN_EVAL_HASH (the "frozen" guarantee).
 */
export function loadFrozenHumanEval(): FrozenHumanEval {
  const p = locate();
  if (!p) throw new Error(`frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})`);
  const parsed = JSON.parse(fs.readFileSync(p, 'utf-8')) as { version?: string; tasks?: HumanEvalTask[] };
  const tasks = parsed.tasks ?? [];
  const corpusHash = humanEvalHash(tasks);
  if (corpusHash !== FROZEN_HUMAN_EVAL_HASH) {
    throw new Error(`frozen human eval hash mismatch — set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit`);
  }
  return { version: parsed.version ?? FROZEN_HUMAN_EVAL_VERSION, tasks, corpusHash };
}

View on GitHub (pinned to 6b01dc5a68)

Solutions

1. For downstream projects: create .claude/eval/flywheel-anchor.manifest.json pointing at your own labelled tasks (see harness-project-anchor.ts).
2. Reinstall the package fully: npm install @claude-flow/cli (ensure the .claude/eval asset is present).
3. If running inside the ruflo/claude-flow repo itself, restore the file from git.
4. Set RUFLO_FLYWHEEL_ALLOW_BUILTIN_ANCHOR=1 only if you intentionally want the built-in set in a non-ruflo repo.

Example fix

# downstream project: create a project-local anchor manifest
cat > .claude/eval/flywheel-anchor.manifest.json <<EOF
{
  "schemaVersion": "ruflo.flywheel-anchor-manifest/v1",
  "path": "./eval/my-tasks.json",
  "sha256": "sha256:<hash-of-my-tasks.json>"
}
EOF

import { existsSync } from 'fs';
function frozenEvalAvailable(): boolean {
  // mirror the locate() candidate paths
  return existsSync(require.resolve('@claude-flow/cli/package.json'));
}
if (!frozenEvalAvailable() && !existsSync('.claude/eval/flywheel-anchor.manifest.json')) {
  throw new Error('no eval anchor available — create .claude/eval/flywheel-anchor.manifest.json');
}

try {
  loadFrozenHumanEval();
} catch (e) {
  if (e instanceof Error && /frozen human eval set not found/.test(e.message)) {
    // fall back to project-local anchor instead
    loadEffectiveFlywheelAnchor(process.cwd());
  } else throw e;
}

When it happens

Trigger: Calling loadFrozenHumanEval() outside the @claude-flow/cli package (e.g. a downstream repo) without a project-local anchor; a partial/corrupted package install where the eval file is missing; running from a stripped dist build that excluded the .claude/eval assets.

Common situations: A downstream repository running the flywheel without creating .claude/eval/flywheel-anchor.manifest.json; an npm install --omit=optional or a bundler that tree-shook the eval file; running from source after the file was git-ignored or deleted.

Related errors

• trajectory envelope not found: ${path}
• frozen human eval hash mismatch — set has drifted (got ${cor
• flywheel anchor path must stay inside project root
• flywheel anchor symlink escapes project root
• Could not find repository root (no package.json found)

AI-assisted analysis of ruvnet/ruflo@6b01dc5a68 (2026-08-12). Data as JSON: /api/errors/8aaa5e6c9bb3b5f3. Report an issue: GitHub.