ruvnet/ruflo · critical · Error

frozen human eval hash mismatch — set has drifted (got ${cor

Error message

frozen human eval hash mismatch — set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit

What it means

Thrown by loadFrozenHumanEval() when the frozen eval file's content hash does not equal the pinned constant FROZEN_HUMAN_EVAL_HASH. This is the tamper-evidence guarantee: the set cannot silently drift. ANY edit (even reordering keys, whitespace that survives canonicalization, or swapping a task) trips it. The message explicitly says to supersede with a new versioned file, not to edit in place.

Source

Thrown at v3/@claude-flow/cli/src/services/harness-frozen-eval.ts:66

  } catch { /* not resolvable in this context */ }
  candidates.push(path.resolve(__dirname, '..', '..', '..', FROZEN_HUMAN_EVAL_FILE)); // dist/src/services → pkg root
  candidates.push(path.resolve(__dirname, '..', '..', FROZEN_HUMAN_EVAL_FILE));        // src/services → pkg root
  for (const c of candidates) if (fs.existsSync(c)) return c;
  return null;
}

/**
 * Load + verify the frozen human eval set. Throws if missing or if its content
 * hash != the pinned FROZEN_HUMAN_EVAL_HASH (the "frozen" guarantee).
 */
export function loadFrozenHumanEval(): FrozenHumanEval {
  const p = locate();
  if (!p) throw new Error(`frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})`);
  const parsed = JSON.parse(fs.readFileSync(p, 'utf-8')) as { version?: string; tasks?: HumanEvalTask[] };
  const tasks = parsed.tasks ?? [];
  const corpusHash = humanEvalHash(tasks);
  if (corpusHash !== FROZEN_HUMAN_EVAL_HASH) {
    throw new Error(`frozen human eval hash mismatch — set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit`);
  }
  return { version: parsed.version ?? FROZEN_HUMAN_EVAL_VERSION, tasks, corpusHash };
}

View on GitHub (pinned to 6b01dc5a68)

Solutions

1. DO NOT edit the file — restore it from git or reinstall the exact package version.
2. To change the eval set legitimately, create a NEW versioned file (e.g. human-relevance-frozen-v2.json) and update FROZEN_HUMAN_EVAL_HASH + FROZEN_HUMAN_EVAL_VERSION constants in source.
3. Verify no build step (prettier, eslint --fix) is rewriting the JSON.
4. Confirm package integrity: npm pack --dry-run or compare against the published tarball.

Example fix

# restore the canonical file
git checkout -- .claude/eval/human-relevance-frozen-v1.json
# OR supersede properly: create v2 and update the pinned constants in harness-frozen-eval.ts

import { humanEvalHash, FROZEN_HUMAN_EVAL_HASH } from './harness-frozen-eval.js';
function evalFileIntact(path: string): boolean {
  try {
    const { tasks } = JSON.parse(fs.readFileSync(path, 'utf8'));
    return humanEvalHash(tasks) === FROZEN_HUMAN_EVAL_HASH;
  } catch { return false; }
}

try {
  loadFrozenHumanEval();
} catch (e) {
  if (e instanceof Error && /hash mismatch/.test(e.message)) {
    throw new Error('FATAL: frozen eval tamper detected — restore from git or supersede with a new versioned file');
  }
  throw e;
}

When it happens

Trigger: The file .claude/eval/human-relevance-frozen-v1.json was edited, reformatted, partially overwritten, or replaced with a different version. The hash is computed over canon-sorted tasks, so structural or content changes both trigger it.

Common situations: A developer hand-edited the eval file to add a task; a formatter/linter rewrote the JSON; a package downgrade/upgrade shipped a different eval version; a merge conflict resolution altered content; an actual tampering attempt.

Related errors

• sha256 mismatch for ${input.assetFilename}: expected ${expec
• frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})
• flywheel anchor path must stay inside project root
• flywheel anchor symlink escapes project root
• SSRF guard: invalid URL — ${rawUrl}

AI-assisted analysis of ruvnet/ruflo@6b01dc5a68 (2026-08-12). Data as JSON: /api/errors/84d11012ca69aa45. Report an issue: GitHub.