ruvnet/ruflo · error

frozen human eval hash mismatch — set has drifted

Error message

frozen human eval hash mismatch — set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit

What it means

The 'frozen' guarantee of ADR-176: the loaded task corpus must hash exactly to the pinned constant `FROZEN_HUMAN_EVAL_HASH` (sha256:6096e48e…). `humanEvalHash` canonicalizes and sorts tasks by id, so the pin is content-based, not byte-based. Any drift between the JSON contents and the pin aborts loading with got/pinned values — by design the set can only change by shipping a new versioned file, never by editing this one.

Solutions

1. Restore the original file: reinstall the package or `git checkout` the vendored file so the corpus matches the pin again
2. If you genuinely need different tasks, supersede — ship a NEW versioned eval file (e.g. human-relevance-frozen-v2.json) with an updated pin constant; never edit the v1 file in place, exactly as the message instructs
3. If nobody edited anything, verify install integrity (`npm cache verify`, clean reinstall) — silent corruption also trips this

// If you ship the frozen set yourself, verify it in CI with the exported helpers:
import { readFileSync } from 'node:fs';
import { FROZEN_HUMAN_EVAL_HASH, humanEvalHash } from '@claude-flow/cli/dist/services/harness-frozen-eval.js';

const tasks = (JSON.parse(readFileSync(file, 'utf8'))).tasks ?? [];
if (humanEvalHash(tasks) !== FROZEN_HUMAN_EVAL_HASH) {
  throw new Error('frozen eval corpus drifted from its pin — restore the file or supersede with a new version');
}

try {
  const frozen = loadFrozenHumanEval();
} catch (e) {
  if (e instanceof Error && e.message.includes('frozen human eval hash mismatch')) {
    // Do NOT regenerate or edit the file to make this pass.
    // Either restore the pinned original (reinstall / git checkout) or,
    // if the change is intentional upstream, upgrade the package so the pin
    // constant and the file move together.
    throw new Error(`Frozen eval integrity failure — possible tampering or version skew: ${e.message}`);
  }
  throw e;
}

When it happens

Trigger: Someone edited `.claude/eval/human-relevance-frozen-v1.json` (added, removed, reworded, or relabelled tasks); a version-skewed install pairing an old pin constant with new contents or vice versa; a corrupted or partially applied package update.

Common situations: Teams hand-tuning eval tasks to make flywheel numbers look better; downstream forks patching the corpus; npm cache corruption; mixing files across releases during upgrade.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• KV cache integrity check failed: hash mismatch
• project flywheel anchor hash mismatch
• frozen human eval set not found
• must be a canonical sha256 digest
• RVFP header magic mismatch

AI-assisted analysis of ruvnet/ruflo@fa13ee4ad6 (2026-08-18). Data as JSON: /api/errors/84d11012ca69aa45. Report an issue: GitHub.

Appendix: source

Thrown at v3/@claude-flow/cli/src/services/harness-frozen-eval.ts:66

  } catch { /* not resolvable in this context */ }
  candidates.push(path.resolve(__dirname, '..', '..', '..', FROZEN_HUMAN_EVAL_FILE)); // dist/src/services → pkg root
  candidates.push(path.resolve(__dirname, '..', '..', FROZEN_HUMAN_EVAL_FILE));        // src/services → pkg root
  for (const c of candidates) if (fs.existsSync(c)) return c;
  return null;
}

/**
 * Load + verify the frozen human eval set. Throws if missing or if its content
 * hash != the pinned FROZEN_HUMAN_EVAL_HASH (the "frozen" guarantee).
 */
export function loadFrozenHumanEval(): FrozenHumanEval {
  const p = locate();
  if (!p) throw new Error(`frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})`);
  const parsed = JSON.parse(fs.readFileSync(p, 'utf-8')) as { version?: string; tasks?: HumanEvalTask[] };
  const tasks = parsed.tasks ?? [];
  const corpusHash = humanEvalHash(tasks);
  if (corpusHash !== FROZEN_HUMAN_EVAL_HASH) {
    throw new Error(`frozen human eval hash mismatch — set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit`);
  }
  return { version: parsed.version ?? FROZEN_HUMAN_EVAL_VERSION, tasks, corpusHash };
}

View on GitHub (pinned to fa13ee4ad6)