{"record":{"id":"84d11012ca69aa45","repo":"ruvnet/ruflo","slug":"frozen-human-eval-hash-mismatch-set-has-drifted","errorCode":null,"errorMessage":"frozen human eval hash mismatch — set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit","messagePattern":"frozen human eval hash mismatch — set has drifted \\(got (.+?), pinned (.+?)\\); supersede with a new versioned file, do not edit","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"v3/@claude-flow/cli/src/services/harness-frozen-eval.ts","lineNumber":66,"sourceCode":"  } catch { /* not resolvable in this context */ }\n  candidates.push(path.resolve(__dirname, '..', '..', '..', FROZEN_HUMAN_EVAL_FILE)); // dist/src/services → pkg root\n  candidates.push(path.resolve(__dirname, '..', '..', FROZEN_HUMAN_EVAL_FILE));        // src/services → pkg root\n  for (const c of candidates) if (fs.existsSync(c)) return c;\n  return null;\n}\n\n/**\n * Load + verify the frozen human eval set. Throws if missing or if its content\n * hash != the pinned FROZEN_HUMAN_EVAL_HASH (the \"frozen\" guarantee).\n */\nexport function loadFrozenHumanEval(): FrozenHumanEval {\n  const p = locate();\n  if (!p) throw new Error(`frozen human eval set not found (${FROZEN_HUMAN_EVAL_FILE})`);\n  const parsed = JSON.parse(fs.readFileSync(p, 'utf-8')) as { version?: string; tasks?: HumanEvalTask[] };\n  const tasks = parsed.tasks ?? [];\n  const corpusHash = humanEvalHash(tasks);\n  if (corpusHash !== FROZEN_HUMAN_EVAL_HASH) {\n    throw new Error(`frozen human eval hash mismatch — set has drifted (got ${corpusHash}, pinned ${FROZEN_HUMAN_EVAL_HASH}); supersede with a new versioned file, do not edit`);\n  }\n  return { version: parsed.version ?? FROZEN_HUMAN_EVAL_VERSION, tasks, corpusHash };\n}\n","sourceCodeStart":48,"sourceCodeEnd":70,"githubUrl":"https://github.com/ruvnet/ruflo/blob/fa13ee4ad60ac2090b1480656eb233521790d640/v3/@claude-flow/cli/src/services/harness-frozen-eval.ts#L48-L70","documentation":"The 'frozen' guarantee of ADR-176: the loaded task corpus must hash exactly to the pinned constant `FROZEN_HUMAN_EVAL_HASH` (sha256:6096e48e…). `humanEvalHash` canonicalizes and sorts tasks by id, so the pin is content-based, not byte-based. Any drift between the JSON contents and the pin aborts loading with got/pinned values — by design the set can only change by shipping a new versioned file, never by editing this one.","triggerScenarios":"Someone edited `.claude/eval/human-relevance-frozen-v1.json` (added, removed, reworded, or relabelled tasks); a version-skewed install pairing an old pin constant with new contents or vice versa; a corrupted or partially applied package update.","commonSituations":"Teams hand-tuning eval tasks to make flywheel numbers look better; downstream forks patching the corpus; npm cache corruption; mixing files across releases during upgrade.","solutions":["Restore the original file: reinstall the package or `git checkout` the vendored file so the corpus matches the pin again","If you genuinely need different tasks, supersede — ship a NEW versioned eval file (e.g. human-relevance-frozen-v2.json) with an updated pin constant; never edit the v1 file in place, exactly as the message instructs","If nobody edited anything, verify install integrity (`npm cache verify`, clean reinstall) — silent corruption also trips this"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// If you ship the frozen set yourself, verify it in CI with the exported helpers:\nimport { readFileSync } from 'node:fs';\nimport { FROZEN_HUMAN_EVAL_HASH, humanEvalHash } from '@claude-flow/cli/dist/services/harness-frozen-eval.js';\n\nconst tasks = (JSON.parse(readFileSync(file, 'utf8'))).tasks ?? [];\nif (humanEvalHash(tasks) !== FROZEN_HUMAN_EVAL_HASH) {\n  throw new Error('frozen eval corpus drifted from its pin — restore the file or supersede with a new version');\n}","typeGuard":null,"tryCatchPattern":"try {\n  const frozen = loadFrozenHumanEval();\n} catch (e) {\n  if (e instanceof Error && e.message.includes('frozen human eval hash mismatch')) {\n    // Do NOT regenerate or edit the file to make this pass.\n    // Either restore the pinned original (reinstall / git checkout) or,\n    // if the change is intentional upstream, upgrade the package so the pin\n    // constant and the file move together.\n    throw new Error(`Frozen eval integrity failure — possible tampering or version skew: ${e.message}`);\n  }\n  throw e;\n}","preventionTips":["Never edit the frozen eval JSON — the pin exists precisely to catch this","Upgrade the whole package so pin and corpus stay in lockstep","Treat a mismatch with no known edits as possible tampering or corrupted install","If you must change the corpus, ship a new versioned file plus an updated pin (supersede, don't edit)"],"tags":["integrity","checksum","frozen-eval","tamper-detection"],"backgroundTag":"checksum-mismatch","analyzedSha":"fa13ee4ad60ac2090b1480656eb233521790d640","analyzedAt":"2026-08-18T21:34:22.708Z","contentChangedAt":"2026-08-18T21:34:22.708Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}