JuliusBrussee/caveman · error · Error

${where} replay trial ${proof.id} replays a case outside the

Error message

${where} replay trial ${proof.id} replays a case outside the composed dataset

What it means

Error "${where} replay trial ${proof.id} replays a case outside the composed dataset" thrown in JuliusBrussee/caveman.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:323

    if (roleCounts.boundary > 1) throw new Error(`${where} dataset carries ${roleCounts.boundary} boundary cases, want at most one`);
    if (roleCounts.adversarial > (confidenceGuard ? 2 : 1)) throw new Error(`${where} dataset carries ${roleCounts.adversarial} adversarial cases`);
    if (roleCounts.boundary !== dataset.boundary_cases.length) throw new Error(`${where} boundary case list disagrees with the composed dataset`);
    if (roleCounts.adversarial !== dataset.generated_cases.length) throw new Error(`${where} generated case list disagrees with the composed dataset`);
    const composed = roleCounts.target_failure + roleCounts.prior_success + roleCounts.boundary + roleCounts.adversarial;
    if (composed !== dataset.cases.length) throw new Error(`${where} dataset roles (${composed}) do not account for its ${dataset.cases.length} cases`);
    if (dataset.replay_case_ids.length !== datasetCaseIDs.size) throw new Error(`${where} replay manifest carries ${dataset.replay_case_ids.length} ids for ${datasetCaseIDs.size} dataset cases`);
    for (const datasetCaseID of dataset.replay_case_ids) {
      if (!datasetCaseIDs.has(datasetCaseID)) throw new Error(`${where} replay manifest case ${datasetCaseID} is not a composed dataset case`);
    }

    // The guard grader exists exactly when there is a perturbed case to catch a
    // candidate on; a required grader with no case behind it is decoration.
    const perturbed = dataset.cases.some((entry) => entry.perturbation !== "none");
    if (perturbed !== item.eval_pack.graders.includes("guard_respected")) {
      throw new Error(`${where} guard_respected grader ${perturbed ? "missing for" : "declared without"} perturbed dataset cases`);
    }
    for (const proof of item.replay.trial_proofs) {
      if (!datasetCaseIDs.has(proof.dataset_case_id)) throw new Error(`${where} replay trial ${proof.id} replays a case outside the composed dataset`);
      for (const arm of [proof.baseline, proof.candidate]) {
        const graders = arm.grader_results.map((grader) => grader.grader).sort();
        const required = [...item.eval_pack.graders].sort();
        if (graders.join(",") !== required.join(",")) {
          throw new Error(`${where} replay trial ${proof.id} graded ${graders.join(",")}, want exactly ${required.join(",")}`);
        }
      }
    }
  }
}
if (!Array.isArray(spans) || spans.length === 0) throw new Error("canonical span fixture is empty");
for (const [index, span] of spans.entries()) {
  if (!validateSpan(span)) throw new Error(`canonical span ${index}: ${ajv.errorsText(validateSpan.errors)}`);
}
if (relationshipCount === 0) throw new Error("no theme relationship reached the shared-analysis-unit threshold in any fixture");
if (motifCount === 0) throw new Error("no workflow motif recurred across two variants of one task family in any fixture");
console.log(`validated ${reports.length} generated improvement report(s), ${relationshipCount} theme relationship(s), ${motifCount} workflow motif(s) and ${spans.length} canonical spans`);

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Replay only cases that belong to the composed dataset.

When it happens

Trigger: Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:323 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/bd8c1ce745f989b3. Report an issue: GitHub.