JuliusBrussee/caveman · error · Error

agent conformance fixtures must cover claude and pi

Error message

agent conformance fixtures must cover claude and pi

What it means

The build-time validator in packages/shared/contracts/scripts/validate-schemas.mjs loads exactly two agent conformance fixtures (sorted by filename) and destructures them as [claude, pi]. It requires the first fixture's `harness` field to be "claude" and the second to be "pi"; otherwise the script throws this error. This guarantees the static contract fixtures actually cover both adapter harnesses before cross-checking their shared fields.

Solutions

1. Open packages/shared/contracts/fixtures/agent/*.json and set `harness` to "claude" in claude-conformance.json and "pi" in pi-conformance.json.
2. If you added/renamed fixture files, ensure exactly two .json files exist and their names sort as claude-conformance.json then pi-conformance.json.
3. If a new harness is genuinely required, update validate-schemas.mjs (and adapter-conformance.schema.json enum) to model more than two harnesses instead of relying on the [claude, pi] destructure.
4. Re-run the validation script (contracts build/test) and confirm the fixtures also pass adapter-conformance.schema.json, whose `harness` enum must include the value you set.

Example fix

// before (packages/shared/contracts/fixtures/agent/pi-conformance.json)
"harness": "p1"

// after
"harness": "pi"

import { readdir, readFile } from "node:fs/promises";
const files = (await readdir("packages/shared/contracts/fixtures/agent")).filter(f => f.endsWith(".json")).sort();
const harnesses = await Promise.all(files.map(async f => JSON.parse(await readFile(`packages/shared/contracts/fixtures/agent/${f}`, "utf8")).harness));
if (JSON.stringify(harnesses) !== JSON.stringify(["claude", "pi"])) {
  throw new Error(`fixture order/harness wrong: ${files} -> ${harnesses}`);
}

function isConformanceFixture(v) {
  return typeof v === "object" && v !== null && (v.harness === "claude" || v.harness === "pi");
}

try {
  await runValidationScript();
} catch (err) {
  if (err.message === "agent conformance fixtures must cover claude and pi") {
    console.error("Check fixtures/agent/*.json: harness must be 'claude' then 'pi' in sorted order.");
    process.exit(1);
  }
  throw err;
}

When it happens

Trigger: Running the contracts build/validate script when: (1) one of the two fixture files in packages/shared/contracts/fixtures/agent/ has a `harness` value other than "claude"/"pi" (e.g. a typo like "p1" or a placeholder), (2) a new harness fixture is added or one is removed so sorted order puts the wrong file first (fixtures.length !== 2 would throw first, but renaming a file while keeping two files changes the [claude, pi] assignment), or (3) a fixture is regenerated with an omitted or renamed `harness` key (the schema must still have passed, so likely an added optional-key path or a schema edit).

Common situations: Regenerating fixtures for a new adapter version and accidentally setting harness to the adapter package name; adding a third fixture file named so it sorts before claude-conformance.json (pairing with the adjacent fixture-count error); hand-editing a fixture while debugging schema changes and leaving `harness` as "" or "test"; copying pi-conformance.json as a template for a new claude fixture and forgetting to change `harness` back.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• middleware
• adapter-specific build_sha256 values must differ
• agent conformance mismatch
• unknown adapter failure must preserve original bytes
• cannot safely launch non-Node Windows command shim

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/f71c07e7f450dbce. Report an issue: GitHub.

Appendix: source

Thrown at packages/shared/contracts/scripts/validate-schemas.mjs:62

const fixtures = await Promise.all(
  fixtureFiles.map(async (file) => JSON.parse(await readFile(path.join(agentFixtureRoot, file), "utf8"))),
);
const validateAdapterConformance = ajv.getSchema(
  "https://caveman.so/schemas/adapter-conformance.schema.json",
);
if (!validateAdapterConformance) {
  throw new Error("adapter conformance schema failed to compile");
}
for (const [index, fixture] of fixtures.entries()) {
  if (!validateAdapterConformance(fixture)) {
    throw new Error(
      `${fixtureFiles[index]}: ${ajv.errorsText(validateAdapterConformance.errors)}`,
    );
  }
}
const [claude, pi] = fixtures;
if (claude.harness !== "claude" || pi.harness !== "pi") {
  throw new Error("agent conformance fixtures must cover claude and pi");
}

for (const key of [
  "normalized_context_digest",
  "plan_sha256",
  "ordered_transform_ids",
  "provider_visible_digest",
  "recovery_handles",
  "accounting_method",
  "failure_fallback",
]) {
  if (JSON.stringify(claude[key]) !== JSON.stringify(pi[key])) {
    throw new Error(`agent conformance mismatch: ${key}`);
  }
}
if (claude.build_sha256 === pi.build_sha256) {
  throw new Error("adapter-specific build_sha256 values must differ");
}

View on GitHub (pinned to 3ee70a1026)