JuliusBrussee/caveman · error · Error

${fixtureFiles[index]}…

Error message

${fixtureFiles[index]}: ${ajv.errorsText(validateAdapterConformance.errors)}

What it means

When a harness fixture fails the adapter-conformance schema, the script throws with the fixture filename plus AJV's errorsText, pinpointing every schema violation (missing required harness fields, wrong types, invalid enum values). This is the fixture-level conformance gate before the script also asserts claude.harness === "claude" and pi.harness === "pi".

Solutions

1. Read the errorsText after the filename in the message to find each violating dataPath and keyword
2. Update the named fixture to satisfy the schema (add required fields, fix types/enums)
3. Regenerate the fixture from the harness capture tooling if the transcript format changed
4. If the contract intentionally changed, update the schema and both fixtures in the same commit

Example fix

// before (claude.json)
{ "harness": "Claude", "steps": [] }
// after
{ "harness": "claude", "steps": [{ "type": "tool_call", "name": "read_file" }] }

const validate = ajv.getSchema('https://caveman.so/schemas/adapter-conformance.schema.json');
for (const f of fixtures) if (!validate(f)) console.error(validate.errors.map(e => `${e.instancePath} ${e.message}`).join('\n'));

const looksConformant = (f) => typeof f?.harness === 'string' && Array.isArray(f?.steps);

try { validateFixtures(fixtures); } catch (e) { if (/\.json: /.test(e.message)) { console.error('Conformance violations:', e.message); process.exitCode = 1; } else throw e; }

When it happens

Trigger: Running scripts/validate-schemas.mjs when validateAdapterConformance(fixture) returns false for any fixture — the fixture's structure or content violates the adapter-conformance schema (bad/missing harness id, malformed step records, wrong types).

Common situations: A harness's recorded transcript format changed and fixtures were regenerated in a new shape; hand-edited fixture introduced a typo; schema gained a new required field not yet present in older fixtures.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• canonical span
• report fixture : theme transition disagrees with its…
• cave_agent_definition_invalid
• cave_context_ir_invalid
• cave_output_schema_mismatch

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/d12a230665131366. Report an issue: GitHub.

Appendix: source

Thrown at packages/shared/contracts/scripts/validate-schemas.mjs:55

  .filter((name) => name.endsWith(".json"))
  .sort();

if (fixtureFiles.length !== 2) {
  throw new Error(`expected exactly 2 agent conformance fixtures, found ${fixtureFiles.length}`);
}

const fixtures = await Promise.all(
  fixtureFiles.map(async (file) => JSON.parse(await readFile(path.join(agentFixtureRoot, file), "utf8"))),
);
const validateAdapterConformance = ajv.getSchema(
  "https://caveman.so/schemas/adapter-conformance.schema.json",
);
if (!validateAdapterConformance) {
  throw new Error("adapter conformance schema failed to compile");
}
for (const [index, fixture] of fixtures.entries()) {
  if (!validateAdapterConformance(fixture)) {
    throw new Error(
      `${fixtureFiles[index]}: ${ajv.errorsText(validateAdapterConformance.errors)}`,
    );
  }
}
const [claude, pi] = fixtures;
if (claude.harness !== "claude" || pi.harness !== "pi") {
  throw new Error("agent conformance fixtures must cover claude and pi");
}

for (const key of [
  "normalized_context_digest",
  "plan_sha256",
  "ordered_transform_ids",
  "provider_visible_digest",
  "recovery_handles",
  "accounting_method",
  "failure_fallback",
]) {

View on GitHub (pinned to 3ee70a1026)