JuliusBrussee/caveman · error · Error

agent conformance mismatch

Error message

agent conformance mismatch: ${key}

What it means

After confirming one claude and one pi fixture, the validator compares seven shared contract fields (normalized_context_digest, plan_sha256, ordered_transform_ids, provider_visible_digest, recovery_handles, accounting_method, failure_fallback) between the two fixtures using JSON.stringify equality. If any of these differ, it throws `agent conformance mismatch: <key>`, proving the two adapter implementations no longer agree on the static transform contract. The fixture contract records the digests/plan both adapters must produce from the same normalized context.

Solutions

  1. Read the error's <key> suffix to see exactly which field diverged, then diff that field between claude-conformance.json and pi-conformance.json.
  2. Recompute the shared value (digest, plan hash, transform id list) for BOTH adapters after the underlying change and regenerate both fixtures together.
  3. If one adapter intentionally diverges, reconcile the contract: either fix the adapter, or move the divergent value out of the shared seven-key list and update validate-schemas.mjs accordingly (requires coordinating the wire contract).
  4. Ensure both fixture files were updated in the same commit/PR so they can't drift again.

Example fix

// before (pi-conformance.json)
"ordered_transform_ids": ["caveman.cache-preserve.v1", "caveman.tool-search.v1"]

// after (match claude-conformance.json)
"ordered_transform_ids": ["caveman.cache-preserve.v1", "caveman.tool-search.v1", "caveman.ccr-results.v1"]
Defensive patterns

Strategy: validation

Validate before calling

const keys = ["normalized_context_digest","plan_sha256","ordered_transform_ids","provider_visible_digest","recovery_handles","accounting_method","failure_fallback"];
const [a, b] = fixtures;
const mismatches = keys.filter(k => JSON.stringify(a[k]) !== JSON.stringify(b[k]));
if (mismatches.length) throw new Error(`fixtures diverge before commit: ${mismatches.join(",")}`);

Type guard

function sharesContract(a, b) {
  return ["normalized_context_digest","plan_sha256"].every(k => a[k] === b[k]);
}

Try / catch

try {
  await runValidationScript();
} catch (err) {
  const m = err.message.match(/agent conformance mismatch: (\w+)/);
  if (m) {
    console.error(`Field ${m[1]} differs between fixtures; regenerate both adapters' fixtures from the current pipeline.`);
    process.exit(1);
  }
  throw err;
}

Prevention

When it happens

Trigger: Running the contracts validation script when a fixture's digest, plan_sha256, transform id list, recovery handles, accounting method, or failure_fallback was updated for one harness but not the other; regenerating digests after a transform change in only one codebase; hand-editing ordered_transform_ids to add a new transform in claude-conformance.json only; changing accounting_method or failure_fallback semantics for one adapter.

Common situations: A transform pipeline change (e.g. adding caveman.foo.v1) where one team regenerated their fixture and the other didn't; a merge that took one fixture from a branch but not its counterpart; intentional divergence for one adapter that needs the field moved to an adapter-specific section first; stale fixtures after bumping adapter_version without recomputing digests.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/b6f7b75d43a07092. Report an issue: GitHub.

Appendix: source

Thrown at packages/shared/contracts/scripts/validate-schemas.mjs:75

    );
  }
}
const [claude, pi] = fixtures;
if (claude.harness !== "claude" || pi.harness !== "pi") {
  throw new Error("agent conformance fixtures must cover claude and pi");
}

for (const key of [
  "normalized_context_digest",
  "plan_sha256",
  "ordered_transform_ids",
  "provider_visible_digest",
  "recovery_handles",
  "accounting_method",
  "failure_fallback",
]) {
  if (JSON.stringify(claude[key]) !== JSON.stringify(pi[key])) {
    throw new Error(`agent conformance mismatch: ${key}`);
  }
}
if (claude.build_sha256 === pi.build_sha256) {
  throw new Error("adapter-specific build_sha256 values must differ");
}
if (claude.failure_fallback !== "original") {
  throw new Error("unknown adapter failure must preserve original bytes");
}

const middleware = JSON.parse(await readFile(path.join(packageRoot, "..", "..", "sdk", "parity", "middleware.fixtures.json"), "utf8"));
for (const [field, schemaName] of Object.entries({ capabilities: "capabilities", request: "optimize", plan: "plan", page: "page" })) {
  const validate = ajv.getSchema(`https://caveman.so/schemas/middleware-${schemaName}.schema.json`);
  if (!validate?.(middleware[field])) throw new Error(`middleware ${field}: ${ajv.errorsText(validate?.errors)}`);
}

console.log(
  `validated ${files.length} JSON schemas and ${fixtureFiles.length} static agent contract fixtures (not executable parity)`,
);

View on GitHub (pinned to 3ee70a1026)