JuliusBrussee/caveman · error

caveman agent: error-rate guardrail max must be in [0,1]

Error message

caveman agent: error-rate guardrail max must be in [0,1]

What it means

Thrown by evalFixture() when an 'error_rate' guardrail's max is not a finite number in [0, 1]. The guardrail bounds the tolerated failure fraction, so negatives, values above 1, NaN, Infinity, and percent-style values like 5 (meaning 5%) are all invalid at construction.

Source

Thrown at packages/agent/src/primitives.ts:458

  quality: QualityGrader[];
  guardrails?: EvalGuardrail[];
}): EvalDefinition {
  if (options.id.trim() === "") throw new Error("caveman agent: eval id is required");
  if (options.quality.length === 0) throw new Error("caveman agent: eval needs at least one quality grader");
  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };
  if (tools.mode === "live" && !tools.sandbox) {
    throw new Error("caveman agent: live eval tools require an explicit sandbox");
  }
  return Object.freeze({
    kind: "eval",
    id: options.id,
    approved: options.approved ?? false,
    required: options.required ?? true,
    input: options.input,
    tools,
    quality: Object.freeze([...options.quality]),
    guardrails: Object.freeze([...(options.guardrails ?? [])]),
  });
}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Express the cap as a fraction: 0.02 for 2%, 0 for zero tolerance, 1 for allowing everything
2. Convert percentages at the config boundary: max: percent / 100
3. Validate with Number.isFinite(v) && v >= 0 && v <= 1 in your fixture loader

Example fix

// before (SLO doc says 'max error rate 2%')
evalFixture({ id: 'e1', input: q, quality: q1, guardrails: [{ type: 'error_rate', max: 2 }] });

// after
evalFixture({ id: 'e1', input: q, quality: q1, guardrails: [{ type: 'error_rate', max: 0.02 }] });

function toErrorRateMax(raw: unknown): number {
  const n = typeof raw === 'number' ? raw : Number(raw);
  if (!Number.isFinite(n) || n < 0 || n > 1) throw new Error(`error_rate max must be a fraction in [0,1], got ${String(raw)} (use 0.02 for 2%)`);
  return n;
}

function isErrorRateMax(value: unknown): value is number { return typeof value === 'number' && Number.isFinite(value) && value >= 0 && value <= 1; }

When it happens

Trigger: Passing guardrails: [{ type: 'error_rate', max: 5 }] (percent, not fraction), max: -0.1, max: 1.5, max: NaN, or a string '0.02' from config.

Common situations: Config authored in percent ('maxErrorRate: 2' meaning 2%) instead of the fraction 0.02; SLO spreadsheets exporting percentages; forgetting to divide by 100 when converting an external SLO.

Related errors

• caveman agent: latency guardrail requires positive integer p
• cave_live_eval_sandbox_profile_missing
• cave_live_eval_sandbox_profile_invalid
• cave_compaction_option_invalid
• caveman agent: memory recallBudget must be a non-negative in

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/889fc4e059768323. Report an issue: GitHub.