JuliusBrussee/caveman · error · Error

caveman agent: error-rate guardrail max must be in [0,1]

Error message

caveman agent: error-rate guardrail max must be in [0,1]

What it means

evalFixture() validates error_rate guardrails: max must be a finite number in the closed interval [0, 1]. It throws when max is non-finite (NaN/Infinity), negative, or greater than 1, because an error rate is a ratio between 0 and 1 inclusive.

Solutions

1. Express the rate as a fraction in [0,1], e.g. 0.05 for 5%.
2. Convert percentages in code: max: percentValue / 100.
3. Clamp/validate before the call: Number.isFinite(v) && v >= 0 && v <= 1.

Example fix

// before
{ type: "error_rate", max: 5 } // meant 5%
// after
{ type: "error_rate", max: 0.05 }

for (const g of guardrails) {
  if (g.type === "error_rate" && !(Number.isFinite(g.max) && g.max >= 0 && g.max <= 1)) {
    throw new Error(`error_rate max must be in [0,1], got ${g.max}`);
  }
}

function isRatio(v: unknown): v is number { return typeof v === "number" && Number.isFinite(v) && v >= 0 && v <= 1; }

try { ev = evalFixture(opts); }
catch (e) { if (String(e.message).includes("error-rate")) throw new Error("Use a fraction in [0,1] for error_rate.max (0.05 = 5%)"); throw e; }

When it happens

Trigger: guardrails: [{ type: "error_rate", max: 5 }] (expressing 5% as 5 instead of 0.05), max: -0.1, max: Infinity/NaN, or a non-numeric value.

Common situations: Specifying percentages as whole numbers (5 for 5%) instead of fractions; a config merge producing undefined that becomes NaN after arithmetic; unit confusion between percent and ratio.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• caveman agent: latency guardrail requires positive integer…
• cave_memory_value_invalid
• caveman agent: eval id is required
• caveman agent: eval needs at least one quality grader
• caveman agent: live eval tools require an explicit sandbox

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/889fc4e059768323. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/primitives.ts:458

  quality: QualityGrader[];
  guardrails?: EvalGuardrail[];
}): EvalDefinition {
  if (options.id.trim() === "") throw new Error("caveman agent: eval id is required");
  if (options.quality.length === 0) throw new Error("caveman agent: eval needs at least one quality grader");
  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };
  if (tools.mode === "live" && !tools.sandbox) {
    throw new Error("caveman agent: live eval tools require an explicit sandbox");
  }
  return Object.freeze({
    kind: "eval",
    id: options.id,
    approved: options.approved ?? false,
    required: options.required ?? true,
    input: options.input,
    tools,
    quality: Object.freeze([...options.quality]),
    guardrails: Object.freeze([...(options.guardrails ?? [])]),
  });
}

View on GitHub (pinned to 3ee70a1026)