JuliusBrussee/caveman · error · Error

caveman agent: latency guardrail requires positive integer…

Error message

caveman agent: latency guardrail requires positive integer p95_ms

What it means

evalFixture() validates latency_threshold guardrails: p95_ms must be a positive safe integer (Number.isSafeInteger and > 0). It throws when a guardrail of type latency_threshold has a missing, zero, negative, fractional, or non-integer p95_ms. This ensures the p95 latency threshold in milliseconds is a usable value.

Solutions

Set p95_ms to a positive integer, e.g. { type: "latency_threshold", p95_ms: 2000 }.
Round thresholds: Math.round(seconds * 1000), and enforce >= 1.
Remove the latency_threshold guardrail if you do not intend to enforce a latency bound.

Example fix

// before
{ type: "latency_threshold", p95_ms: 1.5 }
// after
{ type: "latency_threshold", p95_ms: 1500 }

Defensive patterns

Strategy: validation

Validate before calling

for (const g of guardrails) {
  if (g.type === "latency_threshold" && !(Number.isSafeInteger(g.p95_ms) && g.p95_ms > 0)) {
    throw new Error(`latency_threshold p95_ms must be a positive integer, got ${g.p95_ms}`);
  }
}

Type guard

function isPositiveInt(v: unknown): v is number { return typeof v === "number" && Number.isSafeInteger(v) && v > 0; }

Try / catch

try { ev = evalFixture(opts); }
catch (e) { if (String(e.message).includes("latency guardrail")) throw new Error("Set p95_ms as a positive integer in milliseconds, e.g. 2000"); throw e; }

Prevention

Always express p95_ms in whole milliseconds (e.g. 2000, not 2 or 1.5).
Convert seconds with Math.round(seconds * 1000).
Omit the guardrail entirely if no latency bound is intended, rather than passing 0.

When it happens

Trigger: evalFixture with guardrails: [{ type: "latency_threshold", p95_ms: 0 }], a negative p95_ms, a float like 250.5, NaN/Infinity, or omitting p95_ms entirely on a latency_threshold guardrail.

Common situations: Using 0 to mean 'no limit' instead of omitting the guardrail; seconds-vs-milliseconds confusion producing small floats; config values that arrive as strings and fail the integer check after partial coercion.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/6f1012cd2d39fcd2. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/primitives.ts:454

  approved?: boolean;
  required?: boolean;
  input: unknown;
  tools?: { mode: "fixture" | "live"; sandbox?: string };
  quality: QualityGrader[];
  guardrails?: EvalGuardrail[];
}): EvalDefinition {
  if (options.id.trim() === "") throw new Error("caveman agent: eval id is required");
  if (options.quality.length === 0) throw new Error("caveman agent: eval needs at least one quality grader");
  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };
  if (tools.mode === "live" && !tools.sandbox) {
    throw new Error("caveman agent: live eval tools require an explicit sandbox");
  }
  return Object.freeze({
    kind: "eval",
    id: options.id,
    approved: options.approved ?? false,
    required: options.required ?? true,
    input: options.input,
    tools,
    quality: Object.freeze([...options.quality]),

View on GitHub (pinned to 3ee70a1026)