JuliusBrussee/caveman · error · Error

caveman agent: eval id is required

Error message

caveman agent: eval id is required

What it means

Thrown by the evalFixture() builder when options.id is empty or whitespace-only. Eval fixture ids identify the fixture in eval-complete selection and lock evidence, so a blank id is rejected at definition construction before any eval run starts.

Source

Thrown at packages/agent/src/primitives.ts:443

  readonly id: string;
  readonly approved: boolean;
  readonly required: boolean;
  readonly input: unknown;
  readonly tools: { mode: "fixture" | "live"; sandbox?: string };
  readonly quality: readonly QualityGrader[];
  readonly guardrails: readonly EvalGuardrail[];
}

export function evalFixture(options: {
  id: string;
  approved?: boolean;
  required?: boolean;
  input: unknown;
  tools?: { mode: "fixture" | "live"; sandbox?: string };
  quality: QualityGrader[];
  guardrails?: EvalGuardrail[];
}): EvalDefinition {
  if (options.id.trim() === "") throw new Error("caveman agent: eval id is required");
  if (options.quality.length === 0) throw new Error("caveman agent: eval needs at least one quality grader");
  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Give every fixture a stable non-empty id, e.g. 'rag-happy-path'
2. When generating ids, fall back to an index or slug: id: raw?.trim() || `fixture-${i}`
3. Fail early in your fixture loader with a clear file/entry reference instead of letting the builder throw

Example fix

// before
evalFixture({ id: entry.name ?? '', input: q, quality: [{ type: 'contains', fragments: ['ok'] }] });

// after
evalFixture({ id: entry.name?.trim() || `fixture-${index}`, input: q, quality: [{ type: 'contains', fragments: ['ok'] }] });

function fixtureId(raw: unknown, index: number): string {
  const id = typeof raw === 'string' ? raw.trim() : '';
  if (id === '') throw new Error(`fixture[${index}] is missing a non-empty id`);
  return id;
}

function isFixtureId(value: unknown): value is string { return typeof value === 'string' && value.trim() !== ''; }

When it happens

Trigger: Calling evalFixture({ id: '' }) or evalFixture({ id: ' ' }) with an otherwise valid input and quality graders.

Common situations: Generating fixture ids from file stems where a file has no name; iterating a config list where one entry is missing its id key; sanitizing user labels down to an empty string before passing them as ids.

Related errors

• cave_live_eval_sandbox_profile_missing
• cave_live_eval_sandbox_profile_invalid
• caveman agent: context id is required
• caveman agent: unknown grader ${(grader as { type: string })
• caveman agent: latency guardrail requires positive integer p

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/7404a237132c231e. Report an issue: GitHub.