What causes "unknown Runner live eval case: ${id}" in paperclipai/paperclip?

List valid ids via new Set(schedule.entries.map((e) => e.caseId)) and fix the selection.

unknown Runner live eval case: ${id}

Error message

unknown Runner live eval case: ${id}

What it means

selectRunnerLiveEvalSchedule validates the caseIds filter of a RunnerLiveEvalSelection against the case ids actually scheduled in schedule.entries. If a requested case id has no scheduled entry, it throws this error. This prevents selections that silently match nothing because a case was never scheduled.

Source

Thrown at packages/paperclip-runner/src/eval/live-workflow-matrix.ts:273

/** Selects a stable paid subset without weakening validation of the full schedule. */
export function selectRunnerLiveEvalSchedule(
  schedule: RunnerLiveEvalSchedule,
  selection: RunnerLiveEvalSelection,
): RunnerLiveEvalSchedule {
  const candidateIds = new Set(selection.candidateIds ?? []);
  const caseIds = new Set(selection.caseIds ?? []);
  for (const id of candidateIds) {
    if (!schedule.candidates.some((candidate) => candidate.id === id)) {
      throw new Error(`unknown Runner live eval candidate: ${id}`);
    }
  }
  const scheduledCaseIds = new Set(
    schedule.entries.map((entry) => entry.caseId),
  );
  for (const id of caseIds) {
    if (!scheduledCaseIds.has(id as RunnerWorkflowEvalCase["id"])) {
      throw new Error(`unknown Runner live eval case: ${id}`);
    }
  }
  if (
    selection.limit !== undefined &&
    (!Number.isSafeInteger(selection.limit) || selection.limit <= 0)
  ) {
    throw new Error(
      "Runner live eval selection limit must be a positive integer",
    );
  }
  let entries = schedule.entries.filter(
    (entry) =>
      (candidateIds.size === 0 || candidateIds.has(entry.candidateId)) &&
      (caseIds.size === 0 || caseIds.has(entry.caseId)),
  );
  if (selection.limit !== undefined)
    entries = entries.slice(0, selection.limit);
  if (entries.length === 0) {

View on GitHub (pinned to 01ad858492)

Solutions

List valid ids via new Set(schedule.entries.map((e) => e.caseId)) and fix the selection.
Synchronize the selection config with the current matrix case definitions.
Pre-validate caseIds against scheduled entries in your tooling before invoking the selector.

Example fix

// before
selectRunnerLiveEvalSchedule(schedule, { caseIds: ['removed-case'] });
// after
const scheduled = new Set(schedule.entries.map((e) => e.caseId));
selectRunnerLiveEvalSchedule(schedule, { caseIds: ['removed-case'].filter((id) => scheduled.has(id)) });

Defensive patterns

Strategy: validation

Validate before calling

const scheduled = new Set(schedule.entries.map((e) => e.caseId));
const bad = (selection.caseIds ?? []).filter((id) => !scheduled.has(id));
if (bad.length) throw new Error(`selection references unscheduled cases: ${bad.join(', ')}`);

Type guard

function casesAreScheduled(schedule: Schedule, ids?: string[]): ids is string[] {
  const scheduled = new Set(schedule.entries.map((e) => e.caseId));
  return (ids ?? []).every((id) => scheduled.has(id));
}

Try / catch

try {
  const plan = selectRunnerLiveEvalSchedule(schedule, selection);
} catch (err) {
  if ((err as Error).message.startsWith('unknown Runner live eval case')) {
    throw new ConfigError(`caseIds invalid; scheduled ids: ${[...new Set(schedule.entries.map((e) => e.caseId))].join(', ')}`);
  }
  throw err;
}

Prevention

Derive caseId lists from the schedule file instead of hard-coding them in CI configs.
Update selection configs in the same change that renames or removes matrix cases.
Add a config test that selects every declared case against the current schedule.
Use TypeScript union types for case ids where possible to catch typos at compile time.

When it happens

Trigger: Passing selection.caseIds containing a case id that is not present in any schedule.entries entry — a typo, a case removed from the matrix, or an id from another schedule version.

Common situations: Referencing an eval case that was renamed in the matrix definition; running a selection file against a newer/older schedule where the case was dropped; hard-coded case lists in CI pipelines drifting from the schedule.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/e68b0eba7e1a6cab. Report an issue: GitHub.