What causes "unknown Runner live eval candidate: ${id}" in paperclipai/paperclip?

Print the schedule's valid ids (schedule.candidates.map(c => c.id)) and correct the selection to use one of them.

unknown Runner live eval candidate: ${id}

Error message

unknown Runner live eval candidate: ${id}

What it means

selectRunnerLiveEvalSchedule validates the candidateIds filter of a RunnerLiveEvalSelection against the candidate list of the compiled schedule. If any requested candidate id does not exist in schedule.candidates, it throws immediately rather than silently producing an empty or partial selection.

Source

Thrown at packages/paperclip-runner/src/eval/live-workflow-matrix.ts:265

  expectedExecutions: number;
}

export interface RunnerLiveEvalSelection {
  candidateIds?: readonly string[];
  caseIds?: readonly string[];
  limit?: number;
}

/** Selects a stable paid subset without weakening validation of the full schedule. */
export function selectRunnerLiveEvalSchedule(
  schedule: RunnerLiveEvalSchedule,
  selection: RunnerLiveEvalSelection,
): RunnerLiveEvalSchedule {
  const candidateIds = new Set(selection.candidateIds ?? []);
  const caseIds = new Set(selection.caseIds ?? []);
  for (const id of candidateIds) {
    if (!schedule.candidates.some((candidate) => candidate.id === id)) {
      throw new Error(`unknown Runner live eval candidate: ${id}`);
    }
  }
  const scheduledCaseIds = new Set(
    schedule.entries.map((entry) => entry.caseId),
  );
  for (const id of caseIds) {
    if (!scheduledCaseIds.has(id as RunnerWorkflowEvalCase["id"])) {
      throw new Error(`unknown Runner live eval case: ${id}`);
    }
  }
  if (
    selection.limit !== undefined &&
    (!Number.isSafeInteger(selection.limit) || selection.limit <= 0)
  ) {
    throw new Error(
      "Runner live eval selection limit must be a positive integer",
    );
  }

View on GitHub (pinned to 01ad858492)

Solutions

Print the schedule's valid ids (schedule.candidates.map(c => c.id)) and correct the selection to use one of them.
Update the selection config after any rename/removal of candidates.
Validate selection ids against the schedule in your own config loader before calling selectRunnerLiveEvalSchedule.

Example fix

// before
selectRunnerLiveEvalSchedule(schedule, { candidateIds: ['old-candidate'] });
// after
const valid = schedule.candidates.map((c) => c.id);
selectRunnerLiveEvalSchedule(schedule, { candidateIds: ['old-candidate'].filter((id) => valid.includes(id)) });

Defensive patterns

Strategy: validation

Validate before calling

const valid = new Set(schedule.candidates.map((c) => c.id));
const bad = (selection.candidateIds ?? []).filter((id) => !valid.has(id));
if (bad.length) throw new Error(`selection references unknown candidates: ${bad.join(', ')}`);

Type guard

function candidatesExist(schedule: Schedule, ids?: string[]): ids is string[] {
  const valid = new Set(schedule.candidates.map((c) => c.id));
  return (ids ?? []).every((id) => valid.has(id));
}

Try / catch

try {
  const plan = selectRunnerLiveEvalSchedule(schedule, selection);
} catch (err) {
  if ((err as Error).message.startsWith('unknown Runner live eval candidate')) {
    throw new ConfigError(`candidateIds invalid; valid ids: ${schedule.candidates.map((c) => c.id).join(', ')}`);
  }
  throw err;
}

Prevention

Regenerate or review selection configs whenever candidate definitions change.
Keep candidate ids in a single shared constants module used by both matrix and selection.
Lint eval selection files against the current schedule in CI.
Copy selections only between compatible schedule versions.

When it happens

Trigger: Passing selection.candidateIds containing an id that is not in the schedule's candidates — e.g. a renamed or removed candidate, a typo, or an id from a different schedule version.

Common situations: Editing candidate definitions in the matrix and forgetting to update selection configs; copying a selection between environments/matrix versions; stale CI config referencing an old candidate id.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/07551c4a573c477c. Report an issue: GitHub.