paperclipai/paperclip · error · Error

live schedule coverage failed: ${JSON.stringify(coverage)}

Error message

live schedule coverage failed: ${JSON.stringify(coverage)}

What it means

The runner live-eval schedule script computes a coverage object via evals.runnerLiveScheduleCoverage(seed) and asserts every dimension of coverage is truthy before building the schedule. If any coverage dimension is false, it throws with the full coverage map serialized so you can see which dimension failed. It is a self-check that the generated schedule fully covers the maintained live roster/protocol matrix.

Source

Thrown at packages/paperclip-runner/scripts/run-runner-live-eval-schedule.mjs:173

  // Candidate sets alternate, so seven compatible weekly baselines require
  // roughly fourteen weeks of history. Keep a little extra scheduling margin.
  const expiry = Date.now() - 120 * 24 * 60 * 60 * 1_000;
  for (const name of await readdir(historyDirectory)) {
    if (!name.endsWith(".json")) continue;
    const metadata = await stat(resolve(historyDirectory, name));
    if (metadata.mtimeMs < expiry) await rm(resolve(historyDirectory, name));
  }
}

if (mode === "nightly") {
  const fullSchedule = evals.buildRunnerLiveEvalSchedule({
    seed,
    rotationDay,
    generatedAt: now,
  });
  const coverage = evals.runnerLiveScheduleCoverage(seed);
  if (!Object.values(coverage).every(Boolean))
    throw new Error(
      `live schedule coverage failed: ${JSON.stringify(coverage)}`,
    );
  const schedule = selectionActive
    ? evals.selectRunnerLiveEvalSchedule(fullSchedule, selection)
    : fullSchedule;
  await writeFile(
    resolve(outputDirectory, "nightly-schedule.json"),
    `${JSON.stringify({ schedule, coverage, selection: selectionActive ? selection : null }, null, 2)}\n`,
  );
  if (!execute) {
    process.stdout.write(
      `Runner live eval schedule ready: ${schedule.expectedExecutions} executions, rotation week ${schedule.rotationDay}. Use --execute to run providers.\n`,
    );
    process.exit(0);
  }

  const campaignCostLimit = evals.parseRunnerLiveCampaignCostLimit(
    process.env.PAPERCLIP_EVAL_MAX_CAMPAIGN_COST_USD,

View on GitHub (pinned to 01ad858492)

Solutions

1. Inspect the JSON in the error message to see which coverage key(s) are false and trace why the schedule generator omits that lane.
2. Regenerate the schedule with a fresh seed/rotation to see if coverage is seed-dependent.
3. Check recently added rosters/campaign lanes (packages/evals/evals/paperclip-runner/rosters, campaigns/live-direct-full.json) for a lane the generator doesn't handle.
4. Update the coverage check or generator in the evals module if a new dimension was intentionally added.

Example fix

// before: generator skips new 'acpx/pi' lane, coverage = { lanes: true, providers: false, ... }
// after: add the provider to the generator's lane mapping so every provider appears in the schedule
const lanes = rosters.flatMap((r) => r.cases.map((c) => ({ roster: r.id, provider: c.config.provider, agent: c.config.acpxAgent })));
// coverage.providers now true

const coverage = evals.runnerLiveScheduleCoverage(seed);
const missing = Object.entries(coverage).filter(([, ok]) => !ok).map(([k]) => k);
if (missing.length > 0) throw new Error(`Schedule coverage incomplete: ${missing.join(", ")}`);

const isFullyCovered = (cov) => Object.values(cov ?? {}).every(Boolean);

try {
  await runSchedule();
} catch (err) {
  if (String(err.message).startsWith("live schedule coverage failed")) {
    const cov = JSON.parse(err.message.slice(err.message.indexOf('{')));
    console.error("Uncovered dimensions:", Object.keys(cov).filter((k) => !cov[k]));
  }
  throw err;
}

When it happens

Trigger: Running packages/paperclip-runner/scripts/run-runner-live-eval-schedule.mjs when the freshly generated seed schedule does not cover every expected eval lane — e.g. a newly added roster case, provider, or execution class that the schedule generator did not place into the schedule.

Common situations: A contributor adds a new eval case or lane to the maintained campaign but the schedule generation logic doesn't pick it up; roster files edited by hand leaving a dimension uncovered; changes to schedule selection rules in the evals module; stale generated schedule checked into the repo.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• schedule references unknown candidate ${entry.candidateId}
• provider turn ended with status ${turn.status}
• live_provider_execution_failed
• PAPERCLIP_EVAL_MAX_CAMPAIGN_COST_USD must be a positive fini
• Runner live rotation requires a valid generated-at time

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/0434dea343ccfc34. Report an issue: GitHub.