{"record":{"id":"0434dea343ccfc34","repo":"paperclipai/paperclip","slug":"live-schedule-coverage-failed-json-stringify-co","errorCode":null,"errorMessage":"live schedule coverage failed: ${JSON.stringify(coverage)}","messagePattern":"live schedule coverage failed: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/paperclip-runner/scripts/run-runner-live-eval-schedule.mjs","lineNumber":173,"sourceCode":"  // Candidate sets alternate, so seven compatible weekly baselines require\n  // roughly fourteen weeks of history. Keep a little extra scheduling margin.\n  const expiry = Date.now() - 120 * 24 * 60 * 60 * 1_000;\n  for (const name of await readdir(historyDirectory)) {\n    if (!name.endsWith(\".json\")) continue;\n    const metadata = await stat(resolve(historyDirectory, name));\n    if (metadata.mtimeMs < expiry) await rm(resolve(historyDirectory, name));\n  }\n}\n\nif (mode === \"nightly\") {\n  const fullSchedule = evals.buildRunnerLiveEvalSchedule({\n    seed,\n    rotationDay,\n    generatedAt: now,\n  });\n  const coverage = evals.runnerLiveScheduleCoverage(seed);\n  if (!Object.values(coverage).every(Boolean))\n    throw new Error(\n      `live schedule coverage failed: ${JSON.stringify(coverage)}`,\n    );\n  const schedule = selectionActive\n    ? evals.selectRunnerLiveEvalSchedule(fullSchedule, selection)\n    : fullSchedule;\n  await writeFile(\n    resolve(outputDirectory, \"nightly-schedule.json\"),\n    `${JSON.stringify({ schedule, coverage, selection: selectionActive ? selection : null }, null, 2)}\\n`,\n  );\n  if (!execute) {\n    process.stdout.write(\n      `Runner live eval schedule ready: ${schedule.expectedExecutions} executions, rotation week ${schedule.rotationDay}. Use --execute to run providers.\\n`,\n    );\n    process.exit(0);\n  }\n\n  const campaignCostLimit = evals.parseRunnerLiveCampaignCostLimit(\n    process.env.PAPERCLIP_EVAL_MAX_CAMPAIGN_COST_USD,","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/scripts/run-runner-live-eval-schedule.mjs#L155-L191","documentation":"The runner live-eval schedule script computes a coverage object via evals.runnerLiveScheduleCoverage(seed) and asserts every dimension of coverage is truthy before building the schedule. If any coverage dimension is false, it throws with the full coverage map serialized so you can see which dimension failed. It is a self-check that the generated schedule fully covers the maintained live roster/protocol matrix.","triggerScenarios":"Running packages/paperclip-runner/scripts/run-runner-live-eval-schedule.mjs when the freshly generated seed schedule does not cover every expected eval lane — e.g. a newly added roster case, provider, or execution class that the schedule generator did not place into the schedule.","commonSituations":"A contributor adds a new eval case or lane to the maintained campaign but the schedule generation logic doesn't pick it up; roster files edited by hand leaving a dimension uncovered; changes to schedule selection rules in the evals module; stale generated schedule checked into the repo.","solutions":["Inspect the JSON in the error message to see which coverage key(s) are false and trace why the schedule generator omits that lane.","Regenerate the schedule with a fresh seed/rotation to see if coverage is seed-dependent.","Check recently added rosters/campaign lanes (packages/evals/evals/paperclip-runner/rosters, campaigns/live-direct-full.json) for a lane the generator doesn't handle.","Update the coverage check or generator in the evals module if a new dimension was intentionally added."],"exampleFix":"// before: generator skips new 'acpx/pi' lane, coverage = { lanes: true, providers: false, ... }\n// after: add the provider to the generator's lane mapping so every provider appears in the schedule\nconst lanes = rosters.flatMap((r) => r.cases.map((c) => ({ roster: r.id, provider: c.config.provider, agent: c.config.acpxAgent })));\n// coverage.providers now true","handlingStrategy":"validation","validationCode":"const coverage = evals.runnerLiveScheduleCoverage(seed);\nconst missing = Object.entries(coverage).filter(([, ok]) => !ok).map(([k]) => k);\nif (missing.length > 0) throw new Error(`Schedule coverage incomplete: ${missing.join(\", \")}`);","typeGuard":"const isFullyCovered = (cov) => Object.values(cov ?? {}).every(Boolean);","tryCatchPattern":"try {\n  await runSchedule();\n} catch (err) {\n  if (String(err.message).startsWith(\"live schedule coverage failed\")) {\n    const cov = JSON.parse(err.message.slice(err.message.indexOf('{')));\n    console.error(\"Uncovered dimensions:\", Object.keys(cov).filter((k) => !cov[k]));\n  }\n  throw err;\n}","preventionTips":["Run the coverage check locally before committing new eval lanes or rosters.","When adding a coverage dimension, update both the generator and the coverage function together.","Keep generated schedules regenerated in CI rather than hand-editing them."],"tags":["eval","schedule","coverage","ci"],"backgroundTag":"schema-validation-failed","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-10T03:14:50.855Z","contentChangedAt":"2026-09-10T03:14:50.855Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}