{"record":{"id":"5462099b76e99e15","repo":"paperclipai/paperclip","slug":"protocol-eval-catalog-exceeds-the-two-shard-github","errorCode":null,"errorMessage":"Protocol eval catalog exceeds the two-shard GitHub matrix limit","messagePattern":"Protocol eval catalog exceeds the two-shard GitHub matrix limit","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs","lineNumber":203,"sourceCode":"  }\n  if (rosters.length === 0) throw new Error(\"No live rosters were selected\");\n\n  const cells = rosters.flatMap((roster) =>\n    roster.cases.map((caseId) => ({\n      cellId: safeId(`${roster.rosterId}--${caseId}`, \"cell ID\"),\n      rosterId: roster.rosterId,\n      rosterFile: roster.rosterFile,\n      caseId,\n      model: roster.model,\n      provider: roster.provider,\n      driver: roster.driver,\n      credentialName: roster.credentialName,\n    })),\n  );\n  const shards = [[], []];\n  cells.forEach((cell, index) => shards[index % shards.length].push(cell));\n  if (shards.some((shard) => shard.length > 256)) {\n    throw new Error(\n      \"Protocol eval catalog exceeds the two-shard GitHub matrix limit\",\n    );\n  }\n  return {\n    schema: \"paperclip.runner-protocol-eval.catalog/v1\",\n    campaignId,\n    source,\n    selection: { kind: requested === null ? \"maintained_full\" : \"subset\", rosters: rosterSelection },\n    rosters,\n    cells,\n    matrices: shards.map((include) => ({ include })),\n    maxParallel,\n    maxParallelPerShard: Math.floor(maxParallel / 2),\n  };\n}\n\nasync function writeGithubOutput(entries) {\n  const output = process.env.GITHUB_OUTPUT;","sourceCodeStart":185,"sourceCodeEnd":221,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs#L185-L221","documentation":"The catalog builder distributes eval cells across exactly two shards (round-robin by index) to drive a GitHub Actions matrix, and GitHub caps matrix jobs at 256 per job. If either shard exceeds 256 cells (i.e. more than 512 total cells), the build refuses to emit a catalog that CI could not run.","triggerScenarios":"Generating a campaign catalog whose total number of cells (rosters × cases) exceeds 512, causing at least one of the two round-robin shards to hold more than 256 cells.","commonSituations":"Adding many new rosters or large case lists until the combined matrix outgrows the two-shard limit; a loop or duplicated roster files inflating the cell count; CI scale-out after merging several case-expanding PRs.","solutions":["Reduce the number of cases per roster or split rosters across multiple campaigns.","Increase the shard count in the script (e.g. shards.length from 2 to 3+) and update the GitHub workflow matrix accordingly.","Audit the rosters directory for accidentally duplicated roster files inflating cell count."],"exampleFix":"// before\nconst shards = [[], []];\ncells.forEach((cell, index) => shards[index % shards.length].push(cell));\nif (shards.some((s) => s.length > 256)) throw new Error(\"...\");\n// after\nconst shards = [[], [], [], []];\ncells.forEach((cell, index) => shards[index % shards.length].push(cell));\nif (shards.some((s) => s.length > 256)) throw new Error(\"...\");","handlingStrategy":"validation","validationCode":"const cellCount = rosters.reduce((n, r) => n + r.cases.length, 0);\nif (cellCount > 512) {\n  throw new Error(`${cellCount} cells exceed the 512 (2 shards x 256) matrix limit`);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Track total case count when adding rosters; keep it below 512 or raise shard count in sync with the workflow matrix.","Add a CI preflight step that runs the catalog builder and fails early with a readable message."],"tags":["ci","limits","matrix"],"backgroundTag":"value-out-of-range","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-10T03:14:50.855Z","contentChangedAt":"2026-09-10T03:14:50.855Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}