paperclipai/paperclip · error · Error

Protocol eval catalog exceeds the two-shard GitHub matrix li

Error message

Protocol eval catalog exceeds the two-shard GitHub matrix limit

What it means

The catalog builder distributes eval cells across exactly two shards (round-robin by index) to drive a GitHub Actions matrix, and GitHub caps matrix jobs at 256 per job. If either shard exceeds 256 cells (i.e. more than 512 total cells), the build refuses to emit a catalog that CI could not run.

Source

Thrown at packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs:203

  }
  if (rosters.length === 0) throw new Error("No live rosters were selected");

  const cells = rosters.flatMap((roster) =>
    roster.cases.map((caseId) => ({
      cellId: safeId(`${roster.rosterId}--${caseId}`, "cell ID"),
      rosterId: roster.rosterId,
      rosterFile: roster.rosterFile,
      caseId,
      model: roster.model,
      provider: roster.provider,
      driver: roster.driver,
      credentialName: roster.credentialName,
    })),
  );
  const shards = [[], []];
  cells.forEach((cell, index) => shards[index % shards.length].push(cell));
  if (shards.some((shard) => shard.length > 256)) {
    throw new Error(
      "Protocol eval catalog exceeds the two-shard GitHub matrix limit",
    );
  }
  return {
    schema: "paperclip.runner-protocol-eval.catalog/v1",
    campaignId,
    source,
    selection: { kind: requested === null ? "maintained_full" : "subset", rosters: rosterSelection },
    rosters,
    cells,
    matrices: shards.map((include) => ({ include })),
    maxParallel,
    maxParallelPerShard: Math.floor(maxParallel / 2),
  };
}

async function writeGithubOutput(entries) {
  const output = process.env.GITHUB_OUTPUT;

View on GitHub (pinned to 01ad858492)

Solutions

Reduce the number of cases per roster or split rosters across multiple campaigns.
Increase the shard count in the script (e.g. shards.length from 2 to 3+) and update the GitHub workflow matrix accordingly.
Audit the rosters directory for accidentally duplicated roster files inflating cell count.

Example fix

// before
const shards = [[], []];
cells.forEach((cell, index) => shards[index % shards.length].push(cell));
if (shards.some((s) => s.length > 256)) throw new Error("...");
// after
const shards = [[], [], [], []];
cells.forEach((cell, index) => shards[index % shards.length].push(cell));
if (shards.some((s) => s.length > 256)) throw new Error("...");

Defensive patterns

Strategy: validation

Validate before calling

const cellCount = rosters.reduce((n, r) => n + r.cases.length, 0);
if (cellCount > 512) {
  throw new Error(`${cellCount} cells exceed the 512 (2 shards x 256) matrix limit`);
}

Prevention

Track total case count when adding rosters; keep it below 512 or raise shard count in sync with the workflow matrix.
Add a CI preflight step that runs the catalog builder and fails early with a readable message.

When it happens

Trigger: Generating a campaign catalog whose total number of cells (rosters × cases) exceeds 512, causing at least one of the two round-robin shards to hold more than 256 cells.

Common situations: Adding many new rosters or large case lists until the combined matrix outgrows the two-shard limit; a loop or duplicated roster files inflating the cell count; CI scale-out after merging several case-expanding PRs.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/5462099b76e99e15. Report an issue: GitHub.