{"record":{"id":"f58dc87bdb764e56","repo":"paperclipai/paperclip","slug":"live-roster-rosterid-has-no-cases","errorCode":null,"errorMessage":"Live roster ${rosterId} has no cases","messagePattern":"Live roster (.+?) has no cases","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs","lineNumber":153,"sourceCode":"        entry.isFile() &&\n        entry.name.startsWith(\"live-\") &&\n        entry.name.endsWith(\".json\"),\n    )\n    .map((entry) => entry.name)\n    .sort();\n  const rosters = [];\n  for (const rosterFile of rosterFiles) {\n    const rosterPath = resolve(rosterRoot, rosterFile);\n    const roster = await loadObject(rosterPath);\n    const rosterId = safeId(roster.id, \"roster ID\");\n    if (selected && !selected.has(rosterId) && !selected.has(rosterFile)) {\n      continue;\n    }\n    if (roster.schema !== \"paperclip-runner/live-roster/v1\") {\n      throw new Error(`Unsupported live roster schema in ${rosterFile}`);\n    }\n    if (!Array.isArray(roster.cases) || roster.cases.length === 0) {\n      throw new Error(`Live roster ${rosterId} has no cases`);\n    }\n    const configPath = inside(\n      programRoot,\n      resolve(rosterRoot, String(roster.config ?? \"\")),\n      `Config for ${rosterId}`,\n    );\n    const config = await loadObject(configPath);\n    const credentialName = credentialForConfig(config);\n    const cases = roster.cases.map((caseId) => safeId(caseId, \"case ID\"));\n    if (new Set(cases).size !== cases.length) {\n      throw new Error(`Live roster ${rosterId} repeats a case`);\n    }\n    rosters.push({\n      rosterId,\n      rosterFile,\n      configFile: relative(programRoot, configPath).split(sep).join(\"/\"),\n      model: String(roster.model ?? config.model ?? \"unknown\"),\n      provider: String(config.provider ?? \"codex\"),","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs#L135-L171","documentation":"buildProtocolEvalCatalog loads every live roster JSON file under the rosters directory and validates its schema. After confirming the schema is 'paperclip-runner/live-roster/v1', it checks that roster.cases is a non-empty array. If cases is missing, not an array, or empty, the script refuses to build an eval campaign around a roster with nothing to run.","triggerScenarios":"A live roster JSON file with schema 'paperclip-runner/live-roster/v1' either omits the 'cases' field entirely, sets it to a non-array value (e.g. a string or object), or sets it to an empty array [].","commonSituations":"Hand-authoring a new roster file and forgetting to fill in cases; copying a roster template with the cases list stripped out; a script or merge that emptied the cases array; renaming the field (e.g. 'tests' instead of 'cases').","solutions":["Open the roster file named in the error path and add at least one valid case ID to the 'cases' array.","Verify the field is literally named 'cases' and is a JSON array of case IDs (strings).","If the roster is intentionally unused, remove the roster file so it is not picked up by the catalog builder."],"exampleFix":"// before\n{ \"schema\": \"paperclip-runner/live-roster/v1\", \"model\": \"gpt-5\", \"cases\": [] }\n// after\n{ \"schema\": \"paperclip-runner/live-roster/v1\", \"model\": \"gpt-5\", \"cases\": [\"case-001\", \"case-002\"] }","handlingStrategy":"validation","validationCode":"const roster = JSON.parse(await readFile(rosterFile, \"utf8\"));\nif (roster.schema === \"paperclip-runner/live-roster/v1\" &&\n    (!Array.isArray(roster.cases) || roster.cases.length === 0)) {\n  throw new Error(`${rosterFile}: cases must be a non-empty array`);\n}","typeGuard":"function hasCases(r) {\n  return Array.isArray(r.cases) && r.cases.length > 0;\n}","tryCatchPattern":null,"preventionTips":["Add a JSON schema check for roster files in CI before running the campaign.","Never commit roster templates with empty cases arrays; fill them in or omit the file."],"tags":["validation","config","empty-array"],"backgroundTag":"schema-validation-failed","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-10T03:14:50.855Z","contentChangedAt":"2026-09-10T03:14:50.855Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}