{"record":{"id":"66193ac26d5fdfe3","repo":"chenglou/pretext","slug":"cannot-summarize-zero-benchmark-runs","errorCode":null,"errorMessage":"Cannot summarize zero benchmark runs","messagePattern":"Cannot summarize zero benchmark runs","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"scripts/benchmark-check.ts","lineNumber":181,"sourceCode":"        assertSame(\n          row[metadataKey],\n          firstRow[metadataKey],\n          `corpusResults[${rowIndex}].${metadataKey}`,\n        )\n      }\n    }\n\n    for (const timingKey of CORPUS_TIMING_KEYS) {\n      result[timingKey] = median(reports.map(report => report.corpusResults![rowIndex]![timingKey]))\n    }\n\n    return result\n  })\n}\n\nfunction medianReport(reports: BenchmarkReport[]): BenchmarkReport {\n  if (reports.length === 0) {\n    throw new Error('Cannot summarize zero benchmark runs')\n  }\n\n  for (const [index, report] of reports.entries()) {\n    if (report.status === 'error') {\n      throw new Error(`Benchmark run ${index + 1} failed: ${report.message ?? 'unknown error'}`)\n    }\n  }\n\n  const report: BenchmarkReport = { status: 'ready' }\n\n  for (const key of BENCHMARK_RESULT_KEYS) {\n    const rows = medianBenchmarkResults(reports, key)\n    if (rows !== undefined) report[key] = rows\n  }\n\n  const corpusResults = medianCorpusResults(reports)\n  if (corpusResults !== undefined) report.corpusResults = corpusResults\n","sourceCodeStart":163,"sourceCodeEnd":199,"githubUrl":"https://github.com/chenglou/pretext/blob/ac49b09b7d83ede19581fa94a8b892b07d309baf/scripts/benchmark-check.ts#L163-L199","documentation":"Thrown by medianReport() if the reports array passed in is empty (length 0). The function cannot compute a median of zero runs, so it refuses rather than returning a fabricated report. In the normal benchmark-check flow this is guarded upstream: error 13 enforces runs >= 1, and the collection loop runs exactly `runs` iterations — so reaching medianReport with zero reports implies the loop body was bypassed or the array was mutated after collection.","triggerScenarios":"Calling medianReport([]) directly, or a code path where the reports-collection loop pushed nothing (e.g. an early break, an exception swallowed per-iteration, or runs being mutated to 0 after the loop guard). Not reachable via the standard CLI path.","commonSituations":"A future refactor that filters reports before mediating and accidentally empties the list; a test that calls medianReport with an empty fixture; a bug where a run error short-circuits collection. Practically unseen in normal CLI usage.","solutions":["If calling medianReport programmatically, guard for empty input and handle it at the call site.","Confirm the collection loop actually pushes each report (no swallowed exceptions or early returns).","Run with --runs=1 to verify at least one report is collected before mediating.","Audit any filtering applied between collection and medianReport."],"exampleFix":"// before\nconst report = medianReport(reports)\n\n// after (guard at call site when reports may be empty)\nif (reports.length === 0) {\n  console.error('No benchmark reports collected')\n  process.exit(1)\n}\nconst report = medianReport(reports)","handlingStrategy":"validation","validationCode":"// Guard before calling medianReport.\nif (reports.length === 0) {\n  console.error('No benchmark reports collected')\n  process.exit(1)\n}\nconst report = medianReport(reports)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Ensure the collection loop runs at least once (runs >= 1, already enforced by error 13).","Do not filter reports to empty between collection and medianReport.","If calling medianReport from a test, pass a non-empty fixture."],"tags":["benchmark-check","aggregation","defensive-guard","empty-input"],"backgroundTag":null,"analyzedSha":"ac49b09b7d83ede19581fa94a8b892b07d309baf","analyzedAt":"2026-08-12T17:03:16.263Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}