What causes "Cannot summarize zero benchmark runs" in chenglou/pretext?

If calling medianReport programmatically, guard for empty input and handle it at the call site.

chenglou/pretext · error · Error

Cannot summarize zero benchmark runs

Error message

Cannot summarize zero benchmark runs

What it means

Thrown by medianReport() if the reports array passed in is empty (length 0). The function cannot compute a median of zero runs, so it refuses rather than returning a fabricated report. In the normal benchmark-check flow this is guarded upstream: error 13 enforces runs >= 1, and the collection loop runs exactly `runs` iterations — so reaching medianReport with zero reports implies the loop body was bypassed or the array was mutated after collection.

Source

Thrown at scripts/benchmark-check.ts:181

        assertSame(
          row[metadataKey],
          firstRow[metadataKey],
          `corpusResults[${rowIndex}].${metadataKey}`,
        )
      }
    }

    for (const timingKey of CORPUS_TIMING_KEYS) {
      result[timingKey] = median(reports.map(report => report.corpusResults![rowIndex]![timingKey]))
    }

    return result
  })
}

function medianReport(reports: BenchmarkReport[]): BenchmarkReport {
  if (reports.length === 0) {
    throw new Error('Cannot summarize zero benchmark runs')
  }

  for (const [index, report] of reports.entries()) {
    if (report.status === 'error') {
      throw new Error(`Benchmark run ${index + 1} failed: ${report.message ?? 'unknown error'}`)
    }
  }

  const report: BenchmarkReport = { status: 'ready' }

  for (const key of BENCHMARK_RESULT_KEYS) {
    const rows = medianBenchmarkResults(reports, key)
    if (rows !== undefined) report[key] = rows
  }

  const corpusResults = medianCorpusResults(reports)
  if (corpusResults !== undefined) report.corpusResults = corpusResults

View on GitHub (pinned to ac49b09b7d)

Solutions

If calling medianReport programmatically, guard for empty input and handle it at the call site.
Confirm the collection loop actually pushes each report (no swallowed exceptions or early returns).
Run with --runs=1 to verify at least one report is collected before mediating.
Audit any filtering applied between collection and medianReport.

Example fix

// before
const report = medianReport(reports)

// after (guard at call site when reports may be empty)
if (reports.length === 0) {
  console.error('No benchmark reports collected')
  process.exit(1)
}
const report = medianReport(reports)

Defensive patterns

Strategy: validation

Validate before calling

// Guard before calling medianReport.
if (reports.length === 0) {
  console.error('No benchmark reports collected')
  process.exit(1)
}
const report = medianReport(reports)

Prevention

Ensure the collection loop runs at least once (runs >= 1, already enforced by error 13).
Do not filter reports to empty between collection and medianReport.
If calling medianReport from a test, pass a non-empty fixture.

When it happens

Trigger: Calling medianReport([]) directly, or a code path where the reports-collection loop pushed nothing (e.g. an early break, an exception swallowed per-iteration, or runs being mutated to 0 after the loop guard). Not reachable via the standard CLI path.

Common situations: A future refactor that filters reports before mediating and accidentally empties the list; a test that calls medianReport with an empty fixture; a bug where a run error short-circuits collection. Practically unseen in normal CLI usage.

Related errors

AI-assisted analysis of chenglou/pretext@ac49b09b7d (2026-08-12). Data as JSON: /api/errors/66193ac26d5fdfe3. Report an issue: GitHub.