chenglou/pretext · error · Error

Benchmark run ${index + 1} failed: ${report.message ?? 'unkn

Error message

Benchmark run ${index + 1} failed: ${report.message ?? 'unknown error'}

What it means

Thrown by medianReport() while scanning collected reports: if any report has status === 'error', the whole benchmark is considered failed and this throw surfaces that run's error message (or 'unknown error' if message is absent). It fires during aggregation, after all runs have been collected, so the failing run's own printed output will precede this. The 1-based index in the message identifies which run failed.

Source

Thrown at scripts/benchmark-check.ts:186

      }
    }

    for (const timingKey of CORPUS_TIMING_KEYS) {
      result[timingKey] = median(reports.map(report => report.corpusResults![rowIndex]![timingKey]))
    }

    return result
  })
}

function medianReport(reports: BenchmarkReport[]): BenchmarkReport {
  if (reports.length === 0) {
    throw new Error('Cannot summarize zero benchmark runs')
  }

  for (const [index, report] of reports.entries()) {
    if (report.status === 'error') {
      throw new Error(`Benchmark run ${index + 1} failed: ${report.message ?? 'unknown error'}`)
    }
  }

  const report: BenchmarkReport = { status: 'ready' }

  for (const key of BENCHMARK_RESULT_KEYS) {
    const rows = medianBenchmarkResults(reports, key)
    if (rows !== undefined) report[key] = rows
  }

  const corpusResults = medianCorpusResults(reports)
  if (corpusResults !== undefined) report.corpusResults = corpusResults

  return report
}

function printReport(report: BenchmarkReport): void {
  if (report.status === 'error') {

View on GitHub (pinned to ac49b09b7d)

Solutions

Find the individual run output above this error — the failing run's message is printed before aggregation.
Re-run with --runs=1 to see the page error in isolation and without the aggregation noise.
Address the underlying page error (font loading, viewport size, corpus availability) rather than this aggregate throw.
If the failure is transient (background tab throttling), run in the foreground — benchmark-check already forces foreground, so check for competing automation or display-sleep.

Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-scan for error-status reports and surface them before aggregation.
const failed = reports.find(r => r.status === 'error')
if (failed !== undefined) {
  throw new Error(`Benchmark run failed: ${failed.message ?? 'unknown error'}`)
}

Type guard

function isErrorReport(report: BenchmarkReport): report is BenchmarkReport & { status: 'error' } {
  return report.status === 'error'
}

Try / catch

// medianReport already throws; wrap the call to exit gracefully with context.
try {
  const report = medianReport(reports)
} catch (error) {
  console.error(error instanceof Error ? error.message : String(error))
  process.exit(1)
}

Prevention

Investigate the failing run's own printed output before this aggregate error.
Run --runs=1 to reproduce the page-side error in isolation.
Keep the environment stable (fonts, foreground, viewport) across runs.
Address the underlying page error rather than suppressing this throw.

When it happens

Trigger: The benchmark page (pages/benchmark.ts) reported status 'error' for one of the runs — e.g. a measurement threw, a corpus failed to load, or the page hit an internal assertion. The harness collects all runs, then medianReport refuses to median-merge because one is unusable.

Common situations: A transient page-side error (font not ready, canvas context lost); a real regression that makes the benchmark page throw; an environment difference (missing font, tiny viewport) causing the page to error on one run but not others; throttling/backgrounding causing a timeout that surfaces as an error report.

Related errors

AI-assisted analysis of chenglou/pretext@ac49b09b7d (2026-08-12). Data as JSON: /api/errors/25b6a4cfd3e5025a. Report an issue: GitHub.