chenglou/pretext · error · Error

Benchmark runs disagree for ${context}: expected ${String(ex

Error message

Benchmark runs disagree for ${context}: expected ${String(expected)}, got ${String(actual)}

What it means

Thrown by assertSame() during medianReport aggregation: when reducing multiple benchmark runs into a median, the harness asserts that structural metadata is identical across runs (result-array lengths, row labels, row descriptions, corpus metadata fields). If any compared value differs, it throws with the context label identifying which field disagreed. This is a correctness guard — benchmark runs over the same page must produce the same shape, only timings may vary.

Source

Thrown at scripts/benchmark-check.ts:105

}

function parseBrowser(value: string | null): BrowserKind {
  const browser = (value ?? process.env['BENCHMARK_CHECK_BROWSER'] ?? 'chrome').toLowerCase()
  if (browser !== 'chrome' && browser !== 'safari') {
    throw new Error(`Unsupported browser ${browser}; expected chrome or safari`)
  }
  return browser
}

function median(values: number[]): number {
  const sorted = [...values].sort((a, b) => a - b)
  const mid = Math.floor(sorted.length / 2)
  return sorted.length % 2 === 0 ? (sorted[mid - 1]! + sorted[mid]!) / 2 : sorted[mid]!
}

function assertSame<T>(actual: T, expected: T, context: string): void {
  if (actual === expected) return
  throw new Error(
    `Benchmark runs disagree for ${context}: expected ${String(expected)}, got ${String(actual)}`,
  )
}

function medianBenchmarkResults(
  reports: BenchmarkReport[],
  key: typeof BENCHMARK_RESULT_KEYS[number],
): BenchmarkResult[] | undefined {
  const firstRows = reports[0]?.[key]
  if (firstRows === undefined) {
    for (let reportIndex = 1; reportIndex < reports.length; reportIndex++) {
      assertSame(reports[reportIndex]![key], undefined, `${key}`)
    }
    return undefined
  }
  for (let reportIndex = 1; reportIndex < reports.length; reportIndex++) {
    assertSame(reports[reportIndex]![key]?.length, firstRows.length, `${key}.length`)
  }

View on GitHub (pinned to ac49b09b7d)

Solutions

Run each benchmark individually and diff the JSON outputs to find which structural field changed.
Ensure the page server, corpus files, and environment are identical across all runs (no edits between runs).
If a row is legitimately optional, make all runs include it (or none) so lengths match.
Reduce to --runs=1 to confirm the page is stable in isolation before investigating cross-run drift.

Example fix

// before: assertSame throws on any structural drift
assertSame(reports[reportIndex]![key]?.length, firstRows.length, `${key}.length`)

// after (not recommended for correctness — prefer fixing the page determinism)
// If structural variance is expected, log and skip rather than throw:
if (reports[reportIndex]![key]?.length !== firstRows.length) {
  console.warn(`Benchmark run ${reportIndex+1} ${key}.length differs; skipping median`)
  return firstRows
}

Defensive patterns

Strategy: validation

Validate before calling

// Before mediating, confirm structural consistency.
function structurallyConsistent(reports: BenchmarkReport[]): boolean {
  const first = reports[0]
  if (first === undefined) return true
  return reports.every(r => (r.results?.length ?? 0) === (first.results?.length ?? 0))
}

Try / catch

// Drop divergent runs and warn rather than aborting if determinism cannot be guaranteed.
function medianReportLenient(reports: BenchmarkReport[]): BenchmarkReport {
  if (reports.length === 0) throw new Error('Cannot summarize zero benchmark runs')
  // ... find the majority shape and filter outliers before assertSame
}

Prevention

Do not edit the page or corpus files between runs.
Keep the environment (fonts, viewport, foreground status) identical across runs.
If a row is conditionally emitted, make its presence deterministic.
Run --runs=1 first to confirm the page is structurally stable before mediating.

When it happens

Trigger: Two benchmark runs returned different result-array lengths (e.g. one run's results had 8 rows, another had 7), or a row's label/desc differed between runs, or a corpusResults metadata field (id/label/font/chars/segments/etc.) changed between runs. Indicates the benchmark page is non-deterministic in structure — e.g. a conditional block that sometimes emits a result row, or a corpus whose segment count shifted.

Common situations: The benchmark page was edited mid-run; a corpus file changed between runs; non-deterministic ordering in a results array; a feature flag/environment difference between runs; font availability differing between runs changing segment counts.

Related errors

AI-assisted analysis of chenglou/pretext@ac49b09b7d (2026-08-12). Data as JSON: /api/errors/d858ee6242881cdd. Report an issue: GitHub.