{"record":{"id":"d858ee6242881cdd","repo":"chenglou/pretext","slug":"benchmark-runs-disagree-for-context-expected","errorCode":null,"errorMessage":"Benchmark runs disagree for ${context}: expected ${String(expected)}, got ${String(actual)}","messagePattern":"Benchmark runs disagree for (.+?): expected (.+?), got (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"scripts/benchmark-check.ts","lineNumber":105,"sourceCode":"}\n\nfunction parseBrowser(value: string | null): BrowserKind {\n  const browser = (value ?? process.env['BENCHMARK_CHECK_BROWSER'] ?? 'chrome').toLowerCase()\n  if (browser !== 'chrome' && browser !== 'safari') {\n    throw new Error(`Unsupported browser ${browser}; expected chrome or safari`)\n  }\n  return browser\n}\n\nfunction median(values: number[]): number {\n  const sorted = [...values].sort((a, b) => a - b)\n  const mid = Math.floor(sorted.length / 2)\n  return sorted.length % 2 === 0 ? (sorted[mid - 1]! + sorted[mid]!) / 2 : sorted[mid]!\n}\n\nfunction assertSame<T>(actual: T, expected: T, context: string): void {\n  if (actual === expected) return\n  throw new Error(\n    `Benchmark runs disagree for ${context}: expected ${String(expected)}, got ${String(actual)}`,\n  )\n}\n\nfunction medianBenchmarkResults(\n  reports: BenchmarkReport[],\n  key: typeof BENCHMARK_RESULT_KEYS[number],\n): BenchmarkResult[] | undefined {\n  const firstRows = reports[0]?.[key]\n  if (firstRows === undefined) {\n    for (let reportIndex = 1; reportIndex < reports.length; reportIndex++) {\n      assertSame(reports[reportIndex]![key], undefined, `${key}`)\n    }\n    return undefined\n  }\n  for (let reportIndex = 1; reportIndex < reports.length; reportIndex++) {\n    assertSame(reports[reportIndex]![key]?.length, firstRows.length, `${key}.length`)\n  }","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/chenglou/pretext/blob/ac49b09b7d83ede19581fa94a8b892b07d309baf/scripts/benchmark-check.ts#L87-L123","documentation":"Thrown by assertSame() during medianReport aggregation: when reducing multiple benchmark runs into a median, the harness asserts that structural metadata is identical across runs (result-array lengths, row labels, row descriptions, corpus metadata fields). If any compared value differs, it throws with the context label identifying which field disagreed. This is a correctness guard — benchmark runs over the same page must produce the same shape, only timings may vary.","triggerScenarios":"Two benchmark runs returned different result-array lengths (e.g. one run's results had 8 rows, another had 7), or a row's label/desc differed between runs, or a corpusResults metadata field (id/label/font/chars/segments/etc.) changed between runs. Indicates the benchmark page is non-deterministic in structure — e.g. a conditional block that sometimes emits a result row, or a corpus whose segment count shifted.","commonSituations":"The benchmark page was edited mid-run; a corpus file changed between runs; non-deterministic ordering in a results array; a feature flag/environment difference between runs; font availability differing between runs changing segment counts.","solutions":["Run each benchmark individually and diff the JSON outputs to find which structural field changed.","Ensure the page server, corpus files, and environment are identical across all runs (no edits between runs).","If a row is legitimately optional, make all runs include it (or none) so lengths match.","Reduce to --runs=1 to confirm the page is stable in isolation before investigating cross-run drift."],"exampleFix":"// before: assertSame throws on any structural drift\nassertSame(reports[reportIndex]![key]?.length, firstRows.length, `${key}.length`)\n\n// after (not recommended for correctness — prefer fixing the page determinism)\n// If structural variance is expected, log and skip rather than throw:\nif (reports[reportIndex]![key]?.length !== firstRows.length) {\n  console.warn(`Benchmark run ${reportIndex+1} ${key}.length differs; skipping median`)\n  return firstRows\n}","handlingStrategy":"validation","validationCode":"// Before mediating, confirm structural consistency.\nfunction structurallyConsistent(reports: BenchmarkReport[]): boolean {\n  const first = reports[0]\n  if (first === undefined) return true\n  return reports.every(r => (r.results?.length ?? 0) === (first.results?.length ?? 0))\n}","typeGuard":null,"tryCatchPattern":"// Drop divergent runs and warn rather than aborting if determinism cannot be guaranteed.\nfunction medianReportLenient(reports: BenchmarkReport[]): BenchmarkReport {\n  if (reports.length === 0) throw new Error('Cannot summarize zero benchmark runs')\n  // ... find the majority shape and filter outliers before assertSame\n}","preventionTips":["Do not edit the page or corpus files between runs.","Keep the environment (fonts, viewport, foreground status) identical across runs.","If a row is conditionally emitted, make its presence deterministic.","Run --runs=1 first to confirm the page is structurally stable before mediating."],"tags":["benchmark-check","correctness-guard","non-determinism","aggregation"],"backgroundTag":null,"analyzedSha":"ac49b09b7d83ede19581fa94a8b892b07d309baf","analyzedAt":"2026-08-12T17:03:16.263Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}