{"record":{"id":"8eacdee8b594868e","repo":"stablyai/orca","slug":"no-comparable-benchmark-metrics-found","errorCode":null,"errorMessage":"No comparable benchmark metrics found.","messagePattern":"No comparable benchmark metrics found\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"config/scripts/compare-benchmark-artifacts.mjs","lineNumber":252,"sourceCode":"        key: baselineMetric.key,\n        reason: `unit mismatch (${formatUnitLabel(baselineMetric.unit)} vs ${formatUnitLabel(candidateMetric.unit)})`\n      })\n      continue\n    }\n    const direction = higherIsBetter.has(baselineMetric.key)\n      ? 'higher-is-better'\n      : baselineMetric.direction\n    metrics.push(compareMetric(baselineMetric, candidateMetric, direction))\n  }\n\n  for (const candidateMetric of candidate.metrics) {\n    if (!baselineMetrics.has(candidateMetric.key)) {\n      skippedMetrics.push({ key: candidateMetric.key, reason: 'missing baseline metric' })\n    }\n  }\n\n  if (metrics.length === 0) {\n    throw new Error('No comparable benchmark metrics found.')\n  }\n\n  return {\n    schemaVersion: 1,\n    createdAt: now().toISOString(),\n    title,\n    baseline: {\n      path: benchmarkDisplayPath(baselinePath),\n      label: baseline.label,\n      kind: baseline.kind\n    },\n    candidate: {\n      path: benchmarkDisplayPath(candidatePath),\n      label: candidate.label,\n      kind: candidate.kind\n    },\n    metrics,\n    skippedMetrics","sourceCodeStart":234,"sourceCodeEnd":270,"githubUrl":"https://github.com/stablyai/orca/blob/1136503c6a231a16dce8f921f6fadb63d181e8db/config/scripts/compare-benchmark-artifacts.mjs#L234-L270","documentation":"Thrown by compareBenchmarkArtifacts when, after iterating all baseline metrics, the comparable metrics list is empty. A metric is only comparable if both baseline and candidate have a finite value for the same key AND matching units; otherwise it is pushed to skippedMetrics. Zero surviving comparisons means the two artifacts cannot be compared (compare-benchmark-artifacts.mjs:222-253).","triggerScenarios":"Baseline and candidate artifacts use disjoint metric keys (e.g., a startup artifact vs a daemon artifact); all shared keys have mismatched units; all metric values are non-finite (NaN/null); comparing artifacts of different kinds that happen to share no keys.","commonSituations":"Comparing a startup benchmark (summaryMedianMs) against a daemon benchmark (summaryMedian); a schema change that renamed metric keys between baseline and candidate runs; comparing a Playwright artifact against a numeric-summary artifact with no overlapping scenario keys.","solutions":["Confirm both artifacts are the same kind (both startup, both daemon, both Playwright, or both summary).","Inspect the skippedMetrics in a try/catch to see why each shared key was dropped (missing baseline, missing candidate, or unit mismatch).","If keys legitimately differ, regenerate one artifact with the matching producer/version."],"exampleFix":"// before: baseline has keys {cold,warm}, candidate has keys {heapUsedMB}\n// after: regenerate candidate with the same startup producer so keys overlap\n//   baseline: { summaryMedianMs: { cold: 120, warm: 90 } }\n//   candidate: { summaryMedianMs: { cold: 115, warm: 88 } }","handlingStrategy":"validation","validationCode":"function sharedComparableKeys(baseline, candidate) {\n  const b = new Map(baseline.metrics.map(m => [m.key, m]))\n  return candidate.metrics.filter(m =>\n    b.has(m.key) && b.get(m.key).unit === m.unit &&\n    Number.isFinite(b.get(m.key).value) && Number.isFinite(m.value)\n  ).map(m => m.key)\n}\n// if sharedComparableKeys(baseline, candidate).length === 0, do not call compareBenchmarkArtifacts","typeGuard":null,"tryCatchPattern":"try {\n  compareBenchmarkArtifacts({ baselinePath, candidatePath })\n} catch (error) {\n  if (/No comparable benchmark metrics/.test(error.message)) {\n    // artifacts are not comparable; log and exit gracefully\n  } else throw error\n}","preventionTips":["Ensure baseline and candidate artifacts come from the same benchmark producer and kind.","Log skippedMetrics (caught via try/catch) to diagnose unit mismatches or disjoint keys."],"tags":["data-validation","benchmark","comparison"],"backgroundTag":null,"analyzedSha":"1136503c6a231a16dce8f921f6fadb63d181e8db","analyzedAt":"2026-08-12T23:15:58.167Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}