stablyai/orca · error · Error

No comparable benchmark metrics found.

Error message

No comparable benchmark metrics found.

What it means

Thrown by compareBenchmarkArtifacts when, after iterating all baseline metrics, the comparable metrics list is empty. A metric is only comparable if both baseline and candidate have a finite value for the same key AND matching units; otherwise it is pushed to skippedMetrics. Zero surviving comparisons means the two artifacts cannot be compared (compare-benchmark-artifacts.mjs:222-253).

Source

Thrown at config/scripts/compare-benchmark-artifacts.mjs:252

        key: baselineMetric.key,
        reason: `unit mismatch (${formatUnitLabel(baselineMetric.unit)} vs ${formatUnitLabel(candidateMetric.unit)})`
      })
      continue
    }
    const direction = higherIsBetter.has(baselineMetric.key)
      ? 'higher-is-better'
      : baselineMetric.direction
    metrics.push(compareMetric(baselineMetric, candidateMetric, direction))
  }

  for (const candidateMetric of candidate.metrics) {
    if (!baselineMetrics.has(candidateMetric.key)) {
      skippedMetrics.push({ key: candidateMetric.key, reason: 'missing baseline metric' })
    }
  }

  if (metrics.length === 0) {
    throw new Error('No comparable benchmark metrics found.')
  }

  return {
    schemaVersion: 1,
    createdAt: now().toISOString(),
    title,
    baseline: {
      path: benchmarkDisplayPath(baselinePath),
      label: baseline.label,
      kind: baseline.kind
    },
    candidate: {
      path: benchmarkDisplayPath(candidatePath),
      label: candidate.label,
      kind: candidate.kind
    },
    metrics,
    skippedMetrics

View on GitHub (pinned to 1136503c6a)

Solutions

1. Confirm both artifacts are the same kind (both startup, both daemon, both Playwright, or both summary).
2. Inspect the skippedMetrics in a try/catch to see why each shared key was dropped (missing baseline, missing candidate, or unit mismatch).
3. If keys legitimately differ, regenerate one artifact with the matching producer/version.

Example fix

// before: baseline has keys {cold,warm}, candidate has keys {heapUsedMB}
// after: regenerate candidate with the same startup producer so keys overlap
//   baseline: { summaryMedianMs: { cold: 120, warm: 90 } }
//   candidate: { summaryMedianMs: { cold: 115, warm: 88 } }

function sharedComparableKeys(baseline, candidate) {
  const b = new Map(baseline.metrics.map(m => [m.key, m]))
  return candidate.metrics.filter(m =>
    b.has(m.key) && b.get(m.key).unit === m.unit &&
    Number.isFinite(b.get(m.key).value) && Number.isFinite(m.value)
  ).map(m => m.key)
}
// if sharedComparableKeys(baseline, candidate).length === 0, do not call compareBenchmarkArtifacts

try {
  compareBenchmarkArtifacts({ baselinePath, candidatePath })
} catch (error) {
  if (/No comparable benchmark metrics/.test(error.message)) {
    // artifacts are not comparable; log and exit gracefully
  } else throw error
}

When it happens

Trigger: Baseline and candidate artifacts use disjoint metric keys (e.g., a startup artifact vs a daemon artifact); all shared keys have mismatched units; all metric values are non-finite (NaN/null); comparing artifacts of different kinds that happen to share no keys.

Common situations: Comparing a startup benchmark (summaryMedianMs) against a daemon benchmark (summaryMedian); a schema change that renamed metric keys between baseline and candidate runs; comparing a Playwright artifact against a numeric-summary artifact with no overlapping scenario keys.

Related errors

• ${path}: unsupported benchmark artifact; expected summaryMed
• No opencode terminal perf annotations found in the provided
• advertised-url-watcher.ts no longer contains \`${marker}\`
• ${name} must be a positive integer, received ${value}
• ORCA_ADVERTISED_URL_BENCH_ROUNDS must be even

AI-assisted analysis of stablyai/orca@1136503c6a (2026-08-12). Data as JSON: /api/errors/8eacdee8b594868e. Report an issue: GitHub.