stablyai/orca · error · Error

${path}: unsupported benchmark artifact; expected summaryMed

Error message

${path}: unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, Playwright suites, or top-level summary

What it means

Thrown by normalizeBenchmarkArtifact when the parsed JSON artifact has none of the recognized top-level shapes: summaryMedianMs (startup), summaryMedian (daemon), suites (Playwright), or summary. The function falls through all four checks and rejects the file as unsupported (compare-benchmark-artifacts.mjs:76-94).

Source

Thrown at config/scripts/compare-benchmark-artifacts.mjs:91

  return JSON.parse(readFileSync(path, 'utf8'))
}

export function normalizeBenchmarkArtifact(path, artifact = readBenchmarkArtifact(path)) {
  if (artifact?.summaryMedianMs != null) {
    return normalizeNumericObject(path, artifact, 'startup', artifact.summaryMedianMs, () => 'ms')
  }
  if (artifact?.summaryMedian != null) {
    return normalizeNumericObject(path, artifact, 'daemon', artifact.summaryMedian, (key) =>
      key.endsWith('Count') || key.endsWith('After') ? 'count' : 'ms'
    )
  }
  if (artifact?.suites != null) {
    return normalizePlaywrightArtifact(path, artifact)
  }
  if (artifact?.summary != null) {
    return normalizeSummaryArtifact(path, artifact)
  }
  throw new Error(
    `${path}: unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, Playwright suites, or top-level summary`
  )
}

function artifactLabel(path, artifact) {
  return typeof artifact?.label === 'string' && artifact.label.length > 0
    ? artifact.label
    : basename(path)
}

function normalizeNumericObject(path, artifact, kind, values, unitForKey) {
  return {
    kind,
    label: artifactLabel(path, artifact),
    metrics: Object.entries(values ?? {})
      .filter(([, value]) => Number.isFinite(value))
      .map(([key, value]) => ({
        direction: 'lower-is-better',

View on GitHub (pinned to 1136503c6a)

Solutions

1. Open the artifact JSON and confirm it contains one of: summaryMedianMs, summaryMedian, suites, or summary at the top level.
2. Re-run the benchmark producer to regenerate a well-formed artifact.
3. If your artifact uses a different schema, normalize it into one of the supported shapes before passing it to the script.

Example fix

// before: artifact = { results: [...] }  (no recognized key)
// after: ensure artifact has e.g. summaryMedianMs or suites
//   { "summaryMedianMs": { "cold": 120.5 }, "label": "baseline" }

import { readFileSync } from 'node:fs'
function isSupportedArtifact(artifact) {
  return artifact != null && typeof artifact === 'object' &&
    (artifact.summaryMedianMs != null || artifact.summaryMedian != null ||
     artifact.suites != null || artifact.summary != null)
}
const art = JSON.parse(readFileSync(path, 'utf8'))
if (!isSupportedArtifact(art)) {
  throw new Error(`${path} is not a supported benchmark artifact`)
}

function isSupportedArtifact(artifact) {
  return artifact != null && typeof artifact === 'object' &&
    (artifact.summaryMedianMs != null || artifact.summaryMedian != null ||
     artifact.suites != null || artifact.summary != null)
}

try {
  normalizeBenchmarkArtifact(path)
} catch (error) {
  if (/unsupported benchmark artifact/.test(error.message)) {
    // log and skip this artifact, or regenerate it
  } else throw error
}

When it happens

Trigger: Passing a path to a JSON file that is not a benchmark artifact (e.g., a Playwright run without suites, a raw metrics dump, a config file); passing an artifact from a version whose schema differs; passing an empty object {}.

Common situations: Pointing --baseline/--candidate at the wrong file (e.g., a test result JSON instead of the benchmark summary); a benchmark tool upgrade that changed output keys; an empty or partially-written artifact from a crashed run.

Related errors

• No comparable benchmark metrics found.
• No opencode terminal perf annotations found in the provided
• advertised-url-watcher.ts no longer contains \`${marker}\`
• ${name} must be a positive integer, received ${value}
• ORCA_ADVERTISED_URL_BENCH_ROUNDS must be even

AI-assisted analysis of stablyai/orca@1136503c6a (2026-08-12). Data as JSON: /api/errors/0bf18fe3003087cb. Report an issue: GitHub.