{"record":{"id":"0bf18fe3003087cb","repo":"stablyai/orca","slug":"path-unsupported-benchmark-artifact-expected","errorCode":null,"errorMessage":"${path}: unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, Playwright suites, or top-level summary","messagePattern":"(.+?): unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, Playwright suites, or top-level summary","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"config/scripts/compare-benchmark-artifacts.mjs","lineNumber":91,"sourceCode":"  return JSON.parse(readFileSync(path, 'utf8'))\n}\n\nexport function normalizeBenchmarkArtifact(path, artifact = readBenchmarkArtifact(path)) {\n  if (artifact?.summaryMedianMs != null) {\n    return normalizeNumericObject(path, artifact, 'startup', artifact.summaryMedianMs, () => 'ms')\n  }\n  if (artifact?.summaryMedian != null) {\n    return normalizeNumericObject(path, artifact, 'daemon', artifact.summaryMedian, (key) =>\n      key.endsWith('Count') || key.endsWith('After') ? 'count' : 'ms'\n    )\n  }\n  if (artifact?.suites != null) {\n    return normalizePlaywrightArtifact(path, artifact)\n  }\n  if (artifact?.summary != null) {\n    return normalizeSummaryArtifact(path, artifact)\n  }\n  throw new Error(\n    `${path}: unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, Playwright suites, or top-level summary`\n  )\n}\n\nfunction artifactLabel(path, artifact) {\n  return typeof artifact?.label === 'string' && artifact.label.length > 0\n    ? artifact.label\n    : basename(path)\n}\n\nfunction normalizeNumericObject(path, artifact, kind, values, unitForKey) {\n  return {\n    kind,\n    label: artifactLabel(path, artifact),\n    metrics: Object.entries(values ?? {})\n      .filter(([, value]) => Number.isFinite(value))\n      .map(([key, value]) => ({\n        direction: 'lower-is-better',","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/stablyai/orca/blob/1136503c6a231a16dce8f921f6fadb63d181e8db/config/scripts/compare-benchmark-artifacts.mjs#L73-L109","documentation":"Thrown by normalizeBenchmarkArtifact when the parsed JSON artifact has none of the recognized top-level shapes: summaryMedianMs (startup), summaryMedian (daemon), suites (Playwright), or summary. The function falls through all four checks and rejects the file as unsupported (compare-benchmark-artifacts.mjs:76-94).","triggerScenarios":"Passing a path to a JSON file that is not a benchmark artifact (e.g., a Playwright run without suites, a raw metrics dump, a config file); passing an artifact from a version whose schema differs; passing an empty object {}.","commonSituations":"Pointing --baseline/--candidate at the wrong file (e.g., a test result JSON instead of the benchmark summary); a benchmark tool upgrade that changed output keys; an empty or partially-written artifact from a crashed run.","solutions":["Open the artifact JSON and confirm it contains one of: summaryMedianMs, summaryMedian, suites, or summary at the top level.","Re-run the benchmark producer to regenerate a well-formed artifact.","If your artifact uses a different schema, normalize it into one of the supported shapes before passing it to the script."],"exampleFix":"// before: artifact = { results: [...] }  (no recognized key)\n// after: ensure artifact has e.g. summaryMedianMs or suites\n//   { \"summaryMedianMs\": { \"cold\": 120.5 }, \"label\": \"baseline\" }","handlingStrategy":"type-guard","validationCode":"import { readFileSync } from 'node:fs'\nfunction isSupportedArtifact(artifact) {\n  return artifact != null && typeof artifact === 'object' &&\n    (artifact.summaryMedianMs != null || artifact.summaryMedian != null ||\n     artifact.suites != null || artifact.summary != null)\n}\nconst art = JSON.parse(readFileSync(path, 'utf8'))\nif (!isSupportedArtifact(art)) {\n  throw new Error(`${path} is not a supported benchmark artifact`)\n}","typeGuard":"function isSupportedArtifact(artifact) {\n  return artifact != null && typeof artifact === 'object' &&\n    (artifact.summaryMedianMs != null || artifact.summaryMedian != null ||\n     artifact.suites != null || artifact.summary != null)\n}","tryCatchPattern":"try {\n  normalizeBenchmarkArtifact(path)\n} catch (error) {\n  if (/unsupported benchmark artifact/.test(error.message)) {\n    // log and skip this artifact, or regenerate it\n  } else throw error\n}","preventionTips":["Validate the artifact shape with a type guard before passing its path to normalizeBenchmarkArtifact.","Keep benchmark producers and this consumer version-aligned so the emitted keys stay recognized."],"tags":["data-validation","benchmark","json-schema"],"backgroundTag":null,"analyzedSha":"1136503c6a231a16dce8f921f6fadb63d181e8db","analyzedAt":"2026-08-12T23:15:58.167Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}