{"record":{"id":"0530ded140840784","repo":"can1357/oh-my-pi","slug":"unsupported-benchmark-benchmark","errorCode":null,"errorMessage":"unsupported benchmark: ${benchmark}","messagePattern":"unsupported benchmark: (.+?)","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/metaharness/src/server.ts","lineNumber":384,"sourceCode":"\t\t\t\tclient.state = SseState.Closed;\n\t\t\t\tsse.delete(client);\n\t\t\t},\n\t\t});\n\t\treturn new Response(stream, {\n\t\t\theaders: {\n\t\t\t\t\"content-type\": \"text/event-stream\",\n\t\t\t\t\"cache-control\": \"no-cache\",\n\t\t\t\tconnection: \"keep-alive\",\n\t\t\t},\n\t\t});\n\t}\n\n\t/** Launch any supported benchmark and register it in the uniform run store. */\n\tlaunch(request: LaunchRequest): { jobName: string; pid: number } {\n\t\tif (!request.model) throw new Error(\"model is required\");\n\t\tconst benchmark = request.benchmark ?? \"harbor\";\n\t\tif (benchmark !== \"harbor\" && benchmark !== \"edit\" && benchmark !== \"snapcompact\") {\n\t\t\tthrow new Error(`unsupported benchmark: ${benchmark}`);\n\t\t}\n\t\tconst dataset =\n\t\t\trequest.dataset ??\n\t\t\t(benchmark === \"harbor\" ? \"terminal-bench@2.0\" : benchmark === \"edit\" ? \"typescript-edit\" : \"squad-dev\");\n\t\tconst stamp = new Date().toISOString().replace(/[:.]/g, \"-\").slice(0, 19);\n\t\tconst modelSlug = request.model.replace(/[^a-zA-Z0-9]+/g, \"-\");\n\t\tconst jobName = request.jobName ?? `${modelSlug}-${stamp}`;\n\t\tif (this.#children.has(jobName) || this.#store.getRun(jobName)?.status === \"running\") {\n\t\t\tthrow new Error(`run ${jobName} is already running`);\n\t\t}\n\t\tconst jobDir = path.join(this.jobsDir, jobName);\n\t\tfs.mkdirSync(jobDir, { recursive: true });\n\n\t\tlet argv: string[];\n\t\tlet cwd: string;\n\t\tif (benchmark === \"edit\") {\n\t\t\tcwd = PKG_DIR;\n\t\t\targv = [\"bun\", \"adapters/edit/cli.ts\", \"--model\", request.model, \"--output\", path.join(jobDir, \"result.json\")];","sourceCodeStart":366,"sourceCodeEnd":402,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/metaharness/src/server.ts#L366-L402","documentation":"launch() only supports three benchmark names: 'harbor', 'edit', and 'snapcompact' (defaulting to 'harbor' when omitted). Any other benchmark string is rejected. This error means request.benchmark was set to a value outside that whitelist — likely a typo or a benchmark this manager build does not route.","triggerScenarios":"launch({ benchmark: 'terminal-bench' }) instead of 'harbor'; passing a dataset name ('squad-dev') as the benchmark; an uppercase or whitespace-padded value like 'Harbor '; a benchmark removed/renamed in this version.","commonSituations":"Migrating scripts from another harness with different benchmark names; case mismatch after copying from docs; guessing the benchmark id from a dataset name.","solutions":["Use one of the supported strings exactly: 'harbor', 'edit', or 'snapcompact'.","Omit request.benchmark entirely to get the default 'harbor'.","Trim/case-normalize user input before passing it (benchmark.trim().toLowerCase()).","Put the dataset in request.dataset, not benchmark — e.g. launch({ benchmark: 'harbor', dataset: 'terminal-bench@2.0' })."],"exampleFix":"// before\nserver.launch({ benchmark: \"terminal-bench\", model: \"m1\" });\n// after\nserver.launch({ benchmark: \"harbor\", dataset: \"terminal-bench@2.0\", model: \"m1\" });","handlingStrategy":"validation","validationCode":"const SUPPORTED = new Set([\"harbor\", \"edit\", \"snapcompact\"]);\nconst benchmark = (req.benchmark ?? \"harbor\").trim().toLowerCase();\nif (!SUPPORTED.has(benchmark)) {\n  throw new Error(`benchmark must be one of ${[...SUPPORTED].join(\", \")}`);\n}","typeGuard":"const isBenchmark = (v: unknown): v is \"harbor\" | \"edit\" | \"snapcompact\" =>\n  v === \"harbor\" || v === \"edit\" || v === \"snapcompact\";","tryCatchPattern":"try {\n  server.launch(req);\n} catch (err) {\n  if (err instanceof Error && err.message.startsWith(\"unsupported benchmark\")) {\n    console.error(`Got '${req.benchmark}'; supported: harbor, edit, snapcompact`);\n  }\n  throw err;\n}","preventionTips":["Keep the whitelist in a shared const/type so callers autocomplete valid names.","Remember dataset names (terminal-bench@2.0, squad-dev) go in dataset, not benchmark.","Normalize case/whitespace on user-supplied benchmark values.","Omit benchmark to get the 'harbor' default when unsure."],"tags":["validation","enum-value","typo"],"backgroundTag":"unsupported-enum-value","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}