can1357/oh-my-pi · error · Error

unsupported benchmark: ${benchmark}

Error message

unsupported benchmark: ${benchmark}

What it means

launch() only supports three benchmark names: 'harbor', 'edit', and 'snapcompact' (defaulting to 'harbor' when omitted). Any other benchmark string is rejected. This error means request.benchmark was set to a value outside that whitelist — likely a typo or a benchmark this manager build does not route.

Source

Thrown at packages/metaharness/src/server.ts:384

				client.state = SseState.Closed;
				sse.delete(client);
			},
		});
		return new Response(stream, {
			headers: {
				"content-type": "text/event-stream",
				"cache-control": "no-cache",
				connection: "keep-alive",
			},
		});
	}

	/** Launch any supported benchmark and register it in the uniform run store. */
	launch(request: LaunchRequest): { jobName: string; pid: number } {
		if (!request.model) throw new Error("model is required");
		const benchmark = request.benchmark ?? "harbor";
		if (benchmark !== "harbor" && benchmark !== "edit" && benchmark !== "snapcompact") {
			throw new Error(`unsupported benchmark: ${benchmark}`);
		}
		const dataset =
			request.dataset ??
			(benchmark === "harbor" ? "terminal-bench@2.0" : benchmark === "edit" ? "typescript-edit" : "squad-dev");
		const stamp = new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19);
		const modelSlug = request.model.replace(/[^a-zA-Z0-9]+/g, "-");
		const jobName = request.jobName ?? `${modelSlug}-${stamp}`;
		if (this.#children.has(jobName) || this.#store.getRun(jobName)?.status === "running") {
			throw new Error(`run ${jobName} is already running`);
		}
		const jobDir = path.join(this.jobsDir, jobName);
		fs.mkdirSync(jobDir, { recursive: true });

		let argv: string[];
		let cwd: string;
		if (benchmark === "edit") {
			cwd = PKG_DIR;
			argv = ["bun", "adapters/edit/cli.ts", "--model", request.model, "--output", path.join(jobDir, "result.json")];

View on GitHub (pinned to 9690622007)

Solutions

1. Use one of the supported strings exactly: 'harbor', 'edit', or 'snapcompact'.
2. Omit request.benchmark entirely to get the default 'harbor'.
3. Trim/case-normalize user input before passing it (benchmark.trim().toLowerCase()).
4. Put the dataset in request.dataset, not benchmark — e.g. launch({ benchmark: 'harbor', dataset: 'terminal-bench@2.0' }).

Example fix

// before
server.launch({ benchmark: "terminal-bench", model: "m1" });
// after
server.launch({ benchmark: "harbor", dataset: "terminal-bench@2.0", model: "m1" });

const SUPPORTED = new Set(["harbor", "edit", "snapcompact"]);
const benchmark = (req.benchmark ?? "harbor").trim().toLowerCase();
if (!SUPPORTED.has(benchmark)) {
  throw new Error(`benchmark must be one of ${[...SUPPORTED].join(", ")}`);
}

const isBenchmark = (v: unknown): v is "harbor" | "edit" | "snapcompact" =>
  v === "harbor" || v === "edit" || v === "snapcompact";

try {
  server.launch(req);
} catch (err) {
  if (err instanceof Error && err.message.startsWith("unsupported benchmark")) {
    console.error(`Got '${req.benchmark}'; supported: harbor, edit, snapcompact`);
  }
  throw err;
}

When it happens

Trigger: launch({ benchmark: 'terminal-bench' }) instead of 'harbor'; passing a dataset name ('squad-dev') as the benchmark; an uppercase or whitespace-padded value like 'Harbor '; a benchmark removed/renamed in this version.

Common situations: Migrating scripts from another harness with different benchmark names; case mismatch after copying from docs; guessing the benchmark id from a dataset name.

Related errors

• Unsupported launch key ${rawKey}
• Unsupported language '{value}'. Supported: {}
• Unable to infer language from file extension: {}. Specify `l
• Invalid pattern: {err}
• Failed to load tree-sitter language: {err}

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/0530ded140840784. Report an issue: GitHub.