can1357/oh-my-pi · error

Terminal-Bench dataset source does not exist: ${source}

Error message

Terminal-Bench dataset source does not exist: ${source}

What it means

When the dataset source is neither a directory nor another existing filesystem entry, resolveDataset checks whether it looks like a git URL (http://, https://, git@). If it does not, there is nothing to resolve from, so it throws this error.

Source

Thrown at packages/metaharness/src/tb/dataset.ts:58

}

function gitError(action: string, source: string, stderr: Uint8Array): Error {
	const detail = new TextDecoder().decode(stderr).trim();
	return new Error(`${action} ${source} failed${detail ? `: ${detail}` : ""}`);
}

export async function resolveDataset(source: string, cacheDir: string): Promise<string> {
	const sourceKind = await directoryStat(source);
	if (sourceKind === "directory") {
		const absoluteSource = path.resolve(source);
		const nestedTasks = path.join(absoluteSource, "tasks");
		const tasksDir = (await directoryStat(nestedTasks)) === "directory" ? nestedTasks : absoluteSource;
		return validateTasksDir(tasksDir);
	}
	if (sourceKind === "other") throw new Error(`Terminal-Bench dataset source is not a directory: ${source}`);

	const isGitUrl = source.startsWith("http://") || source.startsWith("https://") || source.startsWith("git@");
	if (!isGitUrl) throw new Error(`Terminal-Bench dataset source does not exist: ${source}`);

	const absoluteCache = path.resolve(cacheDir);
	await fs.mkdir(absoluteCache, { recursive: true });
	const checkoutDir = path.join(absoluteCache, repositoryName(source));
	const gitDir = path.join(checkoutDir, ".git");
	if ((await directoryStat(gitDir)) === "directory") {
		const result = await $`git pull --ff-only`.cwd(checkoutDir).quiet().nothrow();
		if (result.exitCode !== 0) throw gitError("Updating", source, result.stderr);
	} else {
		const result = await $`git clone --depth 1 ${source} ${checkoutDir}`.quiet().nothrow();
		if (result.exitCode !== 0) throw gitError("Cloning", source, result.stderr);
	}

	return validateTasksDir(path.join(checkoutDir, "tasks"));
}

function asRecord(value: unknown): Record<string, unknown> {
	if (typeof value !== "object" || value === null || Array.isArray(value)) return {};

View on GitHub (pinned to 9690622007)

Solutions

1. Use an existing local directory, or a git URL starting with http://, https://, or git@
2. Convert ssh remotes to https or git@ form, e.g. git@github.com:owner/repo.git
3. Fix path typos or run from the intended working directory

Example fix

// before
tb run --dataset gitssh:owner/repo.git   # not recognized, doesn't exist
// after
tb run --dataset git@github.com:owner/repo.git

import { existsSync } from "node:fs";
const isGitUrl = /^(https?:\/\/|git@)/.test(source);
if (!existsSync(source) && !isGitUrl) throw new Error(`Dataset source not found and not a git URL: ${source}`);

When it happens

Trigger: Calling resolveDataset with a non-existent path that is not a git URL — e.g. a typo'd local path, ssh-style 'user@host:path' (not matched), or a git:// scheme URL.

Common situations: Typos in a local dataset path; using scp-style SSH remotes or git:// scheme which the URL check doesn't recognize; deleted cache with a relative path resolved from the wrong cwd.

Related errors

• Terminal-Bench tasks directory does not exist: ${absolute}
• Terminal-Bench tasks directory contains no task.toml files:
• Cannot determine repository name from dataset source: ${sour
• Terminal-Bench dataset source is not a directory: ${source}
• unknown filetype: {ft_debug}

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/2ee47fdece568872. Report an issue: GitHub.