can1357/oh-my-pi · error

Terminal-Bench dataset source is not a directory: ${source}

Error message

Terminal-Bench dataset source is not a directory: ${source}

What it means

resolveDataset classifies the source via directoryStat: 'directory', 'other' (exists but not a directory), or missing. When the source exists but is a file/symlink-to-file/special file, it cannot be used as a dataset, so this error is thrown.

Source

Thrown at packages/metaharness/src/tb/dataset.ts:55

	const basename = trimmed.slice(trimmed.lastIndexOf("/") + 1).replace(/\.git$/, "");
	if (!basename) throw new Error(`Cannot determine repository name from dataset source: ${source}`);
	return basename;
}

function gitError(action: string, source: string, stderr: Uint8Array): Error {
	const detail = new TextDecoder().decode(stderr).trim();
	return new Error(`${action} ${source} failed${detail ? `: ${detail}` : ""}`);
}

export async function resolveDataset(source: string, cacheDir: string): Promise<string> {
	const sourceKind = await directoryStat(source);
	if (sourceKind === "directory") {
		const absoluteSource = path.resolve(source);
		const nestedTasks = path.join(absoluteSource, "tasks");
		const tasksDir = (await directoryStat(nestedTasks)) === "directory" ? nestedTasks : absoluteSource;
		return validateTasksDir(tasksDir);
	}
	if (sourceKind === "other") throw new Error(`Terminal-Bench dataset source is not a directory: ${source}`);

	const isGitUrl = source.startsWith("http://") || source.startsWith("https://") || source.startsWith("git@");
	if (!isGitUrl) throw new Error(`Terminal-Bench dataset source does not exist: ${source}`);

	const absoluteCache = path.resolve(cacheDir);
	await fs.mkdir(absoluteCache, { recursive: true });
	const checkoutDir = path.join(absoluteCache, repositoryName(source));
	const gitDir = path.join(checkoutDir, ".git");
	if ((await directoryStat(gitDir)) === "directory") {
		const result = await $`git pull --ff-only`.cwd(checkoutDir).quiet().nothrow();
		if (result.exitCode !== 0) throw gitError("Updating", source, result.stderr);
	} else {
		const result = await $`git clone --depth 1 ${source} ${checkoutDir}`.quiet().nothrow();
		if (result.exitCode !== 0) throw gitError("Cloning", source, result.stderr);
	}

	return validateTasksDir(path.join(checkoutDir, "tasks"));
}

View on GitHub (pinned to 9690622007)

Solutions

1. Point --dataset at a directory (or its tasks/ subdir) containing per-task task.toml files
2. Extract the archive first and pass the extracted directory
3. Fix shell globs so they expand to the dataset directory

Example fix

// before
tb run --dataset tb-tasks.tar.gz
// after
tar xzf tb-tasks.tar.gz
tb run --dataset ./tb-tasks

import { stat } from "node:fs/promises";
const s = await stat(source);
if (!s.isDirectory()) throw new Error(`${source} must be a directory (extract archives first)`);

When it happens

Trigger: Calling resolveDataset (from main/tasksDir) with a --dataset value that resolves to a regular file (e.g. an archive, a task.toml itself, or a tarball path).

Common situations: Passing a zip/tar of the dataset instead of an extracted directory; passing a single task's task.toml; shell glob expanding to a file.

Related errors

• Terminal-Bench tasks directory does not exist: ${absolute}
• Terminal-Bench tasks directory contains no task.toml files:
• Terminal-Bench dataset source does not exist: ${source}
• unknown filetype: {ft_debug}
• Is a directory

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/1777cfda47384f8d. Report an issue: GitHub.