What causes "'{search_dir}' is not a directory." in microsoft/autogen?

Confirm the path is a directory and exists: `ls -d `.

microsoft/autogen · error · ValueError

'{search_dir}' is not a directory.

Error message

'{search_dir}' is not a directory.

What it means

agbench's tabulate command validates its search_dir argument up front: after abspath normalization, if the path is not an existing directory it raises ValueError. This is an input validation error before the upward directory walk for the tabulate script begins.

Source

Thrown at python/packages/agbench/src/agbench/tabulate_cmd.py:37

SUCCESS_STRINGS = [
    "ALL TESTS PASSED !#!#",
]

COMPLETED_STRINGS = [
    "SCENARIO.PY COMPLETE !#!#",
]

EXCLUDE_DIR_NAMES = ["__pycache__"]

TIMER_REGEX = r"RUNTIME:\s*([\d.]+) !#!#"


def find_tabulate_module(search_dir: str, stop_dir: Optional[str] = None) -> Optional[str]:
    """Hunt for the tabulate script."""

    search_dir = os.path.abspath(search_dir)
    if not os.path.isdir(search_dir):
        raise ValueError(f"'{search_dir}' is not a directory.")

    stop_dir = None if stop_dir is None else os.path.abspath(stop_dir)

    while True:
        path = os.path.join(search_dir, TABULATE_FILE)
        if os.path.isfile(path):
            return path

        path = os.path.join(search_dir, "Scripts", TABULATE_FILE)
        if os.path.isfile(path):
            return path

        path = os.path.join(search_dir, "scripts", TABULATE_FILE)
        if os.path.isfile(path):
            return path

        # Stop if we hit the stop_dir
        if search_dir == stop_dir:

View on GitHub (pinned to 027ecf0a37)

Solutions

Confirm the path is a directory and exists: `ls -d <path>`.
Pass the benchmark's output/run directory, not an individual results file.
Run the benchmark first if the results directory has not been created yet.

Example fix

# before
agbench tabulate results/run1.json

# after
agbench tabulate results/run1/

Defensive patterns

Strategy: validation

Validate before calling

import os
search_dir = os.path.abspath(search_dir)
if not os.path.isdir(search_dir):
    raise SystemExit(f"{search_dir} is not a directory; pass the benchmark results directory")

Try / catch

try:
    tabulate(search_dir)
except ValueError as e:
    if "is not a directory" in str(e):
        sys.exit("Pass the run/results directory to tabulate, not a file")
    raise

Prevention

Chain commands so tabulate only runs after a successful benchmark run produced the directory.
Use os.path.dirname on result file paths to derive the directory programmatically.

When it happens

Trigger: Calling the tabulate CLI/function with a results directory path that does not exist or is actually a file, e.g. pointing at a results.json file instead of its parent folder, or a typo'd/benchmark-not-yet-run directory.

Common situations: Tabulating before the benchmark run created the output directory, passing a file where a directory is expected, relative paths from the wrong cwd, or cleaned-up temp results.

Related errors

AI-assisted analysis of microsoft/autogen@027ecf0a37 (2026-08-15). Data as JSON: /api/errors/95beee13ed675fba. Report an issue: GitHub.