unslothai/unsloth · error · FileNotFoundError

data_dir is not a directory: {data_dir}

Error message

data_dir is not a directory: {data_dir}

What it means

The dataset builder expands the data_dir path and requires it to be an existing directory. A missing or non-directory path raises FileNotFoundError before any image discovery runs. This is deliberately filesystem-only so it is unit-testable without torch.

Source

Thrown at studio/backend/core/training/diffusion_train_common.py:1382

      3. ``instance_prompt`` (dreambooth) for any remaining image.

    A sidecar wins over the metadata row because it is the user's explicit per-image edit
    (the labeling grid writes a .txt sidecar), which must override the bulk metadata file.
    Must agree with ``routes.training._image_record``, which resolves captions the same way.

    Images with no caption from any source are skipped. Pure filesystem + JSON, so it is
    unit-testable without torch. Raises FileNotFoundError for a missing dir and ValueError
    when nothing is captionable.

    ``verify_images`` (opt-in) additionally runs a cheap PIL header probe on each captioned
    image and raises ValueError on a corrupt/zero-byte/truncated file. The start route enables
    it so a bad upload is rejected BEFORE the resident GPU models are freed, instead of crashing
    the spawned trainer after teardown; the trainers leave it off (they decode every image
    anyway, so a second probe pass would be redundant).
    """
    root = Path(data_dir).expanduser()
    if not root.is_dir():
        raise FileNotFoundError(f"data_dir is not a directory: {data_dir}")

    images = sorted(p for p in root.iterdir() if p.is_file() and p.suffix.lower() in _IMAGE_EXTS)

    # 1. metadata.jsonl / captions.jsonl (either name accepted).
    meta_caption: dict[str, str] = {}
    for meta_name in ("metadata.jsonl", "captions.jsonl"):
        meta_path = root / meta_name
        if not meta_path.is_file():
            continue
        # Tolerate a bad upload (invalid UTF-8, or non-object JSON): skip the record so the instance_prompt fallback still applies.
        try:
            meta_lines = meta_path.read_text(encoding = "utf-8").splitlines()
        except (OSError, UnicodeError):
            continue
        for line in meta_lines:
            line = line.strip()
            if not line:
                continue

View on GitHub (pinned to 203007d190)

Solutions

  1. Check the path exists and is a directory before starting training: Path(data_dir).expanduser().is_dir().
  2. If it is an upload race, poll for the dataset directory's existence before invoking the trainer.
  3. Verify the mount/volume and that the path points at the dataset root (the directory containing the images), not at an image file.

Example fix

# before
build_caption_pairs('/data/mydateset', ...)
# after
from pathlib import Path
root = Path('/data/mydataset').expanduser()
assert root.is_dir(), f'{root} missing'
build_caption_pairs(root, ...)
Defensive patterns

Strategy: validation

Validate before calling

from pathlib import Path
root = Path(data_dir).expanduser()
if not root.is_dir():
    raise FileNotFoundError(f'data_dir is not a directory: {data_dir}')

Try / catch

try:
    pairs = build_caption_pairs(data_dir, ...)
except FileNotFoundError as e:
    # surface to the user: dataset path missing/unmounted
    abort_run(str(e))

Prevention

When it happens

Trigger: Calling the caption-pair collection function with a typo'd path, a file path instead of a directory, a path on an unmounted volume, or '~' that fails expanduser resolution; also paths whose containing directory was deleted between upload and training start.

Common situations: Upload race (training starts before the dataset dir is materialized), docker volume mount mistakes, a stale absolute path after the workspace moved, or a Windows path used on Linux.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/40ad02ac4ad05d1f. Report an issue: GitHub.