hiyouga/LlamaFactory · error · ValueError

Path does not exist: {path}.

Error message

Path does not exist: {path}.

What it means

After building the fsspec filesystem in setup_fs, fs.exists(path) is checked and ValueError is raised when the bucket/prefix does not exist. This distinguishes 'wrong credentials' from 'wrong path': existence probing happens with the constructed (anonymous or credentialed) filesystem.

Source

Thrown at src/llamafactory/data/data_utils.py:169

    else:  # single dataset
        dataset_module["train_dataset"] = dataset

    return dataset_module


def setup_fs(path: str, anon: bool = False) -> "fsspec.AbstractFileSystem":
    r"""Set up a filesystem object based on the path protocol."""
    storage_options = {"anon": anon} if anon else {}
    if path.startswith("s3://"):
        fs = fsspec.filesystem("s3", **storage_options)
    elif path.startswith(("gs://", "gcs://")):
        fs = fsspec.filesystem("gcs", **storage_options)
    else:
        raise ValueError(f"Unsupported protocol in path: {path}. Use 's3://' or 'gs://'.")

    if not fs.exists(path):
        raise ValueError(f"Path does not exist: {path}.")

    return fs


def _read_json_with_fs(fs: "fsspec.AbstractFileSystem", path: str) -> list[Any]:
    r"""Helper function to read JSON/JSONL files using fsspec."""
    with fs.open(path, "r") as f:
        if path.endswith(".jsonl"):
            return [json.loads(line) for line in f if line.strip()]
        else:
            return json.load(f)


def read_cloud_json(cloud_path: str) -> list[Any]:
    r"""Read a JSON/JSONL file from cloud storage (S3 or GCS).

    Args:
        cloud_path: str

View on GitHub (pinned to f28afaf635)

Solutions

1. Verify the path with the CLI of the store: aws s3 ls s3://bucket/prefix or gsutil ls gs://bucket/prefix.
2. For private data, export proper credentials (AWS credentials env vars or gcloud auth application-default login) before running.
3. Check bucket name spelling and region; ensure the listing permission exists (s3:ListBucket) since exists() needs it.
4. For public buckets, keep anon semantics in mind: confirm the objects really are public.

Example fix

# shell: verify before running training
aws s3 ls s3://my-bucket/datasets/data.json  # must succeed

gcloud auth application-default login  # for GCS private data

fs = fsspec.filesystem("s3")  # or "gcs"
if not fs.exists(cloud_path):
    raise ValueError(f"path not visible — check spelling, region, and credentials: {cloud_path}")

try:
    fs = setup_fs(cloud_path, anon=True)
except ValueError:
    fs = setup_fs(cloud_path)  # credentialed retry mirrors the library's own fallback

When it happens

Trigger: An s3:// or gs:// path with a typo'd bucket name, a wrong region endpoint, or an object the credentials cannot list; also anonymous access (anon=True tried first) against a private bucket whose existence is not visible anonymously — the credential retry may then raise here or an access error earlier.

Common situations: Private buckets accessed without credentials in the environment (no AWS_* / GOOGLE_APPLICATION_CREDENTIALS); bucket in another account; path uses the wrong separator or extra slash.

Related errors

• Unsupported protocol in path: {path}. Use 's3://' or 'gs://'
• No JSON/JSONL files found in the specified path: {cloud_path
• The model does not have a submodule named '{submodule_name}'
• Unsupported model type: {getattr(config, 'model_type')}.
• Stage does not supported: {stage}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/635264aed57db69d. Report an issue: GitHub.