hiyouga/LlamaFactory · error · ValueError

Unsupported protocol in path: {path}. Use 's3://' or 'gs://'

Error message

Unsupported protocol in path: {path}. Use 's3://' or 'gs://'.

What it means

setup_fs only accepts s3:// and gs:// (or gcs://) path prefixes and raises ValueError for anything else. It is the entry point for cloud dataset loading, constructing an fsspec filesystem with optional anon=True for anonymous access.

Source

Thrown at src/llamafactory/data/data_utils.py:166

            if len(eval_dataset):
                dataset_module["eval_dataset"] = eval_dataset

    else:  # single dataset
        dataset_module["train_dataset"] = dataset

    return dataset_module


def setup_fs(path: str, anon: bool = False) -> "fsspec.AbstractFileSystem":
    r"""Set up a filesystem object based on the path protocol."""
    storage_options = {"anon": anon} if anon else {}
    if path.startswith("s3://"):
        fs = fsspec.filesystem("s3", **storage_options)
    elif path.startswith(("gs://", "gcs://")):
        fs = fsspec.filesystem("gcs", **storage_options)
    else:
        raise ValueError(f"Unsupported protocol in path: {path}. Use 's3://' or 'gs://'.")

    if not fs.exists(path):
        raise ValueError(f"Path does not exist: {path}.")

    return fs


def _read_json_with_fs(fs: "fsspec.AbstractFileSystem", path: str) -> list[Any]:
    r"""Helper function to read JSON/JSONL files using fsspec."""
    with fs.open(path, "r") as f:
        if path.endswith(".jsonl"):
            return [json.loads(line) for line in f if line.strip()]
        else:
            return json.load(f)


def read_cloud_json(cloud_path: str) -> list[Any]:
    r"""Read a JSON/JSONL file from cloud storage (S3 or GCS).

View on GitHub (pinned to f28afaf635)

Solutions

1. Use a fully-qualified s3://bucket/key or gs://bucket/key (gcs:// also accepted) path.
2. For local files, use ordinary local paths through the normal dataset loading route, not the cloud helper.
3. For other object stores, download/sync the data first (aws s3 cp, gsutil) or extend setup_fs with the needed fsspec protocol.

Example fix

# before
{"file_name": "azure://container/data.json"}

# after
# sync to local first, then
{"file_name": "data/local_copy/data.json"}

def is_supported_cloud_path(p: str) -> bool:
    return p.startswith(("s3://", "gs://", "gcs://"))

assert is_supported_cloud_path(cloud_path), "only s3:// and gs:// are supported"

def is_supported_cloud_path(p: str) -> bool:
    return p.startswith(("s3://", "gs://", "gcs://"))

When it happens

Trigger: Calling the cloud JSON loading path (load云 datasets from S3/GCS) with an hdfs://, file://, azure://, or plain local path; missing or mistyped scheme (e.g. 's3:/bucket/x' with one slash).

Common situations: Pointing dataset paths at Azure Blob or local NFS paths while the data-loading code only supports S3/GCS; scheme typos in dataset_info.json entries.

Related errors

• Path does not exist: {path}.
• No JSON/JSONL files found in the specified path: {cloud_path
• Unsupported model type: {getattr(config, 'model_type')}.
• Dataset converter {name} not found.
• Unknown mixing strategy: {data_args.mix_strategy}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/62006a3e22beca4f. Report an issue: GitHub.