unslothai/unsloth · error · ValueError

S3 prefix contains mixed dataset formats ({', '.join(familie

Error message

S3 prefix contains mixed dataset formats ({', '.join(families)}). Keep one dataset format under the selected prefix.

What it means

Raised by _validate_single_extension_family during S3 dataset preparation when the objects listed under the chosen bucket/prefix mix more than one dataset format family. Supported extensions are .parquet, .json, .jsonl, .csv (s3_dataset.py:29); .json/.jsonl collapse into one 'json' family. The loader can only interpret one serialization format per dataset, so mixing (e.g. parquet + csv) aborts before download.

Source

Thrown at studio/backend/core/training/s3_dataset.py:125

def _extension_family(key: str) -> str:
    ext = os.path.splitext(key)[1].lower()
    if ext in _JSON_EXTENSIONS:
        return "json"
    return ext.lstrip(".")


def _validate_single_extension_family(keys: list[str]) -> None:
    families: list[str] = []
    for key in keys:
        family = _extension_family(key)
        if family not in families:
            families.append(family)

    if len(families) <= 1:
        return

    raise ValueError(
        "S3 prefix contains mixed dataset formats "
        f"({', '.join(families)}). Keep one dataset format under the selected prefix."
    )


def _unique_local_path(target_dir: str, filename: str, used_paths: set[str]) -> str:
    """Return an unused flattened path for an S3 object basename."""
    stem, ext = os.path.splitext(filename)
    candidate = os.path.join(target_dir, filename)
    suffix = 1
    while candidate in used_paths or os.path.exists(candidate):
        candidate = os.path.join(target_dir, f"{stem}_{suffix}{ext}")
        suffix += 1
    used_paths.add(candidate)
    return candidate


def _raise_if_cancelled(cancel_callback: Optional[Callable[[], bool]]) -> None:

View on GitHub (pinned to 203007d190)

Solutions

Narrow s3_config.prefix so it covers only the objects of a single format.
Delete or move the stray files of the other format (including old exports) under that prefix.
Re-export the dataset in one consistent format to a fresh prefix.

Example fix

// before
prepare_s3_dataset_download({"bucket": "my-data", "prefix": "datasets/"})
// after  # only the parquet export lives here
prepare_s3_dataset_download({"bucket": "my-data", "prefix": "datasets/parquet-export-2026-08/"})

Defensive patterns

Strategy: validation

Validate before calling

EXTS = ('.parquet', '.json', '.jsonl', '.csv')
FAMILY = {'.json': 'json', '.jsonl': 'json'}

def single_family(keys: list[str]) -> bool:
    fams = {FAMILY.get(k[k.rfind('.'):].lower(), k[k.rfind('.'):].lstrip('.')) for k in keys if k.lower().endswith(EXTS)}
    return len(fams) <= 1

# list keys first (aws s3 ls / boto3 list_objects_v2), then:
assert single_family(keys), "mixed dataset formats under prefix"

Try / catch

try:
    dl = prepare_s3_dataset_download(cfg)
except ValueError as e:
    if "mixed dataset formats" in str(e):
        # narrow prefix or clean stray files, then retry
        ...

Prevention

Use one dedicated prefix per dataset export, never the bucket root.
When changing export format, write to a new prefix rather than mixing with old files.
Run a listing + extension audit before kicking off training.

When it happens

Trigger: Calling prepare_s3_dataset_download with s3_config whose prefix covers objects like data/part-0.parquet and data/labels.csv; typically a prefix that is too broad and picks up unrelated files, or a bucket where two export jobs wrote different formats.

Common situations: Pointing the prefix at a bucket root instead of a run-specific folder; a new export pipeline changed format (csv -> parquet) while old files remain; metadata sidecar files with supported extensions uploaded next to the dataset.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/bd0c0e43fbedf2b7. Report an issue: GitHub.