{"record":{"id":"bd0c0e43fbedf2b7","repo":"unslothai/unsloth","slug":"s3-prefix-contains-mixed-dataset-formats-jo","errorCode":null,"errorMessage":"S3 prefix contains mixed dataset formats ({', '.join(families)}). Keep one dataset format under the selected prefix.","messagePattern":"S3 prefix contains mixed dataset formats \\((.+?)\\)\\. Keep one dataset format under the selected prefix\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/training/s3_dataset.py","lineNumber":125,"sourceCode":"\ndef _extension_family(key: str) -> str:\n    ext = os.path.splitext(key)[1].lower()\n    if ext in _JSON_EXTENSIONS:\n        return \"json\"\n    return ext.lstrip(\".\")\n\n\ndef _validate_single_extension_family(keys: list[str]) -> None:\n    families: list[str] = []\n    for key in keys:\n        family = _extension_family(key)\n        if family not in families:\n            families.append(family)\n\n    if len(families) <= 1:\n        return\n\n    raise ValueError(\n        \"S3 prefix contains mixed dataset formats \"\n        f\"({', '.join(families)}). Keep one dataset format under the selected prefix.\"\n    )\n\n\ndef _unique_local_path(target_dir: str, filename: str, used_paths: set[str]) -> str:\n    \"\"\"Return an unused flattened path for an S3 object basename.\"\"\"\n    stem, ext = os.path.splitext(filename)\n    candidate = os.path.join(target_dir, filename)\n    suffix = 1\n    while candidate in used_paths or os.path.exists(candidate):\n        candidate = os.path.join(target_dir, f\"{stem}_{suffix}{ext}\")\n        suffix += 1\n    used_paths.add(candidate)\n    return candidate\n\n\ndef _raise_if_cancelled(cancel_callback: Optional[Callable[[], bool]]) -> None:","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/training/s3_dataset.py#L107-L143","documentation":"Raised by _validate_single_extension_family during S3 dataset preparation when the objects listed under the chosen bucket/prefix mix more than one dataset format family. Supported extensions are .parquet, .json, .jsonl, .csv (s3_dataset.py:29); .json/.jsonl collapse into one 'json' family. The loader can only interpret one serialization format per dataset, so mixing (e.g. parquet + csv) aborts before download.","triggerScenarios":"Calling prepare_s3_dataset_download with s3_config whose prefix covers objects like data/part-0.parquet and data/labels.csv; typically a prefix that is too broad and picks up unrelated files, or a bucket where two export jobs wrote different formats.","commonSituations":"Pointing the prefix at a bucket root instead of a run-specific folder; a new export pipeline changed format (csv -> parquet) while old files remain; metadata sidecar files with supported extensions uploaded next to the dataset.","solutions":["Narrow s3_config.prefix so it covers only the objects of a single format.","Delete or move the stray files of the other format (including old exports) under that prefix.","Re-export the dataset in one consistent format to a fresh prefix."],"exampleFix":"// before\nprepare_s3_dataset_download({\"bucket\": \"my-data\", \"prefix\": \"datasets/\"})\n// after  # only the parquet export lives here\nprepare_s3_dataset_download({\"bucket\": \"my-data\", \"prefix\": \"datasets/parquet-export-2026-08/\"})","handlingStrategy":"validation","validationCode":"EXTS = ('.parquet', '.json', '.jsonl', '.csv')\nFAMILY = {'.json': 'json', '.jsonl': 'json'}\n\ndef single_family(keys: list[str]) -> bool:\n    fams = {FAMILY.get(k[k.rfind('.'):].lower(), k[k.rfind('.'):].lstrip('.')) for k in keys if k.lower().endswith(EXTS)}\n    return len(fams) <= 1\n\n# list keys first (aws s3 ls / boto3 list_objects_v2), then:\nassert single_family(keys), \"mixed dataset formats under prefix\"","typeGuard":null,"tryCatchPattern":"try:\n    dl = prepare_s3_dataset_download(cfg)\nexcept ValueError as e:\n    if \"mixed dataset formats\" in str(e):\n        # narrow prefix or clean stray files, then retry\n        ...","preventionTips":["Use one dedicated prefix per dataset export, never the bucket root.","When changing export format, write to a new prefix rather than mixing with old files.","Run a listing + extension audit before kicking off training."],"tags":["s3","dataset","validation","data-format"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}