{"record":{"id":"0d86798bc3fa91d5","repo":"unslothai/unsloth","slug":"unsupported-local-dataset-format-all-files-0","errorCode":null,"errorMessage":"Unsupported local dataset format: {all_files[0]}","messagePattern":"Unsupported local dataset format: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/training/worker.py","lineNumber":5059,"sourceCode":"                    if candidates:\n                        all_files.extend(str(c) for c in candidates)\n                        continue\n                    raise ValueError(f\"No supported data files in directory: {file_path_obj}\")\n                else:\n                    all_files.append(file_path)\n\n            if not all_files:\n                raise ValueError(\"No local dataset files found\")\n\n            first_ext = Path(all_files[0]).suffix.lower()\n            if first_ext in (\".json\", \".jsonl\"):\n                loader = \"json\"\n            elif first_ext == \".csv\":\n                loader = \"csv\"\n            elif first_ext == \".parquet\":\n                loader = \"parquet\"\n            else:\n                raise ValueError(f\"Unsupported local dataset format: {all_files[0]}\")\n            return load_dataset(loader, data_files = all_files, split = \"train\")\n\n        if hf_dataset:\n            dataset = _load_embedding_hf_dataset(\n                config,\n                load_dataset,\n                lambda message: _send_status(event_queue, message),\n            )\n        elif local_datasets:\n            dataset = _load_local_embedding_dataset(local_datasets)\n        elif config.get(\"s3_config\"):\n            from core.training.s3_dataset import (\n                S3DownloadCancelled,\n                prepare_s3_dataset_download,\n            )\n\n            _send_status(event_queue, \"Downloading dataset from S3...\")\n            s3_download = None","sourceCodeStart":5041,"sourceCodeEnd":5077,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/training/worker.py#L5041-L5077","documentation":"Raised when the first collected local dataset file has an extension other than .json/.jsonl/.csv/.parquet. The loader type is chosen from the first file's suffix, so an unsupported first file aborts the load even if later files are supported.","triggerScenarios":"all_files[0] ends in something like .arrow, .txt, .tsv, .xlsx, or no extension at all. Because only the first file's extension is inspected, a mixed directory where an unsupported file sorts first (sorted glob order) triggers this even when supported files exist alongside it.","commonSituations":"datasets.save_to_disk output (.arrow) pointed at the loader; .tsv files mistaken for .csv; mixed-format directories where e.g. notes.txt sorts before data.jsonl; files with compound suffixes like data.jsonl.tmp.","solutions":["Remove or relocate unsupported files from the dataset directory so the first (sorted) file is .json/.jsonl/.csv/.parquet.","Rename .tsv to .csv (with correct delimiter handling) or convert .arrow/.txt to jsonl/parquet.","Ensure the directory holds a single consistent format."],"exampleFix":"# before\n# /data contains: notes.txt, train.jsonl -> first_ext == '.txt' -> raises\n\n# after\n# /data contains only: train.jsonl -> loader = 'json'","handlingStrategy":"validation","validationCode":"from pathlib import Path\n\nSUPPORTED = {\".json\", \".jsonl\", \".csv\", \".parquet\"}\nfiles = sorted(f for f in Path(data_dir).iterdir() if f.is_file())\nassert files and files[0].suffix.lower() in SUPPORTED, f\"first file {files[0].name} has unsupported format\"","typeGuard":"def is_supported_dataset_file(path: str) -> bool:\n    return Path(path).suffix.lower() in {\".json\", \".jsonl\", \".csv\", \".parquet\"}","tryCatchPattern":"try:\n    ds = _load_local_embedding_dataset(paths)\nexcept ValueError as e:\n    if \"Unsupported local dataset format\" in str(e):\n        # filter/convert the offending files, then retry once\n        raise\n    raise","preventionTips":["Keep one format per dataset directory; remove stray .txt/.arrow files.","Convert .arrow (save_to_disk) outputs with to_json/to_parquet before use.","Watch sort order: the first file in sorted order decides the loader."],"tags":["dataset","file-format","training","validation"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}