{"record":{"id":"464a6c2784203bc6","repo":"hiyouga/LlamaFactory","slug":"unknown-dataset-filetype-filetype","errorCode":null,"errorMessage":"Unknown dataset filetype: {filetype}.","messagePattern":"Unknown dataset filetype: (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/data_plugins/loader.py","lineNumber":49,"sourceCode":"        split = dataset_info.get(\"split\", \"train\")\n        streaming = dataset_info.get(\"streaming\", False)\n        return super().__call__(path, split, streaming)\n\n\ndef _get_builder_name(path: str) -> Literal[\"arrow\", \"csv\", \"json\", \"parquet\", \"text\"]:\n    \"\"\"Get dataset builder name.\n\n    Args:\n        path (str): Dataset path.\n\n    Returns:\n        Literal[\"arrow\", \"csv\", \"json\", \"parquet\", \"text\"]: Dataset builder name.\n    \"\"\"\n    filetype = os.path.splitext(path)[-1][1:]\n    if filetype in [\"arrow\", \"csv\", \"json\", \"jsonl\", \"parquet\", \"txt\"]:\n        return filetype.replace(\"jsonl\", \"json\").replace(\"txt\", \"text\")\n    else:\n        raise ValueError(f\"Unknown dataset filetype: {filetype}.\")\n\n\n@DataLoaderPlugin(\"local\").register()\ndef load_data_from_file(filepath: str, split: str, streaming: bool) -> HFDataset:\n    if os.path.isdir(filepath):\n        filetype = _get_builder_name(os.listdir(filepath)[0])\n        dataset = load_dataset(filetype, data_dir=filepath, split=split)\n    elif os.path.isfile(filepath):\n        filetype = _get_builder_name(filepath)\n        dataset = load_dataset(filetype, data_files=filepath, split=split)\n    else:\n        raise ValueError(f\"Can not load dataset from {filepath}.\")\n\n    if streaming:  # faster when data is streamed from local files\n        dataset = dataset.to_iterable_dataset()\n\n    return dataset\n","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/data_plugins/loader.py#L31-L67","documentation":"When loading a local dataset, the loader derives the HF datasets builder from the file extension. Only arrow, csv, json, jsonl, parquet and txt are recognized; any other extension raises this ValueError before load_dataset is attempted.","triggerScenarios":"Passing a dataset_dir/data_files path whose extension is not in [arrow, csv, json, jsonl, parquet, txt] — e.g. .tsv, .xlsx, .md, .pkl, or a file with no extension. Also fires when loading from a directory whose first file has such an extension.","commonSituations":"Exporting data from Excel/pandas as .xlsx or .tsv and pointing dataset_dir at it; pointing at a README.md or metadata file inside a data directory (os.listdir order makes the first file decide).","solutions":["Convert the data to a supported format: JSON/JSONL is usually the least friction.","Rename .tsv to .csv only if it is genuinely comma-separated (TSV is not valid CSV).","If loading from a directory, ensure every file inside has a supported extension; move READMEs and metadata out.","For arbitrary files, pre-convert with pandas: df.to_json('data.jsonl', orient='records', lines=True)."],"exampleFix":"# before\ndataset_dir: data/my_notes.md\n\n# after\n# convert first:\nimport pandas as pd\npd.read_excel('data.xlsx').to_json('data/train.jsonl', orient='records', lines=True)\n# then:\ndataset_dir: data/train.jsonl","handlingStrategy":"validation","validationCode":"import os\nSUPPORTED = {'.arrow', '.csv', '.json', '.jsonl', '.parquet', '.txt'}\n\ndef path_loadable(p: str) -> bool:\n    if os.path.isdir(p):\n        return bool(os.listdir(p)) and all(os.path.splitext(f)[-1] in SUPPORTED for f in os.listdir(p))\n    return os.path.splitext(p)[-1] in SUPPORTED\n\nassert path_loadable(dataset_dir)","typeGuard":"def has_supported_extension(path: str) -> bool:\n    \"\"\"True when the path (or every file in the dir) uses a builder-supported extension.\"\"\"\n    return path_loadable(path)","tryCatchPattern":"try:\n    ds = load_data_from_file(path, split='train', streaming=False)\nexcept ValueError as e:\n    if 'Unknown dataset filetype' in str(e):\n        raise SystemExit(f'convert {path} to json/parquet/csv first') from None\n    raise","preventionTips":["Standardize training data on .jsonl in preprocessing pipelines.","Validate extensions in a data-prep Makefile target before training.","Keep metadata/README files out of dataset directories."],"tags":["data","file-format","dataset-loading","validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}