{"record":{"id":"77f757d5725668a0","repo":"hiyouga/LlamaFactory","slug":"file-types-should-be-identical","errorCode":null,"errorMessage":"File types should be identical.","messagePattern":"File types should be identical\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/data/loader.py","lineNumber":89,"sourceCode":"        data_path = dataset_attr.dataset_name\n\n    elif dataset_attr.load_from == \"file\":\n        data_files = []\n        local_path = os.path.join(data_args.dataset_dir, dataset_attr.dataset_name)\n        if os.path.isdir(local_path):  # is directory\n            for file_name in os.listdir(local_path):\n                data_files.append(os.path.join(local_path, file_name))\n        elif os.path.isfile(local_path):  # is file\n            data_files.append(local_path)\n        else:\n            raise ValueError(f\"File {local_path} not found.\")\n\n        data_path = FILEEXT2TYPE.get(os.path.splitext(data_files[0])[-1][1:], None)\n        if data_path is None:\n            raise ValueError(\"Allowed file types: {}.\".format(\",\".join(FILEEXT2TYPE.keys())))\n\n        if any(data_path != FILEEXT2TYPE.get(os.path.splitext(data_file)[-1][1:], None) for data_file in data_files):\n            raise ValueError(\"File types should be identical.\")\n    else:\n        raise NotImplementedError(f\"Unknown load type: {dataset_attr.load_from}.\")\n\n    if dataset_attr.load_from == \"ms_hub\":\n        check_version(\"modelscope>=1.14.0\", mandatory=True)\n        from modelscope import MsDataset  # type: ignore\n        from modelscope.utils.config_ds import MS_DATASETS_CACHE  # type: ignore\n\n        cache_dir = model_args.cache_dir or MS_DATASETS_CACHE\n        dataset = MsDataset.load(\n            dataset_name=data_path,\n            subset_name=data_name,\n            data_dir=data_dir,\n            data_files=data_files,\n            split=dataset_attr.split,\n            cache_dir=cache_dir,\n            token=model_args.ms_hub_token,\n            use_streaming=data_args.streaming,","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/data/loader.py#L71-L107","documentation":"Raised when a dataset loaded from a directory contains files with differing extensions: the type inferred from the first file is compared against every other file, and any mismatch aborts loading. LlamaFactory can only merge local files of a single format per dataset entry.","triggerScenarios":"Pointing dataset_name at a directory that mixes .json and .jsonl, or .csv and .parquet, or contains stray files (e.g. .DS_Store, README.md, a lock file, or a backup copy with a different suffix).","commonSituations":"Incrementally exported data where a later export used a different format; macOS .DS_Store or editor backup files inside the data folder; combining sharded datasets saved with different serializers.","solutions":["Make every file in the directory the same format/extension; convert the odd ones out (e.g. rewrite .json shards as .jsonl).","Remove non-data files from the directory (.DS_Store, README, backups).","Alternatively split the formats into separate directories and register each as its own dataset entry, then list both in the dataset field.","Point dataset_name at a single file instead of the directory if only one file is needed."],"exampleFix":"# before: data/mydir/ contains part1.json, part2.jsonl\n\n# after: data/mydir/ contains part1.jsonl, part2.jsonl\n# (convert part1.json: one JSON object per line, then rename)","handlingStrategy":"validation","validationCode":"import os\n\ndef dir_uniform(d: str) -> bool:\n    exts = {os.path.splitext(f)[-1] for f in os.listdir(d)}\n    return len(exts) == 1","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep one format per dataset directory; convert shards before adding them.","Exclude non-data files (.DS_Store, README, backups) from data directories.","Register each format as a separate dataset entry if mixing is required."],"tags":["dataset","file-format","directory","config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}