{"record":{"id":"8b0411dbe37bf870","repo":"unslothai/unsloth","slug":"folder-contains-more-than-the-config-folder-max-f","errorCode":null,"errorMessage":"Folder contains more than the {config.FOLDER_MAX_FILES} supported files limit","messagePattern":"Folder contains more than the (.+?) supported files limit","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/rag/folder_sync.py","lineNumber":1007,"sourceCode":"                        st = os.lstat(full)\n                        from_path = True\n                    except OSError:\n                        # A file that vanished mid-scan must still reach _snapshot as a failure\n                        # rather than abort the scan, which is never authoritative for deletion.\n                        pass\n                rel = os.path.relpath(full, root).replace(os.sep, \"/\")\n                found[rel] = {\n                    \"path\": full,\n                    \"size_bytes\": st.st_size,\n                    \"mtime_ns\": st.st_mtime_ns,\n                    \"device\": st.st_dev,\n                    \"inode\": st.st_ino,\n                    # A recovered identity is comparable to the next scan's, but not to os.fstat's:\n                    # shared-folder and WebDAV drivers report different ids for the two call paths.\n                    \"identity_from_path\": from_path,\n                }\n                if config.FOLDER_MAX_FILES and len(found) > config.FOLDER_MAX_FILES:\n                    raise RuntimeError(\n                        f\"Folder contains more than the {config.FOLDER_MAX_FILES} supported files limit\"\n                    )\n    if _root_identity(root) != identity:\n        raise RuntimeError(\"Linked folder root identity changed during scan\")\n    return found, identity\n\n\ndef _snapshot(root: str, metadata: dict) -> str:\n    source = metadata[\"path\"]\n    resolved = os.path.realpath(source)\n    if not _is_within(root, resolved):\n        raise RuntimeError(\"File escaped the linked folder\")\n    # os.fdopen already forces this descriptor binary on Windows; O_BINARY only guards a raw os.read.\n    flags = os.O_RDONLY | getattr(os, \"O_NOFOLLOW\", 0) | getattr(os, \"O_BINARY\", 0)\n    fd = os.open(source, flags)\n    target = None\n    try:\n        ext = os.path.splitext(source)[1].lower()","sourceCodeStart":989,"sourceCodeEnd":1025,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/rag/folder_sync.py#L989-L1025","documentation":"_scan aborts once the count of collected files exceeds config.FOLDER_MAX_FILES (default 10000, env RAG_FOLDER_MAX_FILES). Each found file records path/size/mtime/device/inode for reconciliation, so unbounded folders would exhaust memory and job time; the limit is checked incrementally during the walk.","triggerScenarios":"Linking a directory tree with more than 10000 supported files (media libraries, dataset dumps, node_modules included) so len(found) passes the cap mid-walk.","commonSituations":"First link of a large Documents/Datasets folder; accidentally linking a folder that contains caches or dependency trees; lowering RAG_FOLDER_MAX_FILES in deployment while folders previously synced fine.","solutions":["Split the data: link smaller subdirectories so each stays under the cap.","Exclude bulky subtrees (caches, node_modules, datasets) from the linked root by restructuring.","If the deployment can afford it, raise the limit via RAG_FOLDER_MAX_FILES env var and re-run the sync job."],"exampleFix":"# before\ncreate_folder(..., path=\"/data/all\")  # 250k files\n\n# after\ncreate_folder(..., path=\"/data/all/docs\")\n# or raise the cap for this deployment:\n#   RAG_FOLDER_MAX_FILES=50000","handlingStrategy":"validation","validationCode":"import os\nfrom core.rag import config\n\ndef file_count_ok(root: str) -> bool:\n    count = 0\n    for _, _, files in os.walk(root):\n        count += len(files)\n        if config.FOLDER_MAX_FILES and count > config.FOLDER_MAX_FILES:\n            return False\n    return True","typeGuard":null,"tryCatchPattern":"try:\n    create_folder_with_sync(...)\nexcept RuntimeError as e:\n    if \"files limit\" in str(e):\n        show_user(f\"Folder has more than {config.FOLDER_MAX_FILES} files; split or prune it.\")\n    else:\n        raise","preventionTips":["Count supported files in the tree before registering it.","Keep caches, node_modules, and dataset dumps out of linked roots.","Set RAG_FOLDER_MAX_FILES deliberately per deployment instead of discovering the default mid-sync."],"tags":["rag","folder-sync","limits","scan","configuration"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}