{"record":{"id":"2e122f5339be858f","repo":"unslothai/unsloth","slug":"linked-source-exceeds-the-rag-file-size-limit","errorCode":null,"errorMessage":"Linked source exceeds the RAG file size limit","messagePattern":"Linked source exceeds the RAG file size limit","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/rag/folder_sync.py","lineNumber":1046,"sourceCode":"        if not stat.S_ISREG(before.st_mode):\n            raise RuntimeError(\"Linked source is not a regular file\")\n        expected = (\n            metadata[\"size_bytes\"],\n            metadata[\"mtime_ns\"],\n            metadata[\"device\"],\n            metadata[\"inode\"],\n        )\n        actual = (before.st_size, before.st_mtime_ns, before.st_dev, before.st_ino)\n        # Only an identity os.fstat can reproduce is comparable here: os.scandir reports none at all\n        # on Windows, and the os.lstat _scan falls back to disagrees with os.fstat on file systems\n        # without stable file ids. Size and mtime still gate those, and the post-copy check below is\n        # fstat to fstat either way.\n        usable = metadata[\"inode\"] not in (None, 0) and not metadata.get(\"identity_from_path\")\n        compared = 4 if usable else 2\n        if actual[:compared] != expected[:compared]:\n            raise RuntimeError(\"Linked source changed during reconciliation\")\n        if config.MAX_UPLOAD_BYTES and before.st_size > config.MAX_UPLOAD_BYTES:\n            raise RuntimeError(\"Linked source exceeds the RAG file size limit\")\n        with os.fdopen(fd, \"rb\", closefd = False) as src, open(target, \"xb\") as dst:\n            _copy_exact(src, dst, before.st_size)\n        after = os.fstat(fd)\n        # Both sides are fstat here, so the identity is always comparable.\n        if (after.st_size, after.st_mtime_ns, after.st_dev, after.st_ino) != actual:\n            raise RuntimeError(\"Linked source changed while it was copied\")\n        return str(target)\n    except Exception:\n        if target is not None:\n            _remove_snapshot(str(target))\n        raise\n    finally:\n        os.close(fd)\n\n\ndef _copy_exact(source, target, size: int) -> None:\n    remaining = size\n    while remaining:","sourceCodeStart":1028,"sourceCodeEnd":1064,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/rag/folder_sync.py#L1028-L1064","documentation":"_snapshot enforces config.MAX_UPLOAD_BYTES (default 200 MiB, env RAG_MAX_UPLOAD_BYTES) on the linked file's size before copying it into the RAG uploads area. The same cap that guards direct uploads guards folder ingestion, so one huge file in a folder cannot blow up storage or ingestion memory.","triggerScenarios":"A file larger than MAX_UPLOAD_BYTES present in the linked tree: ISOs, VM images, model checkpoints, video files, database dumps — discovered during reconciliation even if the folder itself was fine at scan time.","commonSituations":"Linking a Documents folder that contains a 2 GB video or zip; data-science folders with large .bin/.safetensors files; deployments that lowered RAG_MAX_UPLOAD_BYTES after folders were registered.","solutions":["Move oversized files out of the linked folder (or into an ignored subfolder that is not linked).","Split large artifacts into chunks under the limit if they must be indexed.","If the deployment allows, raise RAG_MAX_UPLOAD_BYTES and restart so the limit admits the file."],"exampleFix":"# before: linked folder contains model-weights.safetensors (5 GB)\n\n# after\nmv /data/folder/model-weights.safetensors /data/archive/\n# or: RAG_MAX_UPLOAD_BYTES=10000000000 in the backend environment","handlingStrategy":"validation","validationCode":"import os\nfrom core.rag import config\n\ndef files_within_upload_cap(root: str) -> list[str]:\n    oversize = []\n    for dirpath, _, files in os.walk(root):\n        for f in files:\n            p = os.path.join(dirpath, f)\n            if os.path.isfile(p) and os.path.getsize(p) > (config.MAX_UPLOAD_BYTES or 0):\n                oversize.append(p)\n    return oversize  # empty list == safe to link","typeGuard":null,"tryCatchPattern":"try:\n    ingest_folder(folder_id)\nexcept RuntimeError as e:\n    if \"file size limit\" in str(e):\n        show_user(f\"Files exceed the {config.MAX_UPLOAD_BYTES}-byte RAG limit; move or raise RAG_MAX_UPLOAD_BYTES.\")\n    else:\n        raise","preventionTips":["Pre-check largest files in a tree before linking (du -a | sort -n).","Keep media/checkpoint/ISO files out of linked folders.","Set RAG_MAX_UPLOAD_BYTES explicitly per deployment rather than relying on the 200 MiB default."],"tags":["rag","folder-sync","limits","configuration","upload"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}