unslothai/unsloth · error · RuntimeError

Linked source exceeds the RAG file size limit

Error message

Linked source exceeds the RAG file size limit

What it means

_snapshot enforces config.MAX_UPLOAD_BYTES (default 200 MiB, env RAG_MAX_UPLOAD_BYTES) on the linked file's size before copying it into the RAG uploads area. The same cap that guards direct uploads guards folder ingestion, so one huge file in a folder cannot blow up storage or ingestion memory.

Source

Thrown at studio/backend/core/rag/folder_sync.py:1046

        if not stat.S_ISREG(before.st_mode):
            raise RuntimeError("Linked source is not a regular file")
        expected = (
            metadata["size_bytes"],
            metadata["mtime_ns"],
            metadata["device"],
            metadata["inode"],
        )
        actual = (before.st_size, before.st_mtime_ns, before.st_dev, before.st_ino)
        # Only an identity os.fstat can reproduce is comparable here: os.scandir reports none at all
        # on Windows, and the os.lstat _scan falls back to disagrees with os.fstat on file systems
        # without stable file ids. Size and mtime still gate those, and the post-copy check below is
        # fstat to fstat either way.
        usable = metadata["inode"] not in (None, 0) and not metadata.get("identity_from_path")
        compared = 4 if usable else 2
        if actual[:compared] != expected[:compared]:
            raise RuntimeError("Linked source changed during reconciliation")
        if config.MAX_UPLOAD_BYTES and before.st_size > config.MAX_UPLOAD_BYTES:
            raise RuntimeError("Linked source exceeds the RAG file size limit")
        with os.fdopen(fd, "rb", closefd = False) as src, open(target, "xb") as dst:
            _copy_exact(src, dst, before.st_size)
        after = os.fstat(fd)
        # Both sides are fstat here, so the identity is always comparable.
        if (after.st_size, after.st_mtime_ns, after.st_dev, after.st_ino) != actual:
            raise RuntimeError("Linked source changed while it was copied")
        return str(target)
    except Exception:
        if target is not None:
            _remove_snapshot(str(target))
        raise
    finally:
        os.close(fd)


def _copy_exact(source, target, size: int) -> None:
    remaining = size
    while remaining:

View on GitHub (pinned to 203007d190)

Solutions

1. Move oversized files out of the linked folder (or into an ignored subfolder that is not linked).
2. Split large artifacts into chunks under the limit if they must be indexed.
3. If the deployment allows, raise RAG_MAX_UPLOAD_BYTES and restart so the limit admits the file.

Example fix

# before: linked folder contains model-weights.safetensors (5 GB)

# after
mv /data/folder/model-weights.safetensors /data/archive/
# or: RAG_MAX_UPLOAD_BYTES=10000000000 in the backend environment

import os
from core.rag import config

def files_within_upload_cap(root: str) -> list[str]:
    oversize = []
    for dirpath, _, files in os.walk(root):
        for f in files:
            p = os.path.join(dirpath, f)
            if os.path.isfile(p) and os.path.getsize(p) > (config.MAX_UPLOAD_BYTES or 0):
                oversize.append(p)
    return oversize  # empty list == safe to link

try:
    ingest_folder(folder_id)
except RuntimeError as e:
    if "file size limit" in str(e):
        show_user(f"Files exceed the {config.MAX_UPLOAD_BYTES}-byte RAG limit; move or raise RAG_MAX_UPLOAD_BYTES.")
    else:
        raise

When it happens

Trigger: A file larger than MAX_UPLOAD_BYTES present in the linked tree: ISOs, VM images, model checkpoints, video files, database dumps — discovered during reconciliation even if the folder itself was fine at scan time.

Common situations: Linking a Documents folder that contains a 2 GB video or zip; data-science folders with large .bin/.safetensors files; deployments that lowered RAG_MAX_UPLOAD_BYTES after folders were registered.

Related errors

• Folder contains more than the {config.FOLDER_MAX_FILES} supp
• Folder nesting exceeds the {_MAX_FOLDER_DEPTH}-level scan li
• Linked folder no longer resolves to its registered path
• Linked folders support only knowledge-base and project scope
• The linked-folder scope no longer exists

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/2e122f5339be858f. Report an issue: GitHub.