unslothai/unsloth · error · HTTPException

Dataset upload too large. Maximum is {get_upload_limit_label

Error message

Dataset upload too large. Maximum is {get_upload_limit_label()} per upload; add the remaining files in another batch.

What it means

HTTP 413 raised while streaming a dataset upload to disk. The handler accumulates bytes across ALL files in one multipart request and aborts as soon as the running total exceeds the configured per-request upload limit (label from get_upload_limit_label(), e.g. '1 GB per upload'). Because it counts bytes as they are read (1 MiB chunks), the check fires mid-file, before any file is committed — the finally-block cleans up staged temp files.

Source

Thrown at studio/backend/routes/training.py:3526

            if ext in _DIFFUSION_DATASET_MEDIA_EXTS:
                media_names_by_stem_cf.setdefault(Path(filename).stem.casefold(), []).append(
                    filename
                )
            names.append(filename)
        # Stage each file to a temp name and move it in only once the whole batch is written, so a mid-batch failure leaves the dataset untouched, including any same-name file a direct write would truncate.
        staged: list[tuple[Path, Path]] = []  # (temp, final)
        committed = False
        try:
            for f, filename in zip(files, names):
                dest = folder / filename
                # A filename-independent temp name so a long (but valid) filename cannot overflow NAME_MAX with the staging suffix.
                tmp = folder / f".upload-{_uuid.uuid4().hex}.part"
                staged.append((tmp, dest))
                with open(tmp, "wb") as out:
                    while chunk := await f.read(1024 * 1024):
                        total_bytes += len(chunk)
                        if total_bytes > limit_bytes:
                            raise HTTPException(
                                status_code = 413,
                                detail = (
                                    "Dataset upload too large. "
                                    f"Maximum is {get_upload_limit_label()} per upload; "
                                    "add the remaining files in another batch."
                                ),
                            )
                        out.write(chunk)
                # Reject a decompression bomb before commit: a small PNG can pass the byte limit yet decode to huge pixels and OOM the trainer's latent cache.
                # Images only. A clip's frames are bounded by the canvas the video trainer resizes
                # to, not by the container, so there is no equivalent still to decode here.
                if Path(filename).suffix.lower() in _DIFFUSION_DATASET_IMAGE_EXTS:
                    _validate_uploaded_training_image(tmp, filename)
                uploaded += 1
            # Re-check the interlock immediately before the commit: the entry guard only saw the
            # pre-upload state, so a /diffusion/start could have reserved the slot while we streamed.
            _require_diffusion_dataset_mutable()
            # Commit every staged file as one transaction: a plain replace loop is not atomic. Back up

View on GitHub (pinned to 203007d190)

Solutions

1. Split the files into multiple upload batches, each under the limit shown in the message.
2. Check the effective limit via get_upload_limit_label()'s backing setting (upload_limits config) and align any reverse proxy (nginx client_max_body_size, Traefik, etc.) with it.
3. If the limit is genuinely too small for your workflow, raise the configured upload limit and restart Studio, keeping the proxy limit >= app limit.
4. Compress oversized assets (e.g. re-encode huge PNGs) before upload.

Example fix

// before
const files = allDatasetFiles; // 3 GB in one request
await api.upload('/training/diffusion/dataset/myset/upload', files);

// after
const LIMIT = 1024**3; // match get_upload_limit_label()
for (const batch of chunkBySize(allDatasetFiles, LIMIT * 0.95)) {
  await api.upload('/training/diffusion/dataset/myset/upload', batch);
}

def batch_under_limit(paths: list[Path], limit_bytes: int) -> bool:
    return sum(p.stat().st_size for p in paths) < limit_bytes

# or client-side before POSTing
const total = files.reduce((n, f) => n + f.size, 0);
if (total >= LIMIT) throw new Error(`Split upload: ${total} bytes > ${LIMIT}`);

def is_upload_too_large_error(exc: HTTPException) -> bool:
    return exc.status_code == 413 and 'too large' in exc.detail

When it happens

Trigger: POST to the diffusion dataset upload endpoint with a batch of files whose combined size exceeds the configured upload limit. The error triggers on the file that pushes total_bytes over limit_bytes, so a single large file or many small files in one request both cause it.

Common situations: Uploading a full dataset folder in one drag-and-drop batch; raising the reverse-proxy body limit but not the app limit (or vice versa); datasets that grew since the last training run; trying to upload a video plus stills together.

Related errors

• File too large ({size_bytes} bytes). Maximum is {UNSTRUCTURE
• Total upload limit ({UNSTRUCTURED_RECIPE_UPLOAD_TOTAL_MAX_LA
• File exceeds the {cap // (1024 * 1024)} MB upload limit.
• lockfile not found: {path}
• Image is too large ({w}x{h}); maximum is {max_side}px per si

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/268071281a98cd42. Report an issue: GitHub.