{"record":{"id":"268071281a98cd42","repo":"unslothai/unsloth","slug":"dataset-upload-too-large-maximum-is-get-upload-l","errorCode":null,"errorMessage":"Dataset upload too large. Maximum is {get_upload_limit_label()} per upload; add the remaining files in another batch.","messagePattern":"Dataset upload too large\\. Maximum is (.+?) per upload; add the remaining files in another batch\\.","errorType":"http","errorClass":"HTTPException","httpStatus":413,"severity":"error","filePath":"studio/backend/routes/training.py","lineNumber":3526,"sourceCode":"            if ext in _DIFFUSION_DATASET_MEDIA_EXTS:\n                media_names_by_stem_cf.setdefault(Path(filename).stem.casefold(), []).append(\n                    filename\n                )\n            names.append(filename)\n        # Stage each file to a temp name and move it in only once the whole batch is written, so a mid-batch failure leaves the dataset untouched, including any same-name file a direct write would truncate.\n        staged: list[tuple[Path, Path]] = []  # (temp, final)\n        committed = False\n        try:\n            for f, filename in zip(files, names):\n                dest = folder / filename\n                # A filename-independent temp name so a long (but valid) filename cannot overflow NAME_MAX with the staging suffix.\n                tmp = folder / f\".upload-{_uuid.uuid4().hex}.part\"\n                staged.append((tmp, dest))\n                with open(tmp, \"wb\") as out:\n                    while chunk := await f.read(1024 * 1024):\n                        total_bytes += len(chunk)\n                        if total_bytes > limit_bytes:\n                            raise HTTPException(\n                                status_code = 413,\n                                detail = (\n                                    \"Dataset upload too large. \"\n                                    f\"Maximum is {get_upload_limit_label()} per upload; \"\n                                    \"add the remaining files in another batch.\"\n                                ),\n                            )\n                        out.write(chunk)\n                # Reject a decompression bomb before commit: a small PNG can pass the byte limit yet decode to huge pixels and OOM the trainer's latent cache.\n                # Images only. A clip's frames are bounded by the canvas the video trainer resizes\n                # to, not by the container, so there is no equivalent still to decode here.\n                if Path(filename).suffix.lower() in _DIFFUSION_DATASET_IMAGE_EXTS:\n                    _validate_uploaded_training_image(tmp, filename)\n                uploaded += 1\n            # Re-check the interlock immediately before the commit: the entry guard only saw the\n            # pre-upload state, so a /diffusion/start could have reserved the slot while we streamed.\n            _require_diffusion_dataset_mutable()\n            # Commit every staged file as one transaction: a plain replace loop is not atomic. Back up","sourceCodeStart":3508,"sourceCodeEnd":3544,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L3508-L3544","documentation":"HTTP 413 raised while streaming a dataset upload to disk. The handler accumulates bytes across ALL files in one multipart request and aborts as soon as the running total exceeds the configured per-request upload limit (label from get_upload_limit_label(), e.g. '1 GB per upload'). Because it counts bytes as they are read (1 MiB chunks), the check fires mid-file, before any file is committed — the finally-block cleans up staged temp files.","triggerScenarios":"POST to the diffusion dataset upload endpoint with a batch of files whose combined size exceeds the configured upload limit. The error triggers on the file that pushes total_bytes over limit_bytes, so a single large file or many small files in one request both cause it.","commonSituations":"Uploading a full dataset folder in one drag-and-drop batch; raising the reverse-proxy body limit but not the app limit (or vice versa); datasets that grew since the last training run; trying to upload a video plus stills together.","solutions":["Split the files into multiple upload batches, each under the limit shown in the message.","Check the effective limit via get_upload_limit_label()'s backing setting (upload_limits config) and align any reverse proxy (nginx client_max_body_size, Traefik, etc.) with it.","If the limit is genuinely too small for your workflow, raise the configured upload limit and restart Studio, keeping the proxy limit >= app limit.","Compress oversized assets (e.g. re-encode huge PNGs) before upload."],"exampleFix":"// before\nconst files = allDatasetFiles; // 3 GB in one request\nawait api.upload('/training/diffusion/dataset/myset/upload', files);\n\n// after\nconst LIMIT = 1024**3; // match get_upload_limit_label()\nfor (const batch of chunkBySize(allDatasetFiles, LIMIT * 0.95)) {\n  await api.upload('/training/diffusion/dataset/myset/upload', batch);\n}","handlingStrategy":"validation","validationCode":"def batch_under_limit(paths: list[Path], limit_bytes: int) -> bool:\n    return sum(p.stat().st_size for p in paths) < limit_bytes\n\n# or client-side before POSTing\nconst total = files.reduce((n, f) => n + f.size, 0);\nif (total >= LIMIT) throw new Error(`Split upload: ${total} bytes > ${LIMIT}`);","typeGuard":"def is_upload_too_large_error(exc: HTTPException) -> bool:\n    return exc.status_code == 413 and 'too large' in exc.detail","tryCatchPattern":null,"preventionTips":["Sum file sizes client-side before each request and split batches with ~5% headroom.","Keep reverse-proxy body limits aligned with the Studio upload limit setting.","Upload fewer, larger files per batch rather than thousands of small ones in one request."],"tags":["upload","http-413","fastapi","size-limit"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}