{"record":{"id":"eee3c50fa92ac1f5","repo":"binary-husky/gpt_academic","slug":"file-size-file-size-mb-1f-mb-exceeds-limit-of","errorCode":null,"errorMessage":"File size ({file_size_mb:.1f}MB) exceeds limit of {max_size_mb}MB","messagePattern":"File size \\((.+?)MB\\) exceeds limit of (.+?)MB","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"warning","filePath":"crazy_functions/doc_fns/read_fns/unstructured_all/unstructured_reader.py","lineNumber":125,"sourceCode":"\n        Raises:\n            ValueError: 文件不存在、格式不支持或大小超限\n            PermissionError: 没有读取权限\n        \"\"\"\n        path = Path(file_path).resolve()\n\n        if not path.exists():\n            raise ValueError(f\"File not found: {path}\")\n\n        if not path.is_file():\n            raise ValueError(f\"Not a file: {path}\")\n\n        if not os.access(path, os.R_OK):\n            raise PermissionError(f\"No read permission: {path}\")\n\n        file_size_mb = path.stat().st_size / (1024 * 1024)\n        if file_size_mb > max_size_mb:\n            raise ValueError(\n                f\"File size ({file_size_mb:.1f}MB) exceeds limit of {max_size_mb}MB\"\n            )\n\n        if path.suffix.lower() not in self.SUPPORTED_EXTENSIONS:\n            raise ValueError(\n                f\"Unsupported format: {path.suffix}. \"\n                f\"Supported: {', '.join(sorted(self.SUPPORTED_EXTENSIONS))}\"\n            )\n\n        return path\n\n    def _cleanup_text(self, text: str) -> str:\n        \"\"\"清理文本\n\n        Args:\n            text: 原始文本\n\n        Returns:","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/binary-husky/gpt_academic/blob/d6bde0fa54373309bd05823a49bda8da019d2c77/crazy_functions/doc_fns/read_fns/unstructured_all/unstructured_reader.py#L107-L143","documentation":"UnstructuredReader's English size guard: ValueError raised when the file's size in MB exceeds max_size_mb. It fires after existence/is_file/read checks and before the extension allowlist check.","triggerScenarios":"Calling the reader with a document larger than the configured max_size_mb — huge scanned PDFs, Word files full of embedded media, or accidentally pointing at a bundled archive-like file that is actually parsed as a document.","commonSituations":"Default limit too small for real corpora; scanned thesis PDFs at hundreds of MB; users zipping content or embedding media; memory pressure or OOM that motivated the limit being forgotten when it blocks a legitimate file.","solutions":["Increase max_size_mb on the call/constructor if your environment can afford the memory.","Compress or split the document (ghostscript -dPDFSETTINGS=/ebook, qpdf --split-pages).","Reject oversized uploads earlier in the flow with a friendly message (413-style).","Stream large files through a lighter extractor instead of the full unstructured pipeline."],"exampleFix":"# before\ntext = reader.read(fp)  # ValueError: exceeds limit\n\n# after\nLIMIT_MB = 200\nsize_mb = fp.stat().st_size / (1024 * 1024)\ntext = reader.read(fp, max_size_mb=max(LIMIT_MB, int(size_mb) + 1)) if size_mb < 500 else None\nif text is None:\n    raise HTTPException(413, 'file too large to process')","handlingStrategy":"validation","validationCode":"MB = 1024 * 1024\nsize_mb = os.path.getsize(fp) / MB\nif size_mb > limit:\n    return reject_413(fp, size_mb, limit)","typeGuard":"def size_ok(path: str, limit_mb: float) -> bool:\n    import os\n    return os.path.getsize(path) / (1024 * 1024) <= limit_mb","tryCatchPattern":"try:\n    reader.read(fp)\nexcept ValueError as e:\n    if 'exceeds limit' in str(e):\n        split_or_compress_then_retry(fp)\n    raise","preventionTips":["Set the limit from your memory budget, document it.","Compress/split oversized docs at ingest.","Surface the limit in the upload UI.","Monitor for files just under the limit (boundary drift)."],"tags":["file-size","validation","memory","python"],"backgroundTag":null,"analyzedSha":"d6bde0fa54373309bd05823a49bda8da019d2c77","analyzedAt":"2026-08-14T22:48:35.038Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}