{"record":{"id":"03fac660f434ca0a","repo":"EpicGames/lore","slug":"len-bytes-exceeds-the-whole-file-limit-byte-whole-file-limit","errorCode":null,"errorMessage":"{len} bytes exceeds the {WHOLE_FILE_LIMIT} byte whole-file limit; open the file and use read_exact_at or write_all_at","messagePattern":"(.+?) bytes exceeds the (.+?) byte whole-file limit; open the file and use read_exact_at or write_all_at","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"lore-io/src/driver.rs","lineNumber":65,"sourceCode":"#[cfg(target_os = \"linux\")]\nuse crate::uring::UringDriver;\n\n/// Largest file the whole-file operations accept.\n///\n/// [`IoDriver::read_file_bytes`] and [`IoDriver::write_file_bytes`] exist to keep a scan over\n/// many small files at one dispatch each. Both hold a pool thread for the whole transfer and hold\n/// the whole file resident, so reaching for them with a large file would occupy one of at most\n/// `min(2 × cores, 16)` threads for its duration. A caller with a large file wants [`open`] plus\n/// [`read_exact_at`] or [`write_all_at`], which read and write a bounded length at a time.\n///\n/// [`open`]: IoDriver::open\n/// [`read_exact_at`]: crate::IoFile::read_exact_at\n/// [`write_all_at`]: crate::IoFile::write_all_at\npub const WHOLE_FILE_LIMIT: usize = 8 * 1024 * 1024;\n\npub(crate) fn check_whole_file_len(len: usize) -> std::io::Result<()> {\n    if len > WHOLE_FILE_LIMIT {\n        return Err(std::io::Error::new(\n            std::io::ErrorKind::InvalidInput,\n            format!(\n                \"{len} bytes exceeds the {WHOLE_FILE_LIMIT} byte whole-file limit; \\\n                 open the file and use read_exact_at or write_all_at\"\n            ),\n        ));\n    }\n    Ok(())\n}\n\n/// Backend selection for an [`IoDriver`].\n#[derive(Clone, Copy, Debug, PartialEq, Eq)]\npub enum BackendKind {\n    /// Probe for the best available backend.\n    Auto,\n    /// Positional syscalls on the bounded syscall pool.\n    Psync,\n    /// Completion-based operations on sharded `io_uring` instances.","sourceCodeStart":47,"sourceCodeEnd":83,"githubUrl":"https://github.com/EpicGames/lore/blob/074eb0b0d1194c997d7cf28b55519e3e197b3e23/lore-io/src/driver.rs#L47-L83","documentation":"The whole-file helpers `read_file_bytes`/`write_file_bytes` are optimized for small files: they hold the entire file in memory and occupy one driver pool thread for the whole transfer. To protect that pool, `check_whole_file_len` rejects any request larger than `WHOLE_FILE_LIMIT` (8 MiB) with InvalidInput, telling the caller to use the bounded `read_exact_at`/`write_all_at` API on an opened `IoFile` instead.","triggerScenarios":"Calling `IoDriver::write_file_bytes` (or `read_file_bytes`) with a buffer/file whose length exceeds 8 * 1024 * 1024 bytes (8 MiB).","commonSituations":"Writing generated logs, dumps, media, or database files that grew past 8 MiB; a caller hardcoding 'just write the whole Vec' without checking size; migrating code that previously used std::fs without a size cap.","solutions":["Open the file with `IoDriver::open` and transfer in bounded chunks via `write_all_at`/`read_exact_at` at increasing offsets.","If the data is known-small, add an assert/size check upstream so oversized payloads take the chunked path automatically.","Raise the batching threshold in your own code (e.g. chunk at 8 MiB boundaries) rather than trying to bypass the limit, which is a fixed constant."],"exampleFix":"// before\ndriver.write_file_bytes(path, &huge_buf).await?;\n// after\nlet f = driver.open(path, /* write */ true).await?;\nfor (i, chunk) in huge_buf.chunks(8 * 1024 * 1024).enumerate() {\n    f.write_all_at(chunk, (i * 8 * 1024 * 1024) as u64).await?;\n}","handlingStrategy":"validation","validationCode":"// Rust\nconst WHOLE_FILE_LIMIT: usize = 8 * 1024 * 1024;\nfn can_use_whole_file_write(len: usize) -> bool {\n    len <= WHOLE_FILE_LIMIT\n}","typeGuard":null,"tryCatchPattern":"// Rust\nmatch driver.write_file_bytes(path, &buf).await {\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidInput\n        && e.to_string().contains(\"whole-file limit\") =>\n    {\n        // switch to chunked path\n        let f = driver.open(path, true).await?;\n        for (i, chunk) in buf.chunks(WHOLE_FILE_LIMIT).enumerate() {\n            f.write_all_at(chunk, (i * WHOLE_FILE_LIMIT) as u64).await?;\n        }\n    }\n    r => r?,\n}","preventionTips":["Check buffer size against WHOLE_FILE_LIMIT before choosing the whole-file helper.","Default to open + write_all_at for any file whose size is unknown or user-controlled.","Keep chunk size constants aligned with the library's WHOLE_FILE_LIMIT."],"tags":["io","limit","async","file"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"074eb0b0d1194c997d7cf28b55519e3e197b3e23","analyzedAt":"2026-09-13T09:00:57.509Z","contentChangedAt":"2026-09-13T09:00:57.509Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}