EpicGames/lore · error · io::Error

bytes exceeds the byte whole-file limit; open the file and…

Error message

{len} bytes exceeds the {WHOLE_FILE_LIMIT} byte whole-file limit; open the file and use read_exact_at or write_all_at

What it means

The whole-file helpers `read_file_bytes`/`write_file_bytes` are optimized for small files: they hold the entire file in memory and occupy one driver pool thread for the whole transfer. To protect that pool, `check_whole_file_len` rejects any request larger than `WHOLE_FILE_LIMIT` (8 MiB) with InvalidInput, telling the caller to use the bounded `read_exact_at`/`write_all_at` API on an opened `IoFile` instead.

Solutions

1. Open the file with `IoDriver::open` and transfer in bounded chunks via `write_all_at`/`read_exact_at` at increasing offsets.
2. If the data is known-small, add an assert/size check upstream so oversized payloads take the chunked path automatically.
3. Raise the batching threshold in your own code (e.g. chunk at 8 MiB boundaries) rather than trying to bypass the limit, which is a fixed constant.

Example fix

// before
driver.write_file_bytes(path, &huge_buf).await?;
// after
let f = driver.open(path, /* write */ true).await?;
for (i, chunk) in huge_buf.chunks(8 * 1024 * 1024).enumerate() {
    f.write_all_at(chunk, (i * 8 * 1024 * 1024) as u64).await?;
}

// Rust
const WHOLE_FILE_LIMIT: usize = 8 * 1024 * 1024;
fn can_use_whole_file_write(len: usize) -> bool {
    len <= WHOLE_FILE_LIMIT
}

// Rust
match driver.write_file_bytes(path, &buf).await {
    Err(e) if e.kind() == std::io::ErrorKind::InvalidInput
        && e.to_string().contains("whole-file limit") =>
    {
        // switch to chunked path
        let f = driver.open(path, true).await?;
        for (i, chunk) in buf.chunks(WHOLE_FILE_LIMIT).enumerate() {
            f.write_all_at(chunk, (i * WHOLE_FILE_LIMIT) as u64).await?;
        }
    }
    r => r?,
}

When it happens

Trigger: Calling `IoDriver::write_file_bytes` (or `read_file_bytes`) with a buffer/file whose length exceeds 8 * 1024 * 1024 bytes (8 MiB).

Common situations: Writing generated logs, dumps, media, or database files that grew past 8 MiB; a caller hardcoding 'just write the whole Vec' without checking size; migrating code that previously used std::fs without a size cap.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• file ended before the requested read length
• file refused further writes
• file ended before the requested read length
• file ended before the requested read length
• file refused further writes

AI-assisted analysis of EpicGames/lore@074eb0b0d1 (2026-09-13). Data as JSON: /api/errors/03fac660f434ca0a. Report an issue: GitHub.

Appendix: source

Thrown at lore-io/src/driver.rs:65

#[cfg(target_os = "linux")]
use crate::uring::UringDriver;

/// Largest file the whole-file operations accept.
///
/// [`IoDriver::read_file_bytes`] and [`IoDriver::write_file_bytes`] exist to keep a scan over
/// many small files at one dispatch each. Both hold a pool thread for the whole transfer and hold
/// the whole file resident, so reaching for them with a large file would occupy one of at most
/// `min(2 × cores, 16)` threads for its duration. A caller with a large file wants [`open`] plus
/// [`read_exact_at`] or [`write_all_at`], which read and write a bounded length at a time.
///
/// [`open`]: IoDriver::open
/// [`read_exact_at`]: crate::IoFile::read_exact_at
/// [`write_all_at`]: crate::IoFile::write_all_at
pub const WHOLE_FILE_LIMIT: usize = 8 * 1024 * 1024;

pub(crate) fn check_whole_file_len(len: usize) -> std::io::Result<()> {
    if len > WHOLE_FILE_LIMIT {
        return Err(std::io::Error::new(
            std::io::ErrorKind::InvalidInput,
            format!(
                "{len} bytes exceeds the {WHOLE_FILE_LIMIT} byte whole-file limit; \
                 open the file and use read_exact_at or write_all_at"
            ),
        ));
    }
    Ok(())
}

/// Backend selection for an [`IoDriver`].
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum BackendKind {
    /// Probe for the best available backend.
    Auto,
    /// Positional syscalls on the bounded syscall pool.
    Psync,
    /// Completion-based operations on sharded `io_uring` instances.

View on GitHub (pinned to 074eb0b0d1)