RyanCodrai/turbovec · error · io::Error

file too large for this platform

Error message

file too large for this platform

What it means

When opening a v7 file the loader sizes a Vec<u8> from the declared length clamped to the on-disk size; if that byte count exceeds usize::MAX on the current platform the conversion fails and this InvalidData error is thrown. It means the index file cannot be addressed in a single in-memory buffer on this architecture.

Source

Thrown at turbovec/src/io_v7.rs:1222

/// Load a v7 file. Blocks carry no checksums — the commit's delta
/// digest covers every unit at the sync that wrote it, which is all
/// the crash protocol needs; detecting later external damage is out of
/// scope, as it is for v6.
pub(crate) fn load(path: &Path, expect_calib_gen: u64, expect_kind: u8) -> io::Result<V7Load> {
    // Allocate for what the file DECLARES, capped by what it actually
    // holds — never for its apparent length. An image's size is fixed by
    // its geometry and row count, so a padded or sparse file is bytes
    // nobody asked for, and reading it wholesale made memory track the
    // file rather than the index. That is the v6 defect #487 fixed, and
    // v7 inherited it when it became the only format. Trailing bytes are
    // ignored, as they were before.
    let f = File::open(path)?;
    let on_disk = f.metadata()?.len();
    let want = declared_len(&f).unwrap_or(on_disk).min(on_disk);
    let mut raw = vec![
        0u8;
        usize::try_from(want).map_err(|_| io::Error::new(
            io::ErrorKind::InvalidData,
            "file too large for this platform"
        ))?
    ];
    crate::io::read_exact_at(&f, &mut raw, 0)?;
    load_image(raw, expect_calib_gen, expect_kind, &path.display().to_string())
}

/// Bytes a complete image of this file's geometry and row count occupies,
/// read from the superblock and the two header slots.
///
/// `None` whenever the prefix is not self-consistent: the parser then
/// sees the whole file and produces its own, better message rather than
/// a truncation artefact of a guess made here.
fn declared_len(f: &File) -> Option<u64> {
    let mut sb = [0u8; 64];
    crate::io::read_exact_at(f, &mut sb, 0).ok()?;
    if &sb[0..4] != V7_MAGIC || sb[4] != V7_VERSION {

View on GitHub (pinned to ccab9f325e)

Solutions

1. Use a 64-bit build of turbovec for very large indexes
2. Split the index into smaller shards that fit the platform limit
3. Verify declared_len in the header is sane; regenerate the file if corrupt
4. Stream/load in chunks if the library offers a streaming loader

Example fix

// before
// 32-bit target loading a 5 GB index -> 'file too large for this platform'
let idx = Index::load_v7("huge.tv")?;
// after
// build for 64-bit: cargo build --target x86_64-unknown-linux-gnu
let idx = Index::load_v7("huge.tv")?;

let len = std::fs::metadata(path)?.len();
if len > usize::MAX as u64 {
    return Err(anyhow!("index too large for this (32-bit?) platform"));
}

fn fits_platform(path: &Path) -> bool {
    std::fs::metadata(path)
        .map(|m| u64::from(usize::try_from(m.len()).unwrap_or(usize::MAX)) == m.len())
        .unwrap_or(false)
}

match Index::load_v7(path) {
    Err(e) if e.to_string().contains("file too large for this platform") => {
        eprintln!("switch to a 64-bit build or shard the index");
        Err(e)
    }
    other => other,
}

When it happens

Trigger: usize::try_from(want) failing because declared_len (or the on-disk size) exceeds the platform's addressable range — practically only on 32-bit targets or absurdly large files.

Common situations: Loading a multi-gigabyte index on a 32-bit build; a corrupt header declaring a bogus huge length.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• {} {detail}.
• truncated file
• {path} was written and committed, but syncing its parent dir

AI-assisted analysis of RyanCodrai/turbovec@ccab9f325e (2026-09-06). Data as JSON: /api/errors/5f7795ddfe12ffd4. Report an issue: GitHub.