tracel-ai/burn · error · std::io::Error

InvalidData

InvalidData

Error message

Storage boundaries not available for key '{}'. Cannot perform lazy loading.

What it means

Lazy loading in burn's PyTorch record reader requires a precomputed map of storage boundaries (byte offset and size per storage key) built from tensor metadata. `LazyData::read` deliberately has no fallback: if the key is absent from that map, it cannot know where the tensor bytes live in the file, so it raises InvalidData rather than reading the whole file eagerly.

Source

Thrown at crates/burn-store/src/pytorch/lazy_data.rs:267

            .storage_map
            .read()
            .unwrap_or_else(|poisoned| poisoned.into_inner());

        if let Some(ref map) = *storage_map
            && let Some(&(offset, size)) = map.get(storage_key)
        {
            // Load only this specific storage
            let mut file = File::open(&self.path)?;
            file.seek(std::io::SeekFrom::Start(self.data_offset + offset))?;

            let mut buffer = vec![0u8; size as usize];
            file.read_exact(&mut buffer)?;
            return Ok(buffer);
        }

        // NO FALLBACK! If we don't have storage boundaries, we cannot load data lazily
        // The storage map MUST be built from tensor metadata for lazy loading to work
        Err(std::io::Error::new(
            std::io::ErrorKind::InvalidData,
            format!(
                "Storage boundaries not available for key '{}'. Cannot perform lazy loading.",
                storage_key
            ),
        ))
    }
}

impl TarSource {
    /// Create a new TAR source by parsing storages data.
    ///
    /// # Arguments
    /// * `storages_data` - Raw storages blob with structure:
    ///   - Count pickle (number of storages)
    ///   - For each storage: metadata pickle + u64 num_elements + raw binary data
    pub fn new(storages_data: Vec<u8>) -> std::io::Result<Self> {
        use super::pickle_reader::{read_pickle, storage_type_to_element_size};

View on GitHub (pinned to d16f7ba2ed)

Solutions

Verify the key exists in the file's tensor metadata before lazy-reading it (list keys from the record/reader first)
Ensure the map builder ran successfully — re-open the file so storage boundaries are built from metadata
Fall back to eager (non-lazy) loading of the record, which reads storages without needing the boundary map
Check the file was produced by a supported torch.save format and is not truncated

Example fix

// before
let data = lazy_data.read("data/42")?; // panics with InvalidData if unmapped
// after
if lazy_data.has_storage("data/42") {
    let data = lazy_data.read("data/42")?;
} else {
    let record = full_loader.load(&path)?; // eager fallback
}

Defensive patterns

Strategy: try-catch

Validate before calling

fn can_lazy_read(src: &LazyData, key: &str) -> bool {
    let storage_key = key.split('/').next_back().unwrap_or(key);
    src.has_storage(storage_key)
}

Try / catch

match lazy_data.read(key) {
    Ok(bytes) => use_bytes(bytes),
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData
        && e.to_string().contains("Storage boundaries not available") =>
    {
        let record = eager_loader.load(path)?; // non-lazy fallback
    }
    Err(e) => return Err(e.into()),
}

Prevention

Validate keys against the record's tensor metadata before lazy reads
Re-open files cleanly so the storage map is rebuilt from metadata
Prefer eager loading for checkpoints from unusual torch versions
Never reuse a LazyData handle after the underlying file changed on disk

When it happens

Trigger: Calling `LazyData::read(key)` (via lazy record loading of a .pt/.pth file) with a storage key that was not present when the storage map was built — e.g. the metadata pickle lacked an entry for that tensor, the map was never initialized, or the key's path suffix (`data/N`) does not match any recorded storage.

Common situations: Loading a PyTorch checkpoint saved by an unusual producer (older/newer torch, quantized or sparse tensors) whose storage metadata doesn't line up; requesting a tensor key that doesn't exist in the file; using lazy loading on a file where zip metadata parsing partially failed.

Related errors

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/4b31d2e6f1e4b2ad. Report an issue: GitHub.