tracel-ai/burn · error

Failed to write weights

Error message

Failed to write weights

What it means

In the DISTS weights module, after `File::create` succeeds, `file.write_all(&bytes).expect("Failed to write weights")` panics if writing the downloaded VGG16 backbone bytes fails. Since the handle is open and the directory was writable, this is almost always the filesystem refusing the write at that moment — disk full/quota, I/O error, or external truncation. A partial `.pth` may remain and be treated as a valid cache entry on the next run.

Source

Thrown at crates/burn-train/src/metric/vision/dists/weights.rs:38

fn get_cache_dir() -> PathBuf {
    let cache_dir = dirs::cache_dir()
        .expect("Could not get cache directory")
        .join("burn-dataset")
        .join("dists");

    if !cache_dir.exists() {
        create_dir_all(&cache_dir).expect("Failed to create cache directory");
    }

    cache_dir
}

/// Download file if not cached.
fn download_if_needed(url: &str, cache_path: &PathBuf, message: &str) {
    if !cache_path.exists() {
        let bytes = download_file_as_bytes(url, message);
        let mut file = File::create(cache_path).expect("Failed to create cache file");
        file.write_all(&bytes).expect("Failed to write weights");
    }
}

/// Download and load pretrained weights into a DISTS module.
///
/// This loads both:
/// 1. ImageNet pretrained VGG16 backbone weights
/// 2. DISTS trained alpha/beta weights
///
/// Weights are cached in the user's cache directory to avoid re-downloading.
///
/// # Arguments
///
/// * `dists` - The DISTS module to load weights into.
///
/// # Returns
///
/// The DISTS module with loaded pretrained weights.

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Free space on the cache filesystem and retry (`df -h ~/.cache`)
2. Delete the partial VGG16 file so a truncated cache entry is never loaded: `rm -f ~/.cache/burn-dataset/dists/vgg16-397923af.pth`
3. Move XDG_CACHE_HOME to a larger, more reliable volume
4. Prevent concurrent writers (file lock or per-job cache dir)

Example fix

// before (shell)
cargo run -- dists   # panics: Failed to write weights
// after (shell)
rm -f ~/.cache/burn-dataset/dists/vgg16-397923af.pth
df -h ~/.cache   # ensure >1GB free
export XDG_CACHE_HOME=/bigvolume/cache
cargo run -- dists

fn has_free_space(dir: &std::path::Path, need_bytes: u64) -> bool {
    fs2::available_space(dir).map(|a| a > need_bytes).unwrap_or(false)
}
// VGG16 weights are ~500MB
assert!(has_free_space(&dirs::cache_dir().unwrap(), 1024 * 1024 * 1024));

fn partial_cache_suspect(path: &std::path::Path) -> bool {
    std::fs::metadata(path).map(|m| m.len() < 1024).unwrap_or(false)
}

match std::panic::catch_unwind(load_pretrained_weights) {
    Ok(w) => w,
    Err(_) => { let _ = std::fs::remove_file(dists_dir.join("vgg16-397923af.pth"));
                retry_with_backoff(3, load_pretrained_weights) }
}

When it happens

Trigger: DISTS `load_pretrained_weights` downloads `vgg16-397923af.pth` (~500MB) and `write_all` fails: disk filled during the write, NFS/EBS I/O error, or a concurrent process deleting the file mid-write.

Common situations: Full ephemeral storage in CI containers; quota limits on shared scratch; flaky network volumes; two jobs writing the same cache path concurrently.

Related errors

• Failed to write weights
• Failed to create cache file
• Failed to create cache file
• Should be able to create the new file '{}': {}
• Can delete model checkpoint.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/62a178b6f57ad730. Report an issue: GitHub.