tracel-ai/burn · error

Failed to write weights

Error message

Failed to write weights

What it means

After creating the cache file, `download_if_needed` in the afine weights module panics with `file.write_all(&bytes).expect("Failed to write weights")` if writing the downloaded `afine.pth` bytes fails. File creation succeeded, so the file handle is open; the write itself fails typically because the disk filled up mid-write or the file was truncated/locked. Leaving a partial `afine.pth` behind can poison the cache for later runs.

Source

Thrown at crates/burn-train/src/metric/vision/afine/weights.rs:63

/// Cached filename inside `~/.cache/burn-dataset/afine/`.
const CACHE_FILENAME: &str = "afine.pth";

fn get_cache_dir() -> PathBuf {
    let cache_dir = dirs::cache_dir()
        .expect("Could not get cache directory")
        .join("burn-dataset")
        .join("afine");
    if !cache_dir.exists() {
        create_dir_all(&cache_dir).expect("Failed to create cache directory");
    }
    cache_dir
}

fn download_if_needed(url: &str, cache_path: &PathBuf, message: &str) {
    if !cache_path.exists() {
        let bytes = download_file_as_bytes(url, message);
        let mut file = File::create(cache_path).expect("Failed to create cache file");
        file.write_all(&bytes).expect("Failed to write weights");
    }
}

/// Download `afine.pth` (if not cached) and load all six shards into
/// the matching submodules of an `Afine` previously produced by
/// `AfineConfig::init`.
///
/// Errors during loading are logged via `log::warn!` and the function
/// returns the module unconditionally — `allow_partial(true)` plus
/// per-shard regex remapping mean unmapped or unknown checkpoint keys
/// are silently dropped, matching the behaviour of LPIPS, DISTS, and
/// FID's pretrained loaders.
pub(crate) fn load_pretrained_weights(mut afine: Afine) -> Afine {
    let cache_dir = get_cache_dir();
    let cache_path = cache_dir.join(CACHE_FILENAME);
    download_if_needed(AFINE_URL, &cache_path, "Downloading A-FINE weights...");

    afine.clip_visual = load_clip_shard(afine.clip_visual, &cache_path);

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Free disk space on the cache filesystem (`df -h ~/.cache`, clean caches) and retry
2. Delete the partial `afine.pth` left by the failed write so a truncated file is never reused
3. Point XDG_CACHE_HOME to a volume with enough space for the weights
4. Serialize concurrent runs (or use a per-run temp path) so two processes don't write the same cache file

Example fix

// before (shell)
cargo run -- train   # panics: Failed to write weights
// after (shell)
df -h ~/.cache                 # check space
rm -f ~/.cache/burn-dataset/afine/afine.pth
cargo run -- train

// verify enough free space before triggering the multi-GB weight download
fn has_free_space(dir: &std::path::Path, need_bytes: u64) -> bool {
    fs2::available_space(dir).map(|a| a > need_bytes).unwrap_or(false)
}
assert!(has_free_space(&dirs::cache_dir().unwrap(), 4 * 1024 * 1024 * 1024));

fn partial_cache_suspect(path: &std::path::Path) -> bool {
    // a zero-byte or unexpectedly small file likely came from a failed write
    std::fs::metadata(path).map(|m| m.len() < 1024).unwrap_or(false)
}

match std::panic::catch_unwind(load_pretrained_weights) {
    Ok(w) => w,
    Err(_) => { let _ = std::fs::remove_file(cache.join("afine.pth")); // drop partial file
                retry_with_backoff(3, load_pretrained_weights) }
}

When it happens

Trigger: `load_pretrained_weights` (afine) downloads weights (multi-GB file) and `write_all` fails mid-write: disk full or quota exceeded during the write, I/O error on the storage device, or the file being removed/locked concurrently.

Common situations: Full disk on CI runners when caching large pretrained weights; container ephemeral storage quota hit; NFS/network volume I/O errors; two processes downloading the same weights concurrently into the same cache path.

Related errors

• Failed to write weights
• Failed to create cache file
• Failed to create cache file
• Should be able to create the new file '{}': {}
• Can delete model checkpoint.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/e0cc807373653b33. Report an issue: GitHub.