{"record":{"id":"e0cc807373653b33","repo":"tracel-ai/burn","slug":"failed-to-write-weights","errorCode":null,"errorMessage":"Failed to write weights","messagePattern":"Failed to write weights","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/burn-train/src/metric/vision/afine/weights.rs","lineNumber":63,"sourceCode":"/// Cached filename inside `~/.cache/burn-dataset/afine/`.\nconst CACHE_FILENAME: &str = \"afine.pth\";\n\nfn get_cache_dir() -> PathBuf {\n    let cache_dir = dirs::cache_dir()\n        .expect(\"Could not get cache directory\")\n        .join(\"burn-dataset\")\n        .join(\"afine\");\n    if !cache_dir.exists() {\n        create_dir_all(&cache_dir).expect(\"Failed to create cache directory\");\n    }\n    cache_dir\n}\n\nfn download_if_needed(url: &str, cache_path: &PathBuf, message: &str) {\n    if !cache_path.exists() {\n        let bytes = download_file_as_bytes(url, message);\n        let mut file = File::create(cache_path).expect(\"Failed to create cache file\");\n        file.write_all(&bytes).expect(\"Failed to write weights\");\n    }\n}\n\n/// Download `afine.pth` (if not cached) and load all six shards into\n/// the matching submodules of an `Afine` previously produced by\n/// `AfineConfig::init`.\n///\n/// Errors during loading are logged via `log::warn!` and the function\n/// returns the module unconditionally — `allow_partial(true)` plus\n/// per-shard regex remapping mean unmapped or unknown checkpoint keys\n/// are silently dropped, matching the behaviour of LPIPS, DISTS, and\n/// FID's pretrained loaders.\npub(crate) fn load_pretrained_weights(mut afine: Afine) -> Afine {\n    let cache_dir = get_cache_dir();\n    let cache_path = cache_dir.join(CACHE_FILENAME);\n    download_if_needed(AFINE_URL, &cache_path, \"Downloading A-FINE weights...\");\n\n    afine.clip_visual = load_clip_shard(afine.clip_visual, &cache_path);","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-train/src/metric/vision/afine/weights.rs#L45-L81","documentation":"After creating the cache file, `download_if_needed` in the afine weights module panics with `file.write_all(&bytes).expect(\"Failed to write weights\")` if writing the downloaded `afine.pth` bytes fails. File creation succeeded, so the file handle is open; the write itself fails typically because the disk filled up mid-write or the file was truncated/locked. Leaving a partial `afine.pth` behind can poison the cache for later runs.","triggerScenarios":"`load_pretrained_weights` (afine) downloads weights (multi-GB file) and `write_all` fails mid-write: disk full or quota exceeded during the write, I/O error on the storage device, or the file being removed/locked concurrently.","commonSituations":"Full disk on CI runners when caching large pretrained weights; container ephemeral storage quota hit; NFS/network volume I/O errors; two processes downloading the same weights concurrently into the same cache path.","solutions":["Free disk space on the cache filesystem (`df -h ~/.cache`, clean caches) and retry","Delete the partial `afine.pth` left by the failed write so a truncated file is never reused","Point XDG_CACHE_HOME to a volume with enough space for the weights","Serialize concurrent runs (or use a per-run temp path) so two processes don't write the same cache file"],"exampleFix":"// before (shell)\ncargo run -- train   # panics: Failed to write weights\n// after (shell)\ndf -h ~/.cache                 # check space\nrm -f ~/.cache/burn-dataset/afine/afine.pth\ncargo run -- train","handlingStrategy":"retry","validationCode":"// verify enough free space before triggering the multi-GB weight download\nfn has_free_space(dir: &std::path::Path, need_bytes: u64) -> bool {\n    fs2::available_space(dir).map(|a| a > need_bytes).unwrap_or(false)\n}\nassert!(has_free_space(&dirs::cache_dir().unwrap(), 4 * 1024 * 1024 * 1024));","typeGuard":"fn partial_cache_suspect(path: &std::path::Path) -> bool {\n    // a zero-byte or unexpectedly small file likely came from a failed write\n    std::fs::metadata(path).map(|m| m.len() < 1024).unwrap_or(false)\n}","tryCatchPattern":"match std::panic::catch_unwind(load_pretrained_weights) {\n    Ok(w) => w,\n    Err(_) => { let _ = std::fs::remove_file(cache.join(\"afine.pth\")); // drop partial file\n                retry_with_backoff(3, load_pretrained_weights) }\n}","preventionTips":["Monitor free disk space on the cache volume before training/eval jobs","After any failure, delete the partial .pth so a truncated cache is never reused","Use a reliable local disk (not a flaky network volume) for the weights cache","Serialize weight downloads across jobs with a lock or a single warm-up job"],"tags":["panic","filesystem","disk-full","download"],"backgroundTag":"disk-full","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}