{"record":{"id":"62a178b6f57ad730","repo":"tracel-ai/burn","slug":"failed-to-write-weights-62a178","errorCode":null,"errorMessage":"Failed to write weights","messagePattern":"Failed to write weights","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/burn-train/src/metric/vision/dists/weights.rs","lineNumber":38,"sourceCode":"fn get_cache_dir() -> PathBuf {\n    let cache_dir = dirs::cache_dir()\n        .expect(\"Could not get cache directory\")\n        .join(\"burn-dataset\")\n        .join(\"dists\");\n\n    if !cache_dir.exists() {\n        create_dir_all(&cache_dir).expect(\"Failed to create cache directory\");\n    }\n\n    cache_dir\n}\n\n/// Download file if not cached.\nfn download_if_needed(url: &str, cache_path: &PathBuf, message: &str) {\n    if !cache_path.exists() {\n        let bytes = download_file_as_bytes(url, message);\n        let mut file = File::create(cache_path).expect(\"Failed to create cache file\");\n        file.write_all(&bytes).expect(\"Failed to write weights\");\n    }\n}\n\n/// Download and load pretrained weights into a DISTS module.\n///\n/// This loads both:\n/// 1. ImageNet pretrained VGG16 backbone weights\n/// 2. DISTS trained alpha/beta weights\n///\n/// Weights are cached in the user's cache directory to avoid re-downloading.\n///\n/// # Arguments\n///\n/// * `dists` - The DISTS module to load weights into.\n///\n/// # Returns\n///\n/// The DISTS module with loaded pretrained weights.","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-train/src/metric/vision/dists/weights.rs#L20-L56","documentation":"In the DISTS weights module, after `File::create` succeeds, `file.write_all(&bytes).expect(\"Failed to write weights\")` panics if writing the downloaded VGG16 backbone bytes fails. Since the handle is open and the directory was writable, this is almost always the filesystem refusing the write at that moment — disk full/quota, I/O error, or external truncation. A partial `.pth` may remain and be treated as a valid cache entry on the next run.","triggerScenarios":"DISTS `load_pretrained_weights` downloads `vgg16-397923af.pth` (~500MB) and `write_all` fails: disk filled during the write, NFS/EBS I/O error, or a concurrent process deleting the file mid-write.","commonSituations":"Full ephemeral storage in CI containers; quota limits on shared scratch; flaky network volumes; two jobs writing the same cache path concurrently.","solutions":["Free space on the cache filesystem and retry (`df -h ~/.cache`)","Delete the partial VGG16 file so a truncated cache entry is never loaded: `rm -f ~/.cache/burn-dataset/dists/vgg16-397923af.pth`","Move XDG_CACHE_HOME to a larger, more reliable volume","Prevent concurrent writers (file lock or per-job cache dir)"],"exampleFix":"// before (shell)\ncargo run -- dists   # panics: Failed to write weights\n// after (shell)\nrm -f ~/.cache/burn-dataset/dists/vgg16-397923af.pth\ndf -h ~/.cache   # ensure >1GB free\nexport XDG_CACHE_HOME=/bigvolume/cache\ncargo run -- dists","handlingStrategy":"retry","validationCode":"fn has_free_space(dir: &std::path::Path, need_bytes: u64) -> bool {\n    fs2::available_space(dir).map(|a| a > need_bytes).unwrap_or(false)\n}\n// VGG16 weights are ~500MB\nassert!(has_free_space(&dirs::cache_dir().unwrap(), 1024 * 1024 * 1024));","typeGuard":"fn partial_cache_suspect(path: &std::path::Path) -> bool {\n    std::fs::metadata(path).map(|m| m.len() < 1024).unwrap_or(false)\n}","tryCatchPattern":"match std::panic::catch_unwind(load_pretrained_weights) {\n    Ok(w) => w,\n    Err(_) => { let _ = std::fs::remove_file(dists_dir.join(\"vgg16-397923af.pth\"));\n                retry_with_backoff(3, load_pretrained_weights) }\n}","preventionTips":["Check free space/quota on the cache volume before jobs that download backbone weights","Remove partial .pth files after any write failure","Avoid NFS/network volumes for the weights cache; prefer reliable local SSD","Prevent concurrent writers to the same cache path"],"tags":["panic","filesystem","disk-full","download"],"backgroundTag":"disk-full","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}