{"record":{"id":"bc2569adafb501b8","repo":"xai-org/grok-build","slug":"failed-to-hash-snapshot","errorCode":null,"errorMessage":"Failed to hash snapshot {}","messagePattern":"Failed to hash snapshot (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-file-utils/src/queue.rs","lineNumber":1897,"sourceCode":"/// copy that would exceed the budget routes to the bounded inline fallback.\nfn snapshot_route(disk_bytes: u64, over_budget: bool) -> SnapshotRoute {\n    if disk_bytes > 0 && over_budget {\n        SnapshotRoute::InlineFallback\n    } else {\n        SnapshotRoute::Queue\n    }\n}\n/// Verify the (immutable) snapshot at `path`. Streamed in 8 KB chunks via the\n/// shared `sha256_hex_from_file` — never a whole-file read, so multi-GB\n/// snapshots stay off the heap. Distinguishes a genuine mismatch/missing\n/// (→ `Stale`) from a transient read error (→ `Io`).\nfn check_snapshot(path: &Path, expected_sha256: &str) -> SnapshotCheck {\n    match crate::sha256_hex_from_file(path, None) {\n        Ok(actual) if actual == expected_sha256 => SnapshotCheck::Match,\n        Ok(_) => SnapshotCheck::Stale,\n        Err(e) if e.kind() == std::io::ErrorKind::NotFound => SnapshotCheck::Stale,\n        Err(e) => SnapshotCheck::Io(\n            anyhow::Error::new(e).context(format!(\"Failed to hash snapshot {}\", path.display())),\n        ),\n    }\n}\n/// Settle an item leaving the queue: drop `inflight` FIRST (so it never\n/// exceeds `pending`), then `pending`/`pending_bytes`, then notify.\nfn settle_pending(stats: &UploadQueueStats, accounted_bytes: u64) {\n    stats.inflight.fetch_sub(1, Ordering::Relaxed);\n    stats.pending.fetch_sub(1, Ordering::Relaxed);\n    stats\n        .pending_bytes\n        .fetch_sub(accounted_bytes, Ordering::Relaxed);\n    stats.notify_transition();\n}\n/// Process a single upload queue item: age check, upload with retries, optional streaming compression.\nasync fn process_item(\n    mut item: UploadQueueItem,\n    resolver: &Arc<dyn TraceExportSource>,\n    retry_policy: &UploadRetryPolicy,","sourceCodeStart":1879,"sourceCodeEnd":1915,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-file-utils/src/queue.rs#L1879-L1915","documentation":"Produced by `check_snapshot` when hashing the staged snapshot file (via sha256_hex_from_file) fails with an io error other than NotFound. The error is wrapped with 'Failed to hash snapshot {path}'. It means integrity verification of the queued upload could not be performed — distinct from a hash mismatch (Stale), which is reported separately.","triggerScenarios":"sha256_hex_from_file on the snapshot path returns Err: snapshot deleted between staging and hashing, permission denied on the queue file, or I/O errors reading from a full/failing disk.","commonSituations":"External cleanup pruned queue temp files mid-flight; queue_dir on an unstable/network volume (NFS) with transient read errors; permissions changed by another process; concurrent removal during queue shutdown.","solutions":["Re-snapshot the source file and retry the upload, since the staged copy may have been removed or corrupted.","Check that no external cleanup (tmpwatch, manual rm) targets the queue_dir while uploads run.","Verify permissions on the snapshot path and queue directory.","If the queue volume is unreliable, move queue_dir to local disk and upload from there."],"exampleFix":"// before\nlet url = queue.enqueue_upload(item).await?;\n// after\nlet url = match queue.enqueue_upload(item).await {\n    Ok(u) => u,\n    Err(e) if e.to_string().contains(\"Failed to hash snapshot\") => {\n        tracing::warn!(%e, \"snapshot unreadable; retrying upload once\");\n        queue.enqueue_upload(item).await?\n    }\n    Err(e) => return Err(e),\n};","handlingStrategy":"validation","validationCode":"fn hashable(path: &Path) -> bool {\n    std::fs::File::open(path)\n        .and_then(|f| f.metadata())\n        .map(|m| m.len() > 0)\n        .unwrap_or(false)\n}\nif !hashable(&snapshot_path) { re_snapshot_and_retry(); }","typeGuard":"fn is_hash_failure(err: &anyhow::Error) -> bool {\n    err.chain().any(|c| c.to_string().starts_with(\"Failed to hash snapshot\"))\n}","tryCatchPattern":"match enqueue_with_integrity_check(item).await {\n    Ok(url) => url,\n    Err(e) if is_hash_failure(&e) => {\n        tracing::warn!(%e, \"snapshot unreadable; re-staging once\");\n        re_snapshot(item).await?\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Prevent external processes from deleting queue temp files while uploads run","Place queue_dir on reliable local storage rather than network volumes","Re-verify the snapshot exists right before hashing","Treat repeated hash failures as a signal the queue volume is unhealthy and alert"],"tags":["upload","hashing","integrity","filesystem","io"],"backgroundTag":"snapshot-hash-failed","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}