xai-org/grok-build · error · anyhow::Error

Failed to hash snapshot {}

Error message

Failed to hash snapshot {}

What it means

Produced by `check_snapshot` when hashing the staged snapshot file (via sha256_hex_from_file) fails with an io error other than NotFound. The error is wrapped with 'Failed to hash snapshot {path}'. It means integrity verification of the queued upload could not be performed — distinct from a hash mismatch (Stale), which is reported separately.

Source

Thrown at crates/codegen/xai-file-utils/src/queue.rs:1897

/// copy that would exceed the budget routes to the bounded inline fallback.
fn snapshot_route(disk_bytes: u64, over_budget: bool) -> SnapshotRoute {
    if disk_bytes > 0 && over_budget {
        SnapshotRoute::InlineFallback
    } else {
        SnapshotRoute::Queue
    }
}
/// Verify the (immutable) snapshot at `path`. Streamed in 8 KB chunks via the
/// shared `sha256_hex_from_file` — never a whole-file read, so multi-GB
/// snapshots stay off the heap. Distinguishes a genuine mismatch/missing
/// (→ `Stale`) from a transient read error (→ `Io`).
fn check_snapshot(path: &Path, expected_sha256: &str) -> SnapshotCheck {
    match crate::sha256_hex_from_file(path, None) {
        Ok(actual) if actual == expected_sha256 => SnapshotCheck::Match,
        Ok(_) => SnapshotCheck::Stale,
        Err(e) if e.kind() == std::io::ErrorKind::NotFound => SnapshotCheck::Stale,
        Err(e) => SnapshotCheck::Io(
            anyhow::Error::new(e).context(format!("Failed to hash snapshot {}", path.display())),
        ),
    }
}
/// Settle an item leaving the queue: drop `inflight` FIRST (so it never
/// exceeds `pending`), then `pending`/`pending_bytes`, then notify.
fn settle_pending(stats: &UploadQueueStats, accounted_bytes: u64) {
    stats.inflight.fetch_sub(1, Ordering::Relaxed);
    stats.pending.fetch_sub(1, Ordering::Relaxed);
    stats
        .pending_bytes
        .fetch_sub(accounted_bytes, Ordering::Relaxed);
    stats.notify_transition();
}
/// Process a single upload queue item: age check, upload with retries, optional streaming compression.
async fn process_item(
    mut item: UploadQueueItem,
    resolver: &Arc<dyn TraceExportSource>,
    retry_policy: &UploadRetryPolicy,

View on GitHub (pinned to bc7f02eddd)

Solutions

1. Re-snapshot the source file and retry the upload, since the staged copy may have been removed or corrupted.
2. Check that no external cleanup (tmpwatch, manual rm) targets the queue_dir while uploads run.
3. Verify permissions on the snapshot path and queue directory.
4. If the queue volume is unreliable, move queue_dir to local disk and upload from there.

Example fix

// before
let url = queue.enqueue_upload(item).await?;
// after
let url = match queue.enqueue_upload(item).await {
    Ok(u) => u,
    Err(e) if e.to_string().contains("Failed to hash snapshot") => {
        tracing::warn!(%e, "snapshot unreadable; retrying upload once");
        queue.enqueue_upload(item).await?
    }
    Err(e) => return Err(e),
};

fn hashable(path: &Path) -> bool {
    std::fs::File::open(path)
        .and_then(|f| f.metadata())
        .map(|m| m.len() > 0)
        .unwrap_or(false)
}
if !hashable(&snapshot_path) { re_snapshot_and_retry(); }

fn is_hash_failure(err: &anyhow::Error) -> bool {
    err.chain().any(|c| c.to_string().starts_with("Failed to hash snapshot"))
}

match enqueue_with_integrity_check(item).await {
    Ok(url) => url,
    Err(e) if is_hash_failure(&e) => {
        tracing::warn!(%e, "snapshot unreadable; re-staging once");
        re_snapshot(item).await?
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: sha256_hex_from_file on the snapshot path returns Err: snapshot deleted between staging and hashing, permission denied on the queue file, or I/O errors reading from a full/failing disk.

Common situations: External cleanup pruned queue temp files mid-flight; queue_dir on an unstable/network volume (NFS) with transient read errors; permissions changed by another process; concurrent removal during queue shutdown.

Related errors

• Failed to snapshot {} into upload queue
• Failed to set working directory to {:?}: {}
• failed to read {}: {e}
• failed to write {}: {e}
• failed to create chrome trace {:?}: {}

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/bc2569adafb501b8. Report an issue: GitHub.