rustfs/rustfs · error · io::Error

TimedOut

TimedOut

Error message

inspect archive generation timed out

What it means

An io::Error with kind TimedOut raised when a single read from the underlying xl.meta/drive reader exceeded the remaining budget of the inspect-archive operation's absolute deadline (INSPECT_ARCHIVE_MAX_DURATION = 30 seconds, set when the request starts). The deadline is enforced with tokio::time::timeout_at around each read, so one stalled drive read is enough to abort the whole encrypted stream. The error is terminal for this request: the design deliberately refuses to let a slow disk stretch an admin diagnostic past its cap.

Source

Thrown at rustfs/src/admin/handlers/inspect_archive.rs:519

    header.extend_from_slice(FORMAT_MAGIC);
    header.extend_from_slice(&INSPECT_ARCHIVE_VERSION.to_be_bytes());
    header.extend_from_slice(&u32::try_from(ARCHIVE_CHUNK_SIZE).map_err(io::Error::other)?.to_be_bytes());
    header.extend_from_slice(&wrapped_len.to_be_bytes());
    header.extend_from_slice(&nonce_prefix);
    header.extend_from_slice(&wrapped_key);

    let mut sent = 0;
    send_bounded(&tx, &mut sent, Bytes::from(header), deadline).await?;
    let mut digest = Sha256::new();
    let mut buffer = vec![0_u8; ARCHIVE_CHUNK_SIZE];
    let mut counter = 0_u32;

    loop {
        let read = tokio::select! {
            biased;
            _ = tx.closed() => return Err(io::Error::new(io::ErrorKind::BrokenPipe, "inspect archive client disconnected")),
            result = tokio::time::timeout_at(deadline, reader.read(&mut buffer)) => {
                result.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "inspect archive generation timed out"))??
            }
        };
        if read == 0 {
            break;
        }
        digest.update(&buffer[..read]);
        let framed = encrypt_record(&cipher, &nonce_prefix, RECORD_DATA, counter, &buffer[..read])?;
        send_bounded(&tx, &mut sent, framed, deadline).await?;
        counter = counter
            .checked_add(1)
            .ok_or_else(|| io::Error::other("inspect archive record counter exhausted"))?;
    }

    match tokio::time::timeout_at(deadline, completion).await {
        Err(_) => return Err(io::Error::new(io::ErrorKind::TimedOut, "inspect archive generation timed out")),
        Ok(Ok(Ok(()))) => {}
        Ok(Ok(Err(error))) => return Err(error),
        Ok(Err(error)) => return Err(io::Error::other(format!("inspect archive producer ended unexpectedly: {error}"))),

View on GitHub (pinned to 201c653dcd)

Solutions

1. Check drive health and latency first (this error usually marks a slow or failing disk, which is exactly what inspect is probing): review dmesg/smart/IO error rates for the drives backing the affected bucket.
2. Retry the request when the node is less loaded, or narrow the inspection scope (single object, smaller bucket) so reads fit in the 30s budget.
3. If large-archive inspection legitimately needs longer on your hardware, raise INSPECT_ARCHIVE_MAX_DURATION in rustfs/src/admin/handlers/inspect_archive.rs and rebuild — the constant is the single knob that governs collection, streaming, and encryption.
4. Verify no client/proxy timeout interplay: the server deadline is independent, so fixing only the client will not stop this error.

Example fix

// before (rustfs/src/admin/handlers/inspect_archive.rs:68)
pub const INSPECT_ARCHIVE_MAX_DURATION: Duration = Duration::from_secs(30);
// after: allow large archives on slow drives (rebuild rustfs)
pub const INSPECT_ARCHIVE_MAX_DURATION: Duration = Duration::from_secs(120);

// before requesting an archive, sanity-check the target drives respond quickly
async fn drive_responsive(path: &str) -> bool {
    tokio::fs::metadata(path).await.is_ok() && tokio::time::timeout(
        std::time::Duration::from_secs(2),
        tokio::fs::read(format!("{path}/format.json")),
    ).await.is_ok()
}

fn is_deadline_timeout(e: &std::io::Error) -> bool {
    e.kind() == std::io::ErrorKind::TimedOut
}

match response {
    Err(e) if e.kind() == std::io::ErrorKind::TimedOut => {
        // fixed 30s server budget exceeded: narrow the scope or retry off-peak,
        // do not blind-loop — a stalled drive will time out again
        narrow_scope_or_retry_later();
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Invoking the inspect-archive admin endpoint for a bucket/object whose metadata read blocks: an NFS/network drive hanging, a drive in error/recovery state, VM storage stall, or simply an archive large enough that sequential ARCHIVE_CHUNK_SIZE reads cannot finish inside the fixed 30-second window shared with artifact collection and encryption.

Common situations: Degraded or ejecting drives during support diagnostics; overloaded nodes where IO latency spikes; inspecting very large objects/metadata sets on slower spinning disks; running under heavy concurrent load so read latency exceeds the fixed budget.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• shard read timed out
• Timeout after {} files
• capacity scan exceeded hard wall-clock budget of {budget:?}
• BrokenPipe
• failed to write hash and data

AI-assisted analysis of rustfs/rustfs@201c653dcd (2026-08-20). Data as JSON: /api/errors/1d08504a32fe91d2. Report an issue: GitHub.