rustfs/rustfs · error · DiskError

timeout

Error message

timeout

What it means

DiskError::Timeout means a disk-layer operation exceeded its time budget. Locally it is produced by disk stall detection (local.rs:2651 wraps the future with timeout(stall, fut)) and for remote cluster disks by deadline exhaustion in cluster/rpc/remote_disk.rs (the client converts elapsed deadlines to DiskError::Timeout, e.g. lines 1456/1477). It propagates to StorageError::Timeout, is recorded via metrics.record_timeout_error (disk_store.rs:1503), and is treated as transient by the RPC client (client.rs:200), rebalance (worker.rs:268), and heal paths.

Source

Thrown at crates/ecstore/src/disk/error.rs:155

    #[error("Rename across devices not allowed, please fix your backend configuration")]
    CrossDeviceLink,

    #[error("less data available than what was requested")]
    LessData,

    #[error("more data was sent than what was advertised")]
    MoreData,

    #[error("outdated XL meta")]
    OutdatedXLMeta,

    #[error("part missing or corrupt")]
    PartMissingOrCorrupt,

    #[error("No healing is required")]
    NoHealRequired,

    #[error("method not allowed")]
    MethodNotAllowed,

    #[error("erasure write quorum")]
    ErasureWriteQuorum,

    #[error("erasure read quorum")]
    ErasureReadQuorum,

    #[error("io error {0}")]
    Io(#[source] io::Error),

    /// Never constructed locally by RustFS (only reachable through wire
    /// decoding, and no current node sends it). The wire code is kept for
    /// cross-version compatibility — do not renumber or remove (backlog#1831).
    #[error("source stalled")]
    SourceStalled,

    #[error("timeout")]

View on GitHub (pinned to 5dca076efe)

Solutions

1. Check disk and filesystem health on the reporting node (iostat latency, dmesg I/O errors, smartctl) — a hung device is the most common root cause
2. Verify load on the blocking thread pool and on remote peers; saturated CPU or network makes deadlines fire
3. Tune the disk stall timeout and RPC deadlines upward for slow media
4. Retry the operation — Timeout is already classified transient by client.rs:200, rebalance worker.rs:268, and heal paths

Example fix

// before
let info = disk.get_disk_info().await?;

// after
let mut attempt = 0;
let info = loop {
    match disk.get_disk_info().await {
        Ok(info) => break info,
        Err(DiskError::Timeout) if attempt < 3 => {
            attempt += 1;
            tokio::time::sleep(Duration::from_millis(200 * attempt)).await;
        }
        Err(err) => return Err(err.into()),
    }
};

fn is_disk_timeout(err: &DiskError) -> bool {
    matches!(err, DiskError::Timeout)
}

match res {
    Ok(v) => v,
    Err(err @ DiskError::Timeout) => {
        // transient: backoff and retry a bounded number of times,
        // then surface — repeated timeouts point at a hung disk
        retry_bounded(op, 3, Duration::from_millis(200))
    }
    Err(other) => return Err(other),
}

When it happens

Trigger: A local disk I/O call hangs longer than the configured disk stall timeout; a remote peer disk RPC outlives its operation/metadata deadline in remote_disk.rs; walk_dir exceeding its per-request timeout (pinned by tests in disk_store.rs:3025+).

Common situations: Slow or failing disks (NFS/NAS-backed volumes, dying HDDs), saturated Tokio blocking pools or event loops, network latency to peer nodes, stall/deadline configuration too tight for the workload or media.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• shard read timed out
• Timeout after {} files
• TimedOut
• failed to write hash and data
• shard read hedged after a slow shard

AI-assisted analysis of rustfs/rustfs@5dca076efe (2026-08-20). Data as JSON: /api/errors/6404de0d9f1547e1. Report an issue: GitHub.