{"record":{"id":"6404de0d9f1547e1","repo":"rustfs/rustfs","slug":"timeout","errorCode":null,"errorMessage":"timeout","messagePattern":"timeout","errorType":"exception","errorClass":"DiskError","httpStatus":null,"severity":"error","filePath":"crates/ecstore/src/disk/error.rs","lineNumber":155,"sourceCode":"    #[error(\"Rename across devices not allowed, please fix your backend configuration\")]\n    CrossDeviceLink,\n\n    #[error(\"less data available than what was requested\")]\n    LessData,\n\n    #[error(\"more data was sent than what was advertised\")]\n    MoreData,\n\n    #[error(\"outdated XL meta\")]\n    OutdatedXLMeta,\n\n    #[error(\"part missing or corrupt\")]\n    PartMissingOrCorrupt,\n\n    #[error(\"No healing is required\")]\n    NoHealRequired,\n\n    #[error(\"method not allowed\")]\n    MethodNotAllowed,\n\n    #[error(\"erasure write quorum\")]\n    ErasureWriteQuorum,\n\n    #[error(\"erasure read quorum\")]\n    ErasureReadQuorum,\n\n    #[error(\"io error {0}\")]\n    Io(#[source] io::Error),\n\n    /// Never constructed locally by RustFS (only reachable through wire\n    /// decoding, and no current node sends it). The wire code is kept for\n    /// cross-version compatibility — do not renumber or remove (backlog#1831).\n    #[error(\"source stalled\")]\n    SourceStalled,\n\n    #[error(\"timeout\")]","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/rustfs/rustfs/blob/5dca076efed96e7b842de07c4c2111035ae7c7a2/crates/ecstore/src/disk/error.rs#L137-L173","documentation":"DiskError::Timeout means a disk-layer operation exceeded its time budget. Locally it is produced by disk stall detection (local.rs:2651 wraps the future with timeout(stall, fut)) and for remote cluster disks by deadline exhaustion in cluster/rpc/remote_disk.rs (the client converts elapsed deadlines to DiskError::Timeout, e.g. lines 1456/1477). It propagates to StorageError::Timeout, is recorded via metrics.record_timeout_error (disk_store.rs:1503), and is treated as transient by the RPC client (client.rs:200), rebalance (worker.rs:268), and heal paths.","triggerScenarios":"A local disk I/O call hangs longer than the configured disk stall timeout; a remote peer disk RPC outlives its operation/metadata deadline in remote_disk.rs; walk_dir exceeding its per-request timeout (pinned by tests in disk_store.rs:3025+).","commonSituations":"Slow or failing disks (NFS/NAS-backed volumes, dying HDDs), saturated Tokio blocking pools or event loops, network latency to peer nodes, stall/deadline configuration too tight for the workload or media.","solutions":["Check disk and filesystem health on the reporting node (iostat latency, dmesg I/O errors, smartctl) — a hung device is the most common root cause","Verify load on the blocking thread pool and on remote peers; saturated CPU or network makes deadlines fire","Tune the disk stall timeout and RPC deadlines upward for slow media","Retry the operation — Timeout is already classified transient by client.rs:200, rebalance worker.rs:268, and heal paths"],"exampleFix":"// before\nlet info = disk.get_disk_info().await?;\n\n// after\nlet mut attempt = 0;\nlet info = loop {\n    match disk.get_disk_info().await {\n        Ok(info) => break info,\n        Err(DiskError::Timeout) if attempt < 3 => {\n            attempt += 1;\n            tokio::time::sleep(Duration::from_millis(200 * attempt)).await;\n        }\n        Err(err) => return Err(err.into()),\n    }\n};","handlingStrategy":"retry","validationCode":null,"typeGuard":"fn is_disk_timeout(err: &DiskError) -> bool {\n    matches!(err, DiskError::Timeout)\n}","tryCatchPattern":"match res {\n    Ok(v) => v,\n    Err(err @ DiskError::Timeout) => {\n        // transient: backoff and retry a bounded number of times,\n        // then surface — repeated timeouts point at a hung disk\n        retry_bounded(op, 3, Duration::from_millis(200))\n    }\n    Err(other) => return Err(other),\n}","preventionTips":["Alert on the timeout metrics recorded via record_timeout_error (disk_store.rs:1503) — a rising rate precedes disk failure","Health-check slow media (smartctl, iostat) and size the stall/deadline config to the device's real latency","Treat DiskError::Timeout as transient in retry policies, matching client.rs:200 and rebalance worker.rs:268"],"tags":["disk-error","timeout","io","stall-detection"],"backgroundTag":"disk-io-timeout","analyzedSha":"5dca076efed96e7b842de07c4c2111035ae7c7a2","analyzedAt":"2026-08-20T21:57:04.799Z","contentChangedAt":"2026-08-20T21:57:04.799Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}