{"record":{"id":"36a3f1dbe5be6097","repo":"rustfs/rustfs","slug":"heal-shard-read-timed-out","errorCode":null,"errorMessage":"heal shard read timed out","messagePattern":"heal shard read timed out","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"warning","filePath":"crates/ecstore/src/erasure/coding/heal.rs","lineNumber":68,"sourceCode":"    {\n        let mut futures = FuturesUnordered::new();\n        for (index, reader) in readers.iter_mut().enumerate() {\n            let Some(reader) = reader else {\n                errs[index] = Some(Error::FileNotFound);\n                continue;\n            };\n\n            futures.push(Box::pin(async move {\n                let mut buf = vec![0; shard_size];\n                let read_result = if read_timeout.is_zero() {\n                    reader.read(&mut buf).await\n                } else {\n                    match tokio::time::timeout(read_timeout, reader.read(&mut buf)).await {\n                        Ok(result) => result,\n                        Err(_) => {\n                            return (\n                                index,\n                                Err(Error::from(io::Error::new(ErrorKind::TimedOut, \"heal shard read timed out\"))),\n                                true,\n                            );\n                        }\n                    }\n                };\n\n                match read_result {\n                    Ok(n) => {\n                        buf.truncate(n);\n                        (index, Ok(buf), false)\n                    }\n                    Err(err) => {\n                        let should_retire = err.kind() == ErrorKind::TimedOut;\n                        (index, Err(Error::from(err)), should_retire)\n                    }\n                }\n            }));\n        }","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/rustfs/rustfs/blob/9e6e02ea09c86bedf44c7bd64a74ea02a0cff1de/crates/ecstore/src/erasure/coding/heal.rs#L50-L86","documentation":"During erasure heal, each shard of a block is read under a per-read timeout. When a single shard read exceeds it, the result is converted to ErrorKind::TimedOut and that shard is treated as unavailable for the current heal pass (the timeout is flagged in the result tuple so it is not misreported as corruption).","triggerScenarios":"Heal running while one disk or endpoint is slow or hung: saturated IO, a dying disk in long retry loops, or network latency to remote endpoints exceeding the configured heal read timeout.","commonSituations":"Heal scheduled during backup or peak-load windows; NFS/network-backed volumes with latency spikes; disks that are failing slowly rather than cleanly erroring.","solutions":["Raise the heal read timeout setting for slow-but-healthy backends","Check the implicated drive's health and IO saturation (iostat, SMART) and replace failing disks","Schedule heal outside peak IO windows","Re-run heal after the slow endpoint recovers - a timeout implies unavailability, not corruption"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// preflight disk latency before a heal window\nfor disk in set.disks() {\n    if disk.read_latency_p99() > heal_read_timeout / 2 {\n        warn!(disk = disk.to_string(), \"heal may time out on this drive\");\n    }\n}","typeGuard":null,"tryCatchPattern":"match res {\n    Err(e) if matches!(e.kind(), std::io::ErrorKind::TimedOut) && e.to_string().contains(\"heal shard read\") => {\n        // transient: back off, rerun heal for this object after the slow disk recovers\n    }\n    other => other,\n}","preventionTips":["Size the heal read timeout to the slowest healthy backend","Schedule heal outside peak IO windows","Retire disks that repeatedly hit read timeouts during heal"],"tags":["heal","timeout","disk-io","erasure-coding"],"backgroundTag":"disk-read-timeout","analyzedSha":"9e6e02ea09c86bedf44c7bd64a74ea02a0cff1de","analyzedAt":"2026-08-16T20:34:17.560Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}