rustfs/rustfs · warning · io::Error

heal shard read timed out

Error message

heal shard read timed out

What it means

During erasure heal, each shard of a block is read under a per-read timeout. When a single shard read exceeds it, the result is converted to ErrorKind::TimedOut and that shard is treated as unavailable for the current heal pass (the timeout is flagged in the result tuple so it is not misreported as corruption).

Source

Thrown at crates/ecstore/src/erasure/coding/heal.rs:68

    {
        let mut futures = FuturesUnordered::new();
        for (index, reader) in readers.iter_mut().enumerate() {
            let Some(reader) = reader else {
                errs[index] = Some(Error::FileNotFound);
                continue;
            };

            futures.push(Box::pin(async move {
                let mut buf = vec![0; shard_size];
                let read_result = if read_timeout.is_zero() {
                    reader.read(&mut buf).await
                } else {
                    match tokio::time::timeout(read_timeout, reader.read(&mut buf)).await {
                        Ok(result) => result,
                        Err(_) => {
                            return (
                                index,
                                Err(Error::from(io::Error::new(ErrorKind::TimedOut, "heal shard read timed out"))),
                                true,
                            );
                        }
                    }
                };

                match read_result {
                    Ok(n) => {
                        buf.truncate(n);
                        (index, Ok(buf), false)
                    }
                    Err(err) => {
                        let should_retire = err.kind() == ErrorKind::TimedOut;
                        (index, Err(Error::from(err)), should_retire)
                    }
                }
            }));
        }

View on GitHub (pinned to 9e6e02ea09)

Solutions

1. Raise the heal read timeout setting for slow-but-healthy backends
2. Check the implicated drive's health and IO saturation (iostat, SMART) and replace failing disks
3. Schedule heal outside peak IO windows
4. Re-run heal after the slow endpoint recovers - a timeout implies unavailability, not corruption

// preflight disk latency before a heal window
for disk in set.disks() {
    if disk.read_latency_p99() > heal_read_timeout / 2 {
        warn!(disk = disk.to_string(), "heal may time out on this drive");
    }
}

match res {
    Err(e) if matches!(e.kind(), std::io::ErrorKind::TimedOut) && e.to_string().contains("heal shard read") => {
        // transient: back off, rerun heal for this object after the slow disk recovers
    }
    other => other,
}

When it happens

Trigger: Heal running while one disk or endpoint is slow or hung: saturated IO, a dying disk in long retry loops, or network latency to remote endpoints exceeding the configured heal read timeout.

Common situations: Heal scheduled during backup or peak-load windows; NFS/network-backed volumes with latency spikes; disks that are failing slowly rather than cleanly erroring.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• shard read timed out
• TimedOut
• inconsistent read source shards
• shard read hedged after a slow shard
• UnexpectedEof

AI-assisted analysis of rustfs/rustfs@9e6e02ea09 (2026-08-16). Data as JSON: /api/errors/36a3f1dbe5be6097. Report an issue: GitHub.