rustfs/rustfs · error · std::io::Error

TimedOut

TimedOut

Error message

tier free-version recovery walk did not stop after cancellation

What it means

After cancellation was signalled, the recovery walker is expected to stop promptly; if the list/walk channel or task keeps producing beyond a bounded shutdown window, the code gives up on graceful stop and returns TimedOut ('walk did not stop after cancellation'). Unlike the Interrupted sibling, this means cancellation was ignored — a hung remote listing, a stuck channel, or a walker bug. It flags a leak: the walk task may still be running after the error returns.

Source

Thrown at crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs:220

        }
        retry_cursor.visit(&oi);
        if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(&api, oi).await) {
            let (bucket_marker, object_marker) = retry_cursor.retry_markers();
            stats.truncated = true;
            stats.next_bucket_marker = bucket_marker;
            stats.next_object_marker = object_marker;
            break;
        }
    }

    Ok(stats)
}

fn tier_free_version_recovery_cancelled() -> crate::error::Error {
    std::io::Error::new(std::io::ErrorKind::Interrupted, "tier free-version recovery cancelled").into()
}

fn tier_free_version_recovery_walk_shutdown_timed_out() -> crate::error::Error {
    std::io::Error::new(
        std::io::ErrorKind::TimedOut,
        "tier free-version recovery walk did not stop after cancellation",
    )
    .into()
}

fn record_recovered_free_version_enqueue(stats: &mut FreeVersionRecoveryStats, queued: bool) -> bool {
    stats.scanned += 1;
    if queued {
        stats.enqueued += 1;
        true
    } else {
        stats.failed += 1;
        false
    }
}

View on GitHub (pinned to 5dca076efe)

Solutions

1. Check remote-tier connectivity and add/tighten request timeouts on the tier client so LIST calls cannot hang indefinitely
2. Restart the affected node/task to reclaim the possibly-still-running walk
3. Review lock/IO health on the pool driving the walk if it is local-listing rather than remote-tier listing
4. If reproducible with a healthy remote, file an issue with the walker's shutdown path in the title

// Give the tier client a request deadline so LIST cannot ignore cancellation
// (configure on the tier/remote client before recovery starts)
// e.g. set per-request timeout in tier config; then cancellation has a bound.

fn is_walk_shutdown_timeout(e: &crate::error::Error) -> bool {
    matches!(e.io_error_kind(), Some(std::io::ErrorKind::TimedOut))
        && e.to_string().contains("did not stop after cancellation")
}

match recover_tier_free_versions(api, limit, marker, &cancel).await {
    Err(e) if is_walk_shutdown_timeout(&e) => {
        // the walk task may still be alive: restart the task/node and alert
        ops_alert("free-version walker ignored cancellation");
        Err(e)
    }
    other => other,
}

When it happens

Trigger: cancel fires but the underlying object-walk (list_tier_free_versions / walker task feeding items) fails to terminate within the shutdown wait (receive_error set at line 448, or the final join/stop check at line 504 times out). Typical root cause: a blocked remote-tier ListObjects call that never observes cancellation.

Common situations: Remote tier (S3-compatible backend) hanging on LIST with no request deadline; network partition to the remote endpoint during recovery; a walker implementation bug that drops the cancel signal.

Related errors

• Interrupted
• InvalidData
• Heal task cancelled
• capacity scan cancelled after hard budget expiry ({file_coun
• InvalidInput

AI-assisted analysis of rustfs/rustfs@5dca076efe (2026-08-20). Data as JSON: /api/errors/1f66ce13e6e70087. Report an issue: GitHub.