{"record":{"id":"1f66ce13e6e70087","repo":"rustfs/rustfs","slug":"timedout","errorCode":"TimedOut","errorMessage":"tier free-version recovery walk did not stop after cancellation","messagePattern":"tier free-version recovery walk did not stop after cancellation","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs","lineNumber":220,"sourceCode":"        }\n        retry_cursor.visit(&oi);\n        if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(&api, oi).await) {\n            let (bucket_marker, object_marker) = retry_cursor.retry_markers();\n            stats.truncated = true;\n            stats.next_bucket_marker = bucket_marker;\n            stats.next_object_marker = object_marker;\n            break;\n        }\n    }\n\n    Ok(stats)\n}\n\nfn tier_free_version_recovery_cancelled() -> crate::error::Error {\n    std::io::Error::new(std::io::ErrorKind::Interrupted, \"tier free-version recovery cancelled\").into()\n}\n\nfn tier_free_version_recovery_walk_shutdown_timed_out() -> crate::error::Error {\n    std::io::Error::new(\n        std::io::ErrorKind::TimedOut,\n        \"tier free-version recovery walk did not stop after cancellation\",\n    )\n    .into()\n}\n\nfn record_recovered_free_version_enqueue(stats: &mut FreeVersionRecoveryStats, queued: bool) -> bool {\n    stats.scanned += 1;\n    if queued {\n        stats.enqueued += 1;\n        true\n    } else {\n        stats.failed += 1;\n        false\n    }\n}\n","sourceCodeStart":202,"sourceCodeEnd":238,"githubUrl":"https://github.com/rustfs/rustfs/blob/5dca076efed96e7b842de07c4c2111035ae7c7a2/crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs#L202-L238","documentation":"After cancellation was signalled, the recovery walker is expected to stop promptly; if the list/walk channel or task keeps producing beyond a bounded shutdown window, the code gives up on graceful stop and returns TimedOut ('walk did not stop after cancellation'). Unlike the Interrupted sibling, this means cancellation was ignored — a hung remote listing, a stuck channel, or a walker bug. It flags a leak: the walk task may still be running after the error returns.","triggerScenarios":"cancel fires but the underlying object-walk (list_tier_free_versions / walker task feeding items) fails to terminate within the shutdown wait (receive_error set at line 448, or the final join/stop check at line 504 times out). Typical root cause: a blocked remote-tier ListObjects call that never observes cancellation.","commonSituations":"Remote tier (S3-compatible backend) hanging on LIST with no request deadline; network partition to the remote endpoint during recovery; a walker implementation bug that drops the cancel signal.","solutions":["Check remote-tier connectivity and add/tighten request timeouts on the tier client so LIST calls cannot hang indefinitely","Restart the affected node/task to reclaim the possibly-still-running walk","Review lock/IO health on the pool driving the walk if it is local-listing rather than remote-tier listing","If reproducible with a healthy remote, file an issue with the walker's shutdown path in the title"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Give the tier client a request deadline so LIST cannot ignore cancellation\n// (configure on the tier/remote client before recovery starts)\n// e.g. set per-request timeout in tier config; then cancellation has a bound.","typeGuard":"fn is_walk_shutdown_timeout(e: &crate::error::Error) -> bool {\n    matches!(e.io_error_kind(), Some(std::io::ErrorKind::TimedOut))\n        && e.to_string().contains(\"did not stop after cancellation\")\n}","tryCatchPattern":"match recover_tier_free_versions(api, limit, marker, &cancel).await {\n    Err(e) if is_walk_shutdown_timeout(&e) => {\n        // the walk task may still be alive: restart the task/node and alert\n        ops_alert(\"free-version walker ignored cancellation\");\n        Err(e)\n    }\n    other => other,\n}","preventionTips":["Set explicit request timeouts on remote-tier clients so LIST calls cannot hang","Alert on TimedOut-after-cancel, not on the benign Interrupted signal","Restart the walker task after this error; its channel may be leaked"],"tags":["tiering","lifecycle","recovery","cancellation","timeout","hang"],"backgroundTag":"shutdown-hang","analyzedSha":"5dca076efed96e7b842de07c4c2111035ae7c7a2","analyzedAt":"2026-08-20T21:57:04.799Z","contentChangedAt":"2026-08-20T21:57:04.799Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}