rustfs/rustfs · info · std::io::Error

Interrupted

Interrupted

Error message

tier free-version recovery cancelled

What it means

Free-version recovery walks tier free-version records with a cancellation token; when the token fires (node shutdown, task cancellation, operator abort) mid-walk, the scan stops and returns this Interrupted error instead of partial stats. Stats accumulated so far are discarded in favor of an explicit signal, and the retry markers (retry_cursor / next_bucket_marker / next_object_marker) exist so the walk can resume where it stopped. This is an expected, benign shutdown signal — not a fault.

Source

Thrown at crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs:216

    let mut retry_cursor = RetryCursor::new(bucket_marker, object_marker);
    for oi in page.items {
        if cancel_token.is_cancelled() {
            return Err(tier_free_version_recovery_cancelled());
        }
        retry_cursor.visit(&oi);
        if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(&api, oi).await) {
            let (bucket_marker, object_marker) = retry_cursor.retry_markers();
            stats.truncated = true;
            stats.next_bucket_marker = bucket_marker;
            stats.next_object_marker = object_marker;
            break;
        }
    }

    Ok(stats)
}

fn tier_free_version_recovery_cancelled() -> crate::error::Error {
    std::io::Error::new(std::io::ErrorKind::Interrupted, "tier free-version recovery cancelled").into()
}

fn tier_free_version_recovery_walk_shutdown_timed_out() -> crate::error::Error {
    std::io::Error::new(
        std::io::ErrorKind::TimedOut,
        "tier free-version recovery walk did not stop after cancellation",
    )
    .into()
}

fn record_recovered_free_version_enqueue(stats: &mut FreeVersionRecoveryStats, queued: bool) -> bool {
    stats.scanned += 1;
    if queued {
        stats.enqueued += 1;
        true
    } else {
        stats.failed += 1;

View on GitHub (pinned to 5dca076efe)

Solutions

1. Nothing to repair — restart or re-invoke recovery and pass the previous next_bucket_marker/next_object_marker to resume
2. If shutdowns repeatedly interrupt long walks, raise the per-run limit or schedule recovery during quiet windows
3. Distinguish from TimedOut sibling error: this one means cancellation worked correctly

// Pass through the cancellation token and check it before starting
if cancel_token.is_cancelled() {
    return Ok(default_stats()); // skip the run instead of eating an Interrupted
}
let stats = recover_tier_free_versions(api, limit, marker, &cancel_token).await?;

fn is_cancellation(e: &crate::error::Error) -> bool {
    matches!(e.io_error_kind(), Some(std::io::ErrorKind::Interrupted))
        && e.to_string().contains("recovery cancelled")
}

match recover_tier_free_versions(api, limit, marker, &cancel).await {
    Err(e) if is_cancellation(&e) => {
        // benign shutdown: persist next_bucket_marker/next_object_marker and resume later
        Ok(partial)
    }
    other => other,
}

When it happens

Trigger: recover/free-version scan is running when cancel_token.cancelled() resolves (select! at line 310), the loop checks is_cancelled() between items (line 189/319), or the walker channel closes due to cancellation (line 422/438). Any graceful shutdown or cancel during an active recovery produces it.

Common situations: Rolling restarts while the lifecycle free-version scanner is mid-bucket; cancellation triggered by reconfiguration of lifecycle/tier settings; CI timeouts cancelling the recovery task.

Related errors

• TimedOut
• InvalidData
• Heal task cancelled
• InvalidInput
• InvalidObjectState

AI-assisted analysis of rustfs/rustfs@5dca076efe (2026-08-20). Data as JSON: /api/errors/d171f66d2a3028e2. Report an issue: GitHub.