rustfs/rustfs · error

remote tier cannot discover the outcome of a probe PUT

Error message

remote tier cannot discover the outcome of a probe PUT

What it means

When a probe PUT's candidate listing on the remote tier yields Unsupported, remove_discovered_probe_candidate cannot discover which remote object version the probe created and therefore cannot clean it up. The backend signals that it cannot determine the outcome of the probe PUT, so the warm-backend check fails rather than guessing.

Source

Thrown at crates/ecstore/src/services/tier/warm_backend.rs:560

async fn remove_discovered_probe_candidate(
    w: &WarmBackendImpl,
    probe_object: &str,
    candidate: TransitionCandidateProbe,
) -> Result<bool, std::io::Error> {
    match candidate {
        TransitionCandidateProbe::Missing => Ok(false),
        TransitionCandidateProbe::VersionedPresent(remote_version_id) => {
            w.remove_exact(probe_object, &remote_version_id).await?;
            Ok(true)
        }
        TransitionCandidateProbe::UnversionedPresent => {
            w.remove(probe_object, "").await?;
            Ok(true)
        }
        TransitionCandidateProbe::Ambiguous => {
            Err(std::io::Error::other("remote tier probe PUT produced multiple possible versions"))
        }
        TransitionCandidateProbe::Unsupported => Err(std::io::Error::new(
            std::io::ErrorKind::Unsupported,
            "remote tier cannot discover the outcome of a probe PUT",
        )),
    }
}

async fn compensate_uncertain_probe_put(
    w: &WarmBackendImpl,
    probe_object: &str,
    settle_deadline: tokio::time::Instant,
) -> Result<(), std::io::Error> {
    let final_deadline = settle_deadline + WARM_BACKEND_PROBE_FINAL_RECONCILE_TIMEOUT;
    let mut removed_any = false;
    while tokio::time::Instant::now() < settle_deadline {
        let candidate = match tokio::time::timeout_at(settle_deadline, w.probe_transition_candidate(probe_object)).await {
            Ok(candidate) => candidate?,
            Err(_) => break,
        };

View on GitHub (pinned to 5dca076efe)

Solutions

1. Use a versioned remote tier backend that supports candidate discovery (S3/GCS with versioning or generation listing enabled).
2. Check the tier target configuration and re-create the tier pointing at a supported backend.
3. If Unsupported is expected for this backend, avoid probe-based warm-backend checks for it and use a simpler reachability test.

let supports_probe = matches!(backend_kind, BackendKind::S3 | BackendKind::GCS);
if !supports_probe { use basic warm-backend reachability check; }

match check_warm_backend_with_deadlines(&mut backend).await {
    Err(e) if e.kind() == std::io::ErrorKind::Unsupported => {
        // backend cannot support probe PUT discovery; fall back to reachability-only check
    }
    other => other?,
}

When it happens

Trigger: Calling check_warm_backend_with_deadlines (which compensates uncertain probe PUTs) against a remote backend whose candidate listing/probe API reports TransitionCandidateProbe::Unsupported — e.g. a warm backend that cannot enumerate object versions for the probe object.

Common situations: Remote tier implementations lacking version-aware listing (unversioned buckets or backends without generation support), misconfigured tier target, or a backend type that does not implement candidate discovery.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• timed out confirming the final remote tier probe state
• an exact tier probe requires a remote version ID
• timed out reconciling a remote tier probe PUT
• timed out removing the final remote tier probe candidate
• timed out confirming remote tier probe cleanup

AI-assisted analysis of rustfs/rustfs@5dca076efe (2026-09-06). Data as JSON: /api/errors/3fefab177d729e99. Report an issue: GitHub.