rustfs/rustfs · error · std::io::Error

InvalidInput

InvalidInput

Error message

an exact tier delete requires a remote version ID

What it means

Default implementation of WarmBackend::remove_exact in rustfs-ecstore. Backends that cannot pin an exact remote version fall back to this default, which refuses to delete without a remote version ID: with rv empty there is no way to guarantee the delete targets the specific transitioned instance, so it fails fast with InvalidInput instead of deleting the wrong or latest version.

Source

Thrown at crates/ecstore/src/services/tier/warm_backend.rs:121

    /// family uses the transition client's declared-length request plus
    /// Content-MD5 for multipart parts, while GCS materializes the body before
    /// awaiting its buffered write response. Test backends may deliberately
    /// violate this contract to exercise transition compensation.
    async fn put(&self, object: &str, r: ReaderImpl, length: i64) -> Result<String, std::io::Error>;
    /// The same completion contract as [`WarmBackend::put`] applies when
    /// metadata is attached.
    async fn put_with_meta(
        &self,
        object: &str,
        r: ReaderImpl,
        length: i64,
        meta: HashMap<String, String>,
    ) -> Result<String, std::io::Error>;
    async fn get(&self, object: &str, rv: &str, opts: WarmBackendGetOpts) -> Result<ReadCloser, std::io::Error>;
    async fn remove(&self, object: &str, rv: &str) -> Result<(), std::io::Error>;
    async fn remove_exact(&self, object: &str, rv: &str) -> Result<(), std::io::Error> {
        if rv.is_empty() {
            return Err(std::io::Error::new(
                std::io::ErrorKind::InvalidInput,
                "an exact tier delete requires a remote version ID",
            ));
        }
        self.remove(object, rv).await
    }
    async fn probe_transition_candidate(&self, _object: &str) -> Result<TransitionCandidateProbe, std::io::Error> {
        Ok(TransitionCandidateProbe::Unsupported)
    }
    async fn in_use(&self) -> Result<bool, std::io::Error>;
}

fn parse_http_timestamp(value: &str) -> Option<OffsetDateTime> {
    OffsetDateTime::parse(value, &Rfc3339)
        .or_else(|_| OffsetDateTime::parse(value, &Rfc2822))
        .ok()
}

View on GitHub (pinned to 9e6e02ea09)

Solutions

1. Enable versioning on the remote tier bucket, then re-transition affected objects so their metadata records a remote version ID
2. Use a provider/backend that returns version IDs on upload (S3-family tiers override remove_exact with exact semantics)
3. Audit transitioned objects whose metadata lacks the remote-version field and re-run their transitions
4. Do not hand-edit xl.meta transition fields; let the transition path populate the remote version

Example fix

# before: tier bucket unversioned
aws s3api create-bucket --bucket tier-bucket
# after: versioned tier bucket records a remote version ID per transition
aws s3api put-bucket-versioning --bucket tier-bucket --versioning-configuration Status=Enabled

// Skip exact-delete when no remote version was recorded
if remote_version_id.is_empty() {
    // tier bucket unversioned or transition predated version tracking;
    // re-transition instead of calling remove_exact
}

match backend.remove_exact(object, rv).await {
    Err(e) if e.kind() == std::io::ErrorKind::InvalidInput && rv.is_empty() => {
        // missing remote version: enable versioning and re-transition
    }
    other => other?,
}

When it happens

Trigger: ILM tier cleanup or transition rollback calls remove_exact(object, "") — the object metadata carries no remote version ID (tier bucket unversioned or the transition response omitted a version) and the backend did not override remove_exact with version-aware semantics.

Common situations: Tier bucket created without versioning enabled so transitions record no version ID; mixed fleet where an older node wrote transition metadata without remote-version state; provider returning no version header on PUT; manual metadata surgery removed the remote-version field.

Related errors

• Unsupported
• InvalidData
• InvalidInput
• InvalidData
• InvalidData

AI-assisted analysis of rustfs/rustfs@9e6e02ea09 (2026-08-16). Data as JSON: /api/errors/34337d93b82ed58b. Report an issue: GitHub.