risingwavelabs/risingwave · critical · ObjectError

s3 error: {inner}

Error message

s3 error: {inner}

What it means

An S3 object-store operation failed; the underlying AWS SDK error is wrapped into the RisingWave ObjectError::S3 variant with the message 's3 error: {inner}'. The variant carries should_retry so callers can decide whether the failure is transient.

Source

Thrown at src/object_store/src/object/error.rs:29

// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

use std::io;

use aws_sdk_s3::operation::get_object::GetObjectError;
use aws_sdk_s3::operation::head_object::HeadObjectError;
use aws_sdk_s3::primitives::ByteStreamError;
use aws_smithy_types::body::SdkBody;
use risingwave_common::error::BoxedError;
use thiserror::Error;
use thiserror_ext::AsReport;
use tokio::sync::oneshot::error::RecvError;

#[derive(Error, thiserror_ext::ReportDebug, thiserror_ext::Box, thiserror_ext::Construct)]
#[thiserror_ext(newtype(name = ObjectError, backtrace))]
pub enum ObjectErrorInner {
    #[error("s3 error: {inner}")]
    S3 {
        // TODO: remove this after switch s3 backend to opendal
        should_retry: bool,
        #[source]
        inner: BoxedError,
    },
    #[error("disk error: {msg}")]
    Disk {
        msg: String,
        #[source]
        inner: io::Error,
    },
    #[error(transparent)]
    Opendal(#[from] opendal::Error),
    #[error(transparent)]
    Mem(#[from] crate::object::mem::Error),
    #[error("Internal error: {0}")]
    #[construct(skip)]

View on GitHub (pinned to 6469eb736d)

Solutions

1. Read the wrapped #[source] inner error for the concrete AWS SDK reason (AccessDenied, NoSuchBucket, etc.) and fix the root cause.
2. Verify S3 config: bucket name, region, endpoint, and credentials (state store / hummock S3 options).
3. Check IAM policy for required s3:GetObject/PutObject/DeleteObject/ListBucket on the bucket.
4. Test connectivity to the S3 endpoint from the compute/meta nodes.
5. If should_retry is true, the operation is transient — retry with backoff.

Example fix

// before: wrong region/endpoint
// config: s3.endpoint = "https://s3.us-east-1.amazonaws.com", bucket in eu-west-1
// after
// config: s3.region = "eu-west-1"; s3.endpoint unset (or matching the bucket's region)
// and ensure credentials/permissions:
// aws iam attach s3:{GetObject,PutObject,DeleteObject,ListBucket} on the bucket

// smoke-test S3 before starting RW
let (client, err) = { let c = aws_sdk_s3::Client::new(&cfg); (c.clone(), c.list_objects_v2().bucket(bucket).send().await.err()) };
assert!(err.is_none(), "S3 preflight failed: {err:?}");

// treat only retryable S3 errors as transient
fn is_retryable(e: &ObjectError) -> bool {
    e.should_retry()
}

match op().await {
    Err(e) if e.should_retry() => backoff_retry(op, 5).await,
    Err(e) => return Err(e.into()),
    Ok(v) => Ok(v),
}

When it happens

Trigger: Any S3 GET/PUT/DELETE/LIST via the object store returns an SDK error: invalid bucket/credentials, missing IAM permissions, network failure, throttling, object not found for required reads, expired session tokens.

Common situations: Misconfigured AWS keys or IRSA role; bucket region mismatch; no network/VPC endpoint to S3; bucket deleted or name typo; S3 throttling under load; ETags/conditional request failures.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• SstableUpload error: {0}
• Storage error: {0}
• s3 url {location} should have a '/' at the start of path.
• check if version hint exist failed: {}
• Fail to check exist

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/629c1d358c43e68a. Report an issue: GitHub.