{"record":{"id":"629c1d358c43e68a","repo":"risingwavelabs/risingwave","slug":"s3-error-inner","errorCode":null,"errorMessage":"s3 error: {inner}","messagePattern":"s3 error: (.+?)","errorType":"exception","errorClass":"ObjectError","httpStatus":null,"severity":"critical","filePath":"src/object_store/src/object/error.rs","lineNumber":29,"sourceCode":"// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nuse std::io;\n\nuse aws_sdk_s3::operation::get_object::GetObjectError;\nuse aws_sdk_s3::operation::head_object::HeadObjectError;\nuse aws_sdk_s3::primitives::ByteStreamError;\nuse aws_smithy_types::body::SdkBody;\nuse risingwave_common::error::BoxedError;\nuse thiserror::Error;\nuse thiserror_ext::AsReport;\nuse tokio::sync::oneshot::error::RecvError;\n\n#[derive(Error, thiserror_ext::ReportDebug, thiserror_ext::Box, thiserror_ext::Construct)]\n#[thiserror_ext(newtype(name = ObjectError, backtrace))]\npub enum ObjectErrorInner {\n    #[error(\"s3 error: {inner}\")]\n    S3 {\n        // TODO: remove this after switch s3 backend to opendal\n        should_retry: bool,\n        #[source]\n        inner: BoxedError,\n    },\n    #[error(\"disk error: {msg}\")]\n    Disk {\n        msg: String,\n        #[source]\n        inner: io::Error,\n    },\n    #[error(transparent)]\n    Opendal(#[from] opendal::Error),\n    #[error(transparent)]\n    Mem(#[from] crate::object::mem::Error),\n    #[error(\"Internal error: {0}\")]\n    #[construct(skip)]","sourceCodeStart":11,"sourceCodeEnd":47,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/object_store/src/object/error.rs#L11-L47","documentation":"An S3 object-store operation failed; the underlying AWS SDK error is wrapped into the RisingWave ObjectError::S3 variant with the message 's3 error: {inner}'. The variant carries should_retry so callers can decide whether the failure is transient.","triggerScenarios":"Any S3 GET/PUT/DELETE/LIST via the object store returns an SDK error: invalid bucket/credentials, missing IAM permissions, network failure, throttling, object not found for required reads, expired session tokens.","commonSituations":"Misconfigured AWS keys or IRSA role; bucket region mismatch; no network/VPC endpoint to S3; bucket deleted or name typo; S3 throttling under load; ETags/conditional request failures.","solutions":["Read the wrapped #[source] inner error for the concrete AWS SDK reason (AccessDenied, NoSuchBucket, etc.) and fix the root cause.","Verify S3 config: bucket name, region, endpoint, and credentials (state store / hummock S3 options).","Check IAM policy for required s3:GetObject/PutObject/DeleteObject/ListBucket on the bucket.","Test connectivity to the S3 endpoint from the compute/meta nodes.","If should_retry is true, the operation is transient — retry with backoff."],"exampleFix":"// before: wrong region/endpoint\n// config: s3.endpoint = \"https://s3.us-east-1.amazonaws.com\", bucket in eu-west-1\n// after\n// config: s3.region = \"eu-west-1\"; s3.endpoint unset (or matching the bucket's region)\n// and ensure credentials/permissions:\n// aws iam attach s3:{GetObject,PutObject,DeleteObject,ListBucket} on the bucket","handlingStrategy":"retry","validationCode":"// smoke-test S3 before starting RW\nlet (client, err) = { let c = aws_sdk_s3::Client::new(&cfg); (c.clone(), c.list_objects_v2().bucket(bucket).send().await.err()) };\nassert!(err.is_none(), \"S3 preflight failed: {err:?}\");","typeGuard":"// treat only retryable S3 errors as transient\nfn is_retryable(e: &ObjectError) -> bool {\n    e.should_retry()\n}","tryCatchPattern":"match op().await {\n    Err(e) if e.should_retry() => backoff_retry(op, 5).await,\n    Err(e) => return Err(e.into()),\n    Ok(v) => Ok(v),\n}","preventionTips":["Preflight bucket access (list + put) before cluster startup.","Keep credentials via IAM roles/IRSA; rotate before expiry.","Match bucket region and endpoint in config.","Monitor S3 5xx/throttle metrics and set alerting.","Apply exponential backoff on retryable errors only."],"tags":["s3","aws","object-store","network","storage"],"backgroundTag":"http-error-response","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}