{"record":{"id":"0e17842295704857","repo":"risingwavelabs/risingwave","slug":"objectstore-failed-with-io-error-0","errorCode":null,"errorMessage":"ObjectStore failed with IO error: {0}","messagePattern":"ObjectStore failed with IO error: (.+?)","errorType":"error_code","errorClass":"HummockError","httpStatus":null,"severity":"error","filePath":"src/storage/src/hummock/error.rs","lineNumber":37,"sourceCode":"use tokio::sync::oneshot::error::RecvError;\n\n// TODO(error-handling): should prefer use error types than strings.\n#[derive(Error, thiserror_ext::ReportDebug, thiserror_ext::Arc)]\n#[thiserror_ext(newtype(name = HummockError, backtrace))]\npub enum HummockErrorInner {\n    #[error(\"Magic number mismatch: expected {expected}, found: {found}\")]\n    MagicMismatch { expected: u32, found: u32 },\n    #[error(\"Invalid format version: {0}\")]\n    InvalidFormatVersion(u32),\n    #[error(\"Checksum mismatch: expected {expected}, found: {found}\")]\n    ChecksumMismatch { expected: u64, found: u64 },\n    #[error(\"Invalid block\")]\n    InvalidBlock,\n    #[error(\"Encode error: {0}\")]\n    EncodeError(String),\n    #[error(\"Decode error: {0}\")]\n    DecodeError(String),\n    #[error(\"ObjectStore failed with IO error: {0}\")]\n    ObjectIoError(\n        #[from]\n        #[backtrace]\n        ObjectError,\n    ),\n    #[error(\"Meta error: {0}\")]\n    MetaError(String),\n    #[error(\"SharedBuffer error: {0}\")]\n    SharedBufferError(String),\n    #[error(\"Wait epoch error: {0}\")]\n    WaitEpoch(String),\n    #[error(\"Next epoch error: {0}\")]\n    NextEpoch(String),\n    #[error(\"Change log retention miss: table {table_id}, epoch {epoch}\")]\n    ChangeLogRetentionMiss { table_id: TableId, epoch: u64 },\n    #[error(\"Time-travel version expired: table {table_id}, epoch {epoch}\")]\n    TimeTravelVersionExpired { table_id: TableId, epoch: u64 },\n    #[error(","sourceCodeStart":19,"sourceCodeEnd":55,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/storage/src/hummock/error.rs#L19-L55","documentation":"An underlying object store operation (S3/OSS/local FS/mininode) returned an ObjectError during a Hummock read/write. Hummock wraps it via #[from] so storage code sees a single HummockError type. The ObjectError payload carries the store-specific message.","triggerScenarios":"Any Hummock object-store call (get/upload/delete SSTs) that fails at the store layer: IO errors, HTTP failures, missing objects, misconfigured credentials.","commonSituations":"S3 outages or throttling; wrong bucket/endpoint/region config; expired or missing credentials; local filesystem full or permission denied for Hummock storage path.","solutions":["Read the wrapped ObjectError to identify the store failure and fix the root cause (credentials, endpoint, network).","Verify object store config (bucket, region, endpoint, access keys) in risewave.toml / Hummock options.","Check the object store service health and quota (S3 status, rate limits).","Retry the failed read/write; for transient store outages Hummock retry logic or a restart may clear it."],"exampleFix":"// before\n[h ummock] s3.endpoint = \"http://wrong-host:9000\"\n// after\n[hummock] s3.endpoint = \"http://minio:9000\", s3.access_key = \"...\", s3.region = \"us-east-1\"","handlingStrategy":"retry","validationCode":"// Before starting, verify object store reachability\naws s3api head-bucket --bucket <bucket> && echo ok","typeGuard":null,"tryCatchPattern":"// Rust\nmatch hummock_op() {\n    Err(e) if e.to_string().starts_with(\"ObjectStore failed with IO error\") => {\n        // inspect inner ObjectError; retry with backoff for transient IO/HTTP failures\n    }\n    r => r?,\n}","preventionTips":["Validate bucket/endpoint/credentials config before cluster start.","Monitor object store availability and rate limits.","Use reliable, low-latency network to the object store endpoint.","Provision local disk space when using local object stores."],"tags":["storage","hummock","object-store","io"],"backgroundTag":"file-read-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}