risingwavelabs/risingwave · error · HummockError

ObjectStore failed with IO error: {0}

Error message

ObjectStore failed with IO error: {0}

What it means

An underlying object store operation (S3/OSS/local FS/mininode) returned an ObjectError during a Hummock read/write. Hummock wraps it via #[from] so storage code sees a single HummockError type. The ObjectError payload carries the store-specific message.

Source

Thrown at src/storage/src/hummock/error.rs:37

use tokio::sync::oneshot::error::RecvError;

// TODO(error-handling): should prefer use error types than strings.
#[derive(Error, thiserror_ext::ReportDebug, thiserror_ext::Arc)]
#[thiserror_ext(newtype(name = HummockError, backtrace))]
pub enum HummockErrorInner {
    #[error("Magic number mismatch: expected {expected}, found: {found}")]
    MagicMismatch { expected: u32, found: u32 },
    #[error("Invalid format version: {0}")]
    InvalidFormatVersion(u32),
    #[error("Checksum mismatch: expected {expected}, found: {found}")]
    ChecksumMismatch { expected: u64, found: u64 },
    #[error("Invalid block")]
    InvalidBlock,
    #[error("Encode error: {0}")]
    EncodeError(String),
    #[error("Decode error: {0}")]
    DecodeError(String),
    #[error("ObjectStore failed with IO error: {0}")]
    ObjectIoError(
        #[from]
        #[backtrace]
        ObjectError,
    ),
    #[error("Meta error: {0}")]
    MetaError(String),
    #[error("SharedBuffer error: {0}")]
    SharedBufferError(String),
    #[error("Wait epoch error: {0}")]
    WaitEpoch(String),
    #[error("Next epoch error: {0}")]
    NextEpoch(String),
    #[error("Change log retention miss: table {table_id}, epoch {epoch}")]
    ChangeLogRetentionMiss { table_id: TableId, epoch: u64 },
    #[error("Time-travel version expired: table {table_id}, epoch {epoch}")]
    TimeTravelVersionExpired { table_id: TableId, epoch: u64 },
    #[error(

View on GitHub (pinned to 6469eb736d)

Solutions

1. Read the wrapped ObjectError to identify the store failure and fix the root cause (credentials, endpoint, network).
2. Verify object store config (bucket, region, endpoint, access keys) in risewave.toml / Hummock options.
3. Check the object store service health and quota (S3 status, rate limits).
4. Retry the failed read/write; for transient store outages Hummock retry logic or a restart may clear it.

Example fix

// before
[h ummock] s3.endpoint = "http://wrong-host:9000"
// after
[hummock] s3.endpoint = "http://minio:9000", s3.access_key = "...", s3.region = "us-east-1"

// Before starting, verify object store reachability
aws s3api head-bucket --bucket <bucket> && echo ok

// Rust
match hummock_op() {
    Err(e) if e.to_string().starts_with("ObjectStore failed with IO error") => {
        // inspect inner ObjectError; retry with backoff for transient IO/HTTP failures
    }
    r => r?,
}

When it happens

Trigger: Any Hummock object-store call (get/upload/delete SSTs) that fails at the store layer: IO errors, HTTP failures, missing objects, misconfigured credentials.

Common situations: S3 outages or throttling; wrong bucket/endpoint/region config; expired or missing credentials; local filesystem full or permission denied for Hummock storage path.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Storage error: {0}
• check if version hint exist failed: {}
• Fail to check exist
• (dataset guard acquisition error)
• StorageError {error}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/0e17842295704857. Report an issue: GitHub.