risingwavelabs/risingwave · error · ObjectError

Internal error

Error message

Internal error: {0}

What it means

ObjectError::Internal is a catch-all for unexpected internal failures in the RisingWave object store layer. It is constructed via ObjectError::internal(msg) (the #[construct(skip)] variant is only creatable through that helper) and signals a condition that should not normally occur, such as an inconsistent store state or an unrecoverable logic error.

Solutions

1. Read the embedded message and the object-store logs around it to identify which internal assertion failed
2. Retry the read/write operation; transient backend inconsistencies often resolve on retry
3. Check for known issues matching the message in the RisingWave repo and upgrade if fixed
4. File an issue with the full message and stack trace if reproducible

Example fix

// before: surfacing a raw unwrap-style failure
let meta = store.metadata(path).unwrap();
// after: handle the ObjectResult properly
let meta = store.metadata(path).await.map_err(|e| {
    tracing::error!(error = ?e, "metadata lookup failed");
    e
})?;

null

fn is_internal_object_error(e: &ObjectError) -> bool {
    e.to_string().starts_with("Internal error:")
}

match store.read(path, range).await {
    Ok(d) => d,
    Err(e) if e.to_string().starts_with("Internal error:") => {
        tracing::error!(error = ?e, "object store internal error");
        return Err(e.into());
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Any object-store code path calling ObjectError::internal(msg) — e.g. invalid store state, impossible responses from a backend, or wrapped logic errors that do not fit a specific variant.

Common situations: Corrupted metadata from remote stores, mismatches between expected and actual object state after failover, or bugs surfaced during Hummock compaction reads.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• (dataset guard acquisition error)
• NotFound error
• Other error
• Passing s3-compatible is not supported, please modify the…
• remote object store only supports s3, minio, gcs, oss, cos…

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/62a1571d9ea15036. Report an issue: GitHub.

Appendix: source

Thrown at src/object_store/src/object/error.rs:46

pub enum ObjectErrorInner {
    #[error("s3 error: {inner}")]
    S3 {
        // TODO: remove this after switch s3 backend to opendal
        should_retry: bool,
        #[source]
        inner: BoxedError,
    },
    #[error("disk error: {msg}")]
    Disk {
        msg: String,
        #[source]
        inner: io::Error,
    },
    #[error(transparent)]
    Opendal(#[from] opendal::Error),
    #[error(transparent)]
    Mem(#[from] crate::object::mem::Error),
    #[error("Internal error: {0}")]
    #[construct(skip)]
    Internal(String),
    #[cfg(madsim)]
    #[error(transparent)]
    Sim(#[from] crate::object::sim::SimError),

    #[error("Timeout error: {0}")]
    Timeout(String),
}

impl ObjectError {
    pub fn internal(msg: impl ToString) -> Self {
        ObjectErrorInner::Internal(msg.to_string()).into()
    }

    /// Tells whether the error indicates the target object is not found.
    pub fn is_object_not_found_error(&self) -> bool {
        match self.inner() {

View on GitHub (pinned to 6469eb736d)